Spark job execution on EKS using IRSA
pi_485
Partner, L2 Admin, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Registered Posts: 6 Partner
Hi,
We have Dataiku on EC2 instance and EKS is present in another account. We have configured Dataiku to access EKS using assume roles.
We had a discussion with AWS and they suggested to use service account and annotations for accessing AWS services (like, S3).
Is there a way that we can implement service accounts and annotations in Dataiku to execute spark jobs in pods?
Answers
-
pi_485 Partner, L2 Admin, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Registered Posts: 6 Partner
IRSA implementation requires Spark docker image rebuilt.