Advanced Designer Learning Path is now live! Read More

Spark job execution on EKS using IRSA

Level 2
Level 2
Spark job execution on EKS using IRSA

Hi,

We have Dataiku on EC2 instance and EKS is present in another account. We have configured Dataiku to access EKS using assume roles.

We had a discussion with AWS and they suggested to use service account and annotations for accessing AWS services (like, S3).

Is there a way that we can implement service accounts and annotations in Dataiku to execute spark jobs in pods?

0 Kudos
1 Reply
Level 2
Level 2
Author

IRSA implementation requires Spark docker image rebuilt.

0 Kudos
A banner prompting to get Dataiku DSS