kubernetes EKS storage file limit

[SunghoPark]

hello

I am contacting you because I am having difficulty using Eks S3 on Dataiku.

When saving data on the Dataiku server, there is no limit to the number of data, but when saving data on EKS s3, no more than 4000 pieces can be saved. I don't think it's a storage issue because the file formats are all different, but I'd like to know the cause.

thank you.

상기

[AlexT]

Hi @SunghoPark ,
This is an unusual limit in terms of 4k pieces to hit. It's unlikely related to S3 per say or EKS but something else like credentials used for S3 e.g you are hitting 1h STS token limit when using role chaining:

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html

To investigate this further, please raise a support ticket with the job diagnostics running on EKS. You could try to use a connection with Access Key and Secret instead.

Kind Regards,