kubernetes EKS storage file limit
hello
I am contacting you because I am having difficulty using Eks S3 on Dataiku.
When saving data on the Dataiku server, there is no limit to the number of data, but when saving data on EKS s3, no more than 4000 pieces can be saved. I don't think it's a storage issue because the file formats are all different, but I'd like to know the cause.
thank you.
상기
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,225 DataikerHi @SunghoPark
,
This is an unusual limit in terms of 4k pieces to hit. It's unlikely related to S3 per say or EKS but something else like credentials used for S3 e.g you are hitting 1h STS token limit when using role chaining:
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html
To investigate this further, please raise a support ticket with the job diagnostics running on EKS. You could try to use a connection with Access Key and Secret instead.
Kind Regards,