Submit your inspiring success story or innovative use case to the 2022 Dataiku Frontrunner Awards! ENTER YOUR SUBMISSION

Error 429 while reading managed folder - Sharepoint

Solved!
zeno_11
Level 3
Error 429 while reading managed folder - Sharepoint
 

Hi,

We are having some problems while trying to access a managed folder which refers to a sharepoint location. I have reauthenticated user accounts for sharepoint.

Error message - 

 

 Oops: an unexpected error occurred
Listing files failed, caused by: Exception: Error 429 (get_files)
Please see our options for getting help

HTTP code: 500, type: java.io.IOException

 

 

 

0 Kudos
1 Solution
AlexB
Dataiker
Dataiker

Hi !

The latest version of the plugin (1.0.11) is finally available on the store. It should fix error 429 related issues during large folders uploads.

 

View solution in original post

0 Kudos
9 Replies
AlexB
Dataiker
Dataiker

Hi !

Error 429 occurs when too many requests have been made to SharePoint Online in a short amount of time . You can reduce the occurrence of this problem by using SSO as a form of authentication (option 2 in the plugin documentation ).

0 Kudos
zeno_11
Level 3
Author

Hi,

I am already using SSO as the auth method using Presets. When I try to write files to a Sharepoint location in a loop, I always get this error (its not fixed at which point this gets thrown).

 Exception: None: b"Failed to write data : <class 'sharepoint_client.SharePointClientError'> : Error 429 (create_folder)"

Thanks. 

0 Kudos
AlexB
Dataiker
Dataiker

Could you give us more details about your setup, so that we can reproduce the problem on our side ? In particular, how do you loop the write ? Is it done from a python code ? What is the size and number of the files and the delay between two rewrites ?

0 Kudos
zeno_11
Level 3
Author
Yes, it is from a python code, I write 1000+ files. 

The loop goes something like this -

out_folder = dataiku.Folder("odbID")
for currentCountry in country1:
           mydataset_df_Country = mydataset_df[mydataset_df['COUNTRY'] == currentCountry]
           compCodeFiltered = mydataset_df_Country['COMPANY CODE'].unique()
#Looping through the company codes in each country
          for compCode in compCodeFiltered:
             mydataset_df_CompCode= mydataset_df_Country[mydataset_df_Country['COMPANY CODE'] == compCode]
             fiscalYrFiltered = mydataset_df_CompCode['COMMENTARY YEAR'].unique()
             #Looping through each year present for the company
                 for fiscalYrCurrent in fiscalYrFiltered:
                  mydataset_df_FiscalYr= mydataset_df_CompCode[mydataset_df_CompCode['COMMENTARY YEAR'] == fiscalYrCurrent]
                  sourceQuarterFiltered = mydataset_df_FiscalYr['SOURCE_QUARTER'].unique()
                  #Looping through each quarter within the year to create separate output files
                     for quarters in sourceQuarterFiltered:

                   df = mydataset_df_FiscalYr[mydataset_df_FiscalYr['SOURCE_QUARTER'] == quarters].copy()
                   #formatting
                   stream = BytesIO()
#

                   excel_writer = pd.ExcelWriter(stream, engine='xlsxwriter')
                   #more formatting
                   excel_writer.save()

                   # Rewind to the begining of the bytes string
                   stream.seek(0)

                   with out_folder.get_writer("/file/%s/%s/%d/static folder/%s_Test_%d_%d_Output.xlsx" %(currentCountry, compCode, commentaryYear, compCode, sourceQuarter,commentaryYear)) as writer:
                   print("Time is ",time.time())
                   writer.write(stream.read())
                   time.sleep(3)

0 Kudos
AlexB
Dataiker
Dataiker

Thank you for sharing this code. Could you also give me a rough idea of the average excel file size once stored on SharePoint ? With this I should have enough information to reproduce the issue on our side...

0 Kudos
zeno_11
Level 3
Author

Hi,

The files are under 50 kB once stored on Sharepoint.

Thanks

0 Kudos
zeno_11
Level 3
Author

Hi,

Were you able to recreate the scenario or do you have any pointers to the solution for us?

Thanks.

0 Kudos
AlexB
Dataiker
Dataiker

Hi !

Yes we managed to recreate the problem on our side and a fix in currently under review. The new version of the plugin should be soon available on the store.

 

AlexB
Dataiker
Dataiker

Hi !

The latest version of the plugin (1.0.11) is finally available on the store. It should fix error 429 related issues during large folders uploads.

 

0 Kudos