Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
We are having some problems while trying to access a managed folder which refers to a sharepoint location. I have reauthenticated user accounts for sharepoint.
Error message -
Oops: an unexpected error occurred Listing files failed, caused by: Exception: Error 429 (get_files) Please see our options for getting help HTTP code: 500, type: java.io.IOException
Error 429 occurs when too many requests have been made to SharePoint Online in a short amount of time . You can reduce the occurrence of this problem by using SSO as a form of authentication (option 2 in the plugin documentation ).
I am already using SSO as the auth method using Presets. When I try to write files to a Sharepoint location in a loop, I always get this error (its not fixed at which point this gets thrown).
Exception: None: b"Failed to write data : <class 'sharepoint_client.SharePointClientError'> : Error 429 (create_folder)"
Could you give us more details about your setup, so that we can reproduce the problem on our side ? In particular, how do you loop the write ? Is it done from a python code ? What is the size and number of the files and the delay between two rewrites ?
out_folder = dataiku.Folder("odbID")
for currentCountry in country1:
mydataset_df_Country = mydataset_df[mydataset_df['COUNTRY'] == currentCountry]
compCodeFiltered = mydataset_df_Country['COMPANY CODE'].unique()
#Looping through the company codes in each country
for compCode in compCodeFiltered:
mydataset_df_CompCode= mydataset_df_Country[mydataset_df_Country['COMPANY CODE'] == compCode]
fiscalYrFiltered = mydataset_df_CompCode['COMMENTARY YEAR'].unique()
#Looping through each year present for the company
for fiscalYrCurrent in fiscalYrFiltered:
mydataset_df_FiscalYr= mydataset_df_CompCode[mydataset_df_CompCode['COMMENTARY YEAR'] == fiscalYrCurrent]
sourceQuarterFiltered = mydataset_df_FiscalYr['SOURCE_QUARTER'].unique()
#Looping through each quarter within the year to create separate output files
for quarters in sourceQuarterFiltered:
df = mydataset_df_FiscalYr[mydataset_df_FiscalYr['SOURCE_QUARTER'] == quarters].copy()
stream = BytesIO()
excel_writer = pd.ExcelWriter(stream, engine='xlsxwriter')
# Rewind to the begining of the bytes string
with out_folder.get_writer("/file/%s/%s/%d/static folder/%s_Test_%d_%d_Output.xlsx" %(currentCountry, compCode, commentaryYear, compCode, sourceQuarter,commentaryYear)) as writer:
print("Time is ",time.time())
Thank you for sharing this code. Could you also give me a rough idea of the average excel file size once stored on SharePoint ? With this I should have enough information to reproduce the issue on our side...