Sharepoint Plugin not returning more than 5,000 records in one list but working in another
Hi,
We have two lists, one with 70,000 records the other with 35,000 records. Both on the same site.
We switched them both from Classic to Modern Experience, and this worked for one list, allowing me to read in the enitre dataset of 70,000+ records into the sample.
However the other list, same site, same setting (modern vs classic), is generating this error:
Failed to read from python dataset : <class 'sharepoint_client.SharePointClientError'> : Error (get_list_items): (get_list_items) The attempted operation is prohibited because it exceeds the list view threshold.
I am not sure why one would work and the other wouldn't, but its a show stopped for our proof of concept of DataIku. My only alternative is to generate the dataset via a python script at the start of the flow using the base sharepoint API, which is able to return all 35009 records without any problem.
Any help appreciated.
i
Answers
-
Hi !
We managed to reproduce this error by sorting the list using a non indexed column. Could you check that the default view is filter / sorting your list, and whether the column for that is indexed or not ? If that's the case, you can either
- remove the sorting / filtering from default view
- index the column used for the sorting / indexing
- create an additional "Dataiku" view on SharePoint, and instruct the plugin to use it (Create new view > List & "Dataiku" on SharePoint, Show advanced parameters > View name > "Dataiku" on DSS)
Hope this helps,
Alex
-
Thanks, but none of these are working.
We tried removing any sorting at all, ensuring sorting was on just one indexed column, and have tried with a new list with both of these options, and the default view. Very strange on works but the other doesnt.
Regards
i
-
From SharePoint, if you right click on the list, then Settings > Advanced settings, is "Automatic Index Management" on "Yes" ?
-
Yes it is, and it is also on with the list which is actually working with 75,000 records.
i
