Sharepoint connection and file parsing

Hi !

In your current setup, you have used a custom dataset to access SharePoint. Doing this, the files contained in the SharePoint directory are already selected and opened, and the following python recipe as only access to the actual content.

To access the file list, you need instead to access your SharePoint directory as a managed folder, and then use the python corresponding API, which can be found here.

To do that, go to your Dataiku project flow > +Dataset > Folder, and pick Sharepoint online / shared document in for the "Store into" box.

 At first an error message will appear because the settings are not selected. Go to Settings > Type of authentication and pick your preset. Next, use the Browse option to go to the SharePoint folder of interest. Once it is selected, press Save. In the View tab, you should now see the content of your SharePoint directory.

From this folder, you can create a python recipe:

The recipe's template code will now use the dataiku.Folder method. Using list_paths_in_partition() should give you the list of files in the SharePoint folder.

 Hope this will help,

Alex

Indeed, in this context you have to use get_download_stream to open the file.

It is actually a good thing to access your files this way, since it makes your code able to work regardless to the underlying method used to access the actual storage (local or not).

If it is one file, with always the same name, you can open it as a dataset (just like you did in your initial flow), and then use the "Export to folder" recipe which let save the data, in various file format, in a managed folder.

If it is something more complex (multiple files, some processing to do in between), you can have a recipe reading from a SharePoint folder, writing into a local folder, and use get_download_stream + read / get_writer + write. For instance, a code like this would copy all the file present in the input folder (be it a SharePoint folder or anything else) into your target folder (which can be local):

# -*- coding: utf-8 -*-
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
InputDLISSharedpoint = dataiku.Folder("RIsZUUUk")
InputDLISSharedpoint_info = InputDLISSharedpoint.get_info()

paths = InputDLISSharedpoint.list_paths_in_partition()
print("File list: {}".format(paths))

# Write recipe outputs
target = dataiku.Folder("wCWCzPXy")
target_info = target.get_info()

for path in paths:
    data = None
    with InputDLISSharedpoint.get_download_stream(path) as input_file:
        data = input_file.read()
    with target.get_writer(path) as output_file:
        output_file.write(data)