Download split CSVs

Options
natejgardner
natejgardner Neuron, Registered, Neuron 2022, Neuron 2023 Posts: 151 Neuron

In some cases, data systems require a maximum data file size. Multiple uploads are needed to load a full dataset. For Dataiku's dataset download functionality, under advanced features, it would be helpful if it were possible to automatically split a CSV into files of a maximum size and quickly download it. The workaround of using the split recipe can be cumbersome since the total number of files is variable depending on the number of records in the dataset. For example, I have a 50 MB dataset I'd like to download as a series of 5 MB files. Right now, my options are the split recipe and a Python recipe storing to a managed folder. But both of these are time consuming when data is needed in ad hoc cases. Building it into the download system would save that time.

0
0 votes

New · Last Updated

Setup Info
    Tags
      Help me…