chunked download of get_download_stream

Registered, Neuron 2022 Posts: 121 ✭✭✭✭✭✭

Hi, where can I find an example of a chunked download of the file from the Folder python object (dataiku.core.managed_folder.Folder)?

In this example I would like to write by a specified chunk size, for example print out something after every 1MB:

x=in_folder.get_download_stream( "/"+f )

with out_folder.get_writer(f) as writer:

parts = list( x.read_chunked(1024*1024) )

for part in parts:

print('Part')

writer.write( part )

print('File uploaded')


ResponseNotChunked: Response is not chunked. Header 'transfer-encoding: chunked' is missing.

Thanks

Best Answer

  • Dataiker, Dataiku DSS Core Designer, Registered, Moderator Posts: 753 Dataiker
    Answer ✓
    Hi,

    The data is not transferred using chunked encoding, so you can't use read_chunked. However, you can just read by blocks using the regular Python IO API: read(size)

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.