Dataiku DSS 7 AMA is live! Learn more

chunked download of get_download_stream

Level 4
chunked download of get_download_stream
Jump to solution

Hi, where can I find an example of a chunked download of the file from the Folder python object (dataiku.core.managed_folder.Folder)? 



In this example I would like to write by a specified chunk size, for example print out something after every 1MB:



        x=in_folder.get_download_stream( "/"+f )

        with out_folder.get_writer(f) as writer:

            parts = list(  x.read_chunked(1024*1024) )

            for part in parts:

                print('Part')

                writer.write( part )

        print('File uploaded')




ResponseNotChunked: Response is not chunked. Header 'transfer-encoding: chunked' is missing.


Thanks

0 Kudos
1 Reply
Dataiker
Dataiker
Hi,

The data is not transferred using chunked encoding, so you can't use read_chunked. However, you can just read by blocks using the regular Python IO API: read(size)
0 Kudos
Labels (3)