Need more dataset file format while exporting dataset to local such as parquet

Actually to do unit testing on the final and intermediate datasets, need more dataset file formats such as parquet, Avro, sas7bdat, ORC, etc while exporting datasets to the local system for large datasets, as CSV format can't handle more than 1 million records.

6 Comments

It would also be great if more file formats were allowed for import.

Microsoft Access, SQLite, and edb come to mind as most frequently needed.  

It would also be great if more file formats were allowed for import.

Microsoft Access, SQLite, and edb come to mind as most frequently needed.  

PANKAJ
Level 3

@natejgardner 

Yes, I agree with you on more file format options for importing dataset will also be very helpful.

@natejgardner 

Yes, I agree with you on more file format options for importing dataset will also be very helpful.

AshleyW
Dataiker

Hi @natejgardner ,

FYI Microsoft Access and SQLite are supported filte formats for importing data into DSS. I've provided links to the relevant referenc documentation and community articles. If there are file formats that we don't support yet that you'd like to see made available in DSS, feel free to add that as a separate post on the Product Ideas board.

Best,

Ashley

Hi @natejgardner ,

FYI Microsoft Access and SQLite are supported filte formats for importing data into DSS. I've provided links to the relevant referenc documentation and community articles. If there are file formats that we don't support yet that you'd like to see made available in DSS, feel free to add that as a separate post on the Product Ideas board.

Best,

Ashley

CoreyS
Dataiker Alumni
 
Looking for more resources to help you use Dataiku effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as โ€˜Accepted Solutionโ€™ to help others like you!
Status changed to: Gathering Input
 

Thanks @AshleyW , unfortunately these approaches require the files to already be exposed on the network or manually uploaded to the Dataiku server. But most teams I've worked with that generate these will just send them as file attachments. Ideally, they could be uploaded and processed as true flat files the same way Excel and CSV files are. Even when teams do upload their flat file databases to a network location, if they use a Windows file share, if the Dataiku instance doesn't have saml authentication configured, there's no way to authenticate. It'd be a big time saver when working with these sorts of files if these drivers that convert flat file databases into sql connections could also be embedded into the file parsing system directly so Access and SQLite become supported as file formats as well.

Thanks @AshleyW , unfortunately these approaches require the files to already be exposed on the network or manually uploaded to the Dataiku server. But most teams I've worked with that generate these will just send them as file attachments. Ideally, they could be uploaded and processed as true flat files the same way Excel and CSV files are. Even when teams do upload their flat file databases to a network location, if they use a Windows file share, if the Dataiku instance doesn't have saml authentication configured, there's no way to authenticate. It'd be a big time saver when working with these sorts of files if these drivers that convert flat file databases into sql connections could also be embedded into the file parsing system directly so Access and SQLite become supported as file formats as well.

MichaelG
Community Manager
Community Manager
 
I hope I helped! Do you Know that if I was Useful to you or Did something Outstanding you can Show your appreciation by giving me a KUDOS?

Looking for more resources to help you use DSS effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as โ€˜Accepted Solutionโ€™ to help others like you!
Status changed to: Gathering Input