This website uses cookies. By clicking OK, you consent to the use of cookies. Read our cookie policy.
Accept
Reject
Community
Survey banner
Share your feedback on the Dataiku documentation with this 5 min survey. Thanks! TAKE THE SURVEY

Add option to support non-pandas dataframes (e.g. polars) in Python recipes

Status: In the Backlog Submitted by on ‎03-08-2022 01:20 PM

Hi,

 

There are many pandas alternatives.  One that is new and very fast is polars.  Polars is built on Rust so it is memory safe and runs in parallel by design.  I use polars in one of my recipes but have to convert it to pandas to write the dataset.

 

thx

Comment
5 Comments
ClemenceB
Dataiker
‎09-01-2022 05:49 PM

Thanks @info-rchitect, this has been added to the backlog!

‎09-01-2022 05:49 PM

Thanks @info-rchitect, this has been added to the backlog!

ClemenceB
Dataiker
‎09-01-2022 05:50 PM
 
‎09-01-2022 05:50 PM
Status changed to: In Backlog
 
MichaelG
Community Manager
Community Manager
‎11-04-2022 11:25 AM
 
I hope I helped! Do you Know that if I was Useful to you or Did something Outstanding you can Show your appreciation by giving me a KUDOS?

Looking for more resources to help you use DSS effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as ‘Accepted Solution’ to help others like you!
‎11-04-2022 11:25 AM
Status changed to: In the Backlog
 
arbolja
Level 1
‎03-16-2023 07:37 AM

Hello! I am very interested in this as well. Is there any easy way of checking the life cycle of the updates in the backlog (for example, if they have a tracking number, if they have been implemented in some version...)? Thanks!

‎03-16-2023 07:37 AM

Hello! I am very interested in this as well. Is there any easy way of checking the life cycle of the updates in the backlog (for example, if they have a tracking number, if they have been implemented in some version...)? Thanks!

zloe
Level 2
‎10-04-2023 11:06 AM

Completely agree!

Currently, there is no benefit in using Polars, duckdb, Ray or Dask, because retrieving dataset always means getting Pandas DataFrame object. Converting it to anything else defeats the purpose.

You can write the results directly to the database, but it is a workaround: you'd still have to initialize the dataset object first. And your code environment would also require you to have db drivers and so on.

Would be nice to have different options for data retrieval and data writing. Polars seems like a good candidate.

‎10-04-2023 11:06 AM

Completely agree!

Currently, there is no benefit in using Polars, duckdb, Ray or Dask, because retrieving dataset always means getting Pandas DataFrame object. Converting it to anything else defeats the purpose.

You can write the results directly to the database, but it is a workaround: you'd still have to initialize the dataset object first. And your code environment would also require you to have db drivers and so on.

Would be nice to have different options for data retrieval and data writing. Polars seems like a good candidate.

Labels

?