Add option to support non-pandas dataframes (e.g. polars) in Python recipes

info-rchitect
info-rchitect Registered Posts: 193 ✭✭✭✭✭✭
in Product Ideas

Hi,

There are many pandas alternatives. One that is new and very fast is polars. Polars is built on Rust so it is memory safe and runs in parallel by design. I use polars in one of my recipes but have to convert it to pandas to write the dataset.

thx

Tagged:
Share on FacebookShare on Twitter
23
Up
23 votes

In the Backlog · Last Updated

Comments

  • ClemenceB
    ClemenceB Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Product Ideas Manager Posts: 18 Dataiker

    Thanks @info-rchitect
    , this has been added to the backlog!

    · Share on FacebookShare on Twitter
  • arbolja
    arbolja Registered Posts: 1 ✭

    Hello! I am very interested in this as well. Is there any easy way of checking the life cycle of the updates in the backlog (for example, if they have a tracking number, if they have been implemented in some version...)? Thanks!

    · Share on FacebookShare on Twitter
  • Niyazi
    Niyazi Registered Posts: 12 ✭✭✭✭

    Completely agree!

    Currently, there is no benefit in using Polars, duckdb, Ray or Dask, because retrieving dataset always means getting Pandas DataFrame object. Converting it to anything else defeats the purpose.

    You can write the results directly to the database, but it is a workaround: you'd still have to initialize the dataset object first. And your code environment would also require you to have db drivers and so on.

    Would be nice to have different options for data retrieval and data writing. Polars seems like a good candidate.

    · Share on FacebookShare on Twitter
  • WH
    WH Registered Posts: 17 ✭✭✭✭

    Would love to see polars supported alongside pandas

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…