Using Dataiku

Hi @samuel_acr_96 ,

If the issue still persists, can you please open support along with the instance diagnostics taken immediately after your tests? The logs may provide more information on the exact exception

Thanks

Hi @samuel_acr_96 ,

If the issue still persists, can you please open support along with the instance diagnostics taken immediately after your tests? The logs may provide more information on the exact exception

Thanks

Hi again!

This use case would be better handled by our team in a support ticket if you'd like to create one and follow up on there:

https://support.dataiku.com/support/tickets/new

We would be happy to provide support and help you with your use case:)

Best,

Yasmine

Hi again!

This use case would be better handled by our team in a support ticket if you'd like to create one and follow up on there:

https://support.dataiku.com/support/tickets/new

We would be happy to provide support and help you with your use case:)

Best,

Yasmine

Well the post you linked says you should pass the key as an HTTP header variable ("X-DKU-APIKey") not using basic authentication. So change your Postman request to pass the key via the required variable. And the API Designer is something completely different than Webapps so it’s not relevant to your issue.

So the first thing you need to realise is that while Dataiku allows you to build a complex data pipeline in a visual way without writting any code this is never going to be the most optimal way of loading/preparing large datasets as fast as possible. The fact that DSS persists all the intermediate datasets is both a big advantage (explainability, debug, etc) and a big dissadvantage too (lots of redundant data, lots of reads and writes). Depending on the recipes and connections that you use you may be able to enable SQL pipelines in part of your flow which should make those recipes run faster.

You should change the flow view to Recipe engines. Any recipe showing as DSS engine should be reviewed because this means the data will have to moved to the DSS server for processing and back to the database for writing the output. This tends to be slower than SQL engine which means the execution happens only on the database without data moving to Dataiku.

Finally you should review your SQL database and make sure it's sized and tuned accordingly. When you start to get into millions of rows traditional RDBMS databases start to struggle so moving to other technologies that can handle billions of rows at speed will help (like Databricks, Snowflake, BigQuery, etc).

Hi HAFEDH,

If you are working working in a large enterprise, I suppose your dataiku instance is managed by an IT service or specific infras service ? In this case, it's not your responsability to change these settings if you have not been infromed of their effectiveness. And if changing the params has an impact on performance, this means that you can affect the availability of the spark queue. It's up to you to decide whether you want to take the risk, given that you specify that the request to use spark jobs in your entreprise is significant and that, you have no knowledge of spark sessions.

Nonetheless, technical you can just try to create a pyspark recipe to script whatever you want to benchmark and try fit tuning a batch of differents config empirically. But the result will depend on your task's resource needs, as each type of job you want to optimize has different needs and performance depending on catching, shuffle, memory and parallelization .

Hi,

I hope that you are doing well.

From my understanding you have two columns (col_1 and col_2) with both geopoint data and would like to calculate the distance between these two points as output in a third column (col_3).

If that is the case, we do have a processor called geo distance that is meant to computes the geodesic distance between a geospatial column and another geospatial object. The computation outputs a number of distance units (kilometers, miles) in another column.

In the case of a distance between two geometries, the distance is the shortest distance between these two.

(see: https://doc.dataiku.com/dss/latest/preparation/processors/geo-distance.html).

We have a step by step tutorial available here: https://knowledge.dataiku.com/latest/data-preparation/prepare-recipe/tutorial-geo-processing.html#compute-the-distance-between-two-geopoints

If this does not cover your use case/your request is different, please let me know!

Best,

Yasmine

No. This has nothing to do with Regex. File based Dataiku datasets use fixed data type schemas. If the file changes you have to manually update the schema. Only Python recipes can write dynamic schemas as their output. And even doing so will complicate your flow so your best option is to pivot the data so months are rows not columns.

Use a SQL recipe and you can copy / paste your SQL.

https://developer.dataiku.com/latest/concepts-and-examples/datasets/datasets-data.html#typing-and-schema