(As there is no obvious pattern in the col names i can't use the pattern in recipe remove col )

Many Thanks

Axel

The number of columns in this is variable but it will always be a multiple of 3.

The expected output:

Feel free to suggest a solution in Python, but I would prefer visual recipes.

Dataiku version used: 14.3.3

Operating system used: Linux

Operating system used: Linux

1 - Usage of "unmatched rows". The column selection occurs after the join does not apply to data that isn't joined. In this case I am using both sets of data so need the option to select columns from both sets.

2 - Removal of unneeded/unwanted columns after filtering. This is especially important when using sensitive HR data.

This enhances the automation of processing the data vs. adding to it by doing cleanup after the project has completed running. It also allows me to confirm that I have sensitive data removed before sharing the data with others rather than relying on a manual process to remove it.

I am working on the project "Advanced Designer Assessment"

after modified Prepare recipe to add column "qualifies", when i open dataset "Online_Retail_Prepared", i got this error message.

 Oops: an unexpected error occurred

java.lang.IllegalStateException: Expected a double but was BEGIN_ARRAY at line 377 column 21 path $.charts[0].def.scatterZoomOptions.scale, caused by: IllegalStateException: Expected a double but was BEGIN_ARRAY at line 377 column 21 path $.charts[0].def.scatterZoomOptions.scale

would you please help me to figure it out?

thanks in advance.

Perhaps by making the 'hamburger' icons on the Output panel draggable.

While working with the embedding recipe, we faced a limitation where we have two datasets, we want to apply the rag on, how can we apply the knowledge bank on them specifically?

Regards

The issue I am having is that the DSS engine is producing a completely different result than when I use the SQL engine. Has anyone faced a similar issue? I would appreciate some insight on this.

Basically, all I want to do is produce a columns with the MAX() value inferred from another column. No partitions, no order bys, simple enough? At least that's what I thought.

It looks like DSS is Ordering By a hidden index on its own and then creating a Window Frame that takes the current row and all preceding rows. Here is an example table to show you what is supposed to happen and what is in fact happening:

Supposed to happen: (This is what is happening in SQL (In-database) engine)

What is in fact happening: (This is what is happening in DSS Engine)

After running my flow, my snowflake table is still empty, I thought the whole point of creating a connection using a data table was to be able to read / write to that table?

Am I understanding this wrong? is there any way I can set up my flow so that my results are outputting to my snowflake table / connection and actually writing to the table?

Operating system used: windows

I have a column that has been parsed and is in UTC, when I try to format the date to be in eastern / New York time I get a new column that is -5 hours, but isn't the current the current difference -4 hours? I'm sure this has something to do with daylight savings time vs normal time, but I just want to ensure that my formula remains working even when times are changed by an hour due to daylight savings.

does anybody know how I can get the correct time difference from UTC to ET?

I've attached the steps I'm currently using below

Operating system used: windows

I'm reading the data from SharePoint and the format of the file is Cost Center_06092024.xlsx

As the file comes with the date format, I partitioned reading the data as /Cost Center_%M%D%Y.xlsx and in my prepare recipe set the option as Last available by that the flow ONLY get the latest file.

I'm trying to create a scenario as a Trigger on dataset change. Can you please help me with the option to use here?

Attached is the screenshot for reference

Thanks

Satish

Operating system used: Browser

I am sharing below a minimum reproducible project that triggered an error in one of our larger workflows involving the stack recipes.We have been seeing these errors for Snowflake tables (they may exist in others) around string length and truncation.

The culprit seems to be that Dataiku is automatically recognizing string length when Snowflake tables are created with specific queries but using the smallest column length to create the output schema for the Stacked Recipe.

Of course I can get around this by manually defining the "Table Creation SQL", but would prefer this is addressed on the product level if possible.

Thanks,

Yash

{
"flowname": "${scenarioName}",
"status": "${outcome}",
"summary": "${failedEventsSummary}"
}

The issue is failedEventsSummary is providing too much text. How can we get just the ERROR on why the scenario failed.

Operating system used: Browser

I'm trying to populate the Tile Num and Tile Sequence Number in the attached picture format. Trying to use windows recipe with no luck.

Can someone please help with this?

Attached is the data, the ask is to make sure generate a tile num for INS column. Highlighted the color combinations in the picture.

Operating system used: Browser

I am following the RAG tutorial link -> https://knowledge.dataiku.com/latest/ml-analytics/gen-ai/tutorial-question-answering-using-rag-approach.html#

While trying to run the Embed Recipe I am getting error as follows.

Oops: an unexpected error occurred

Error in Python process: <class 'dataikuapi.utils.DataikuException'>: com.dataiku.dip.io.SocketBlockLink$SecretKernelTimeoutException: Subprocess failed to connect, it probably crashed at startup. Check the logs., caused by: SocketException: Socket operation on nonsocket: configureBlocking

HTTP code: , type: <class 'dataikuapi.utils.DataikuException'>

Kindly help me to fix this issue. Have attached the logs

Operating system used: Windows 10

Operating system used: Windows 10

Operating system used: Windows 10

1. I have deployed a model to the flow
2. I want to retrain that model with its associated "train" recipe
3. I understand that the model's performance is evaluated using a test set or K-folds under a cross-validation strategy

My question: after retraining the model using the "train" recipe, is the resulting new active model fit on the entire dataset (as best practice sometimes suggests to do so)?

I can't find any information on this final fitting strategy in the recipe (see screenshot below) and failed to find such information in dataiku's documentation.

Operating system used: WIndows 10

I have a question regarding the coalesce recipe in dataiku. I wanted to use the coalesce funtion in dataiku Formula and the preview that I have in the prepare recipe shows that the function works and it shows the value that I want. But after executing the recipe, the resulting column shows a different output from the preview.

I have made sure that the order of the values in the coalesce function is correct and that the empty cells are NULL instead of an empty string. I could not comprehend where the mistake is. The pictures are attached below. Picture 1 shows the preview in prepare recipe and Picture 2 shows the resulting dataset.

Thanks in advance!

When working with Redshift tables in DSS visual recipes we noticed that the table creation settings sometimes defaults to setting certain column lengths to the redshift max (65,000). In many cases this becomes excessive. For example, in the screenshot below the "brand" column has a length of 65k but most of the column has text that span less than 10 characters.

We wanted to better understand the logic of column length setting defaults for Redshift and if there is a safe / proper way to modify this.

Q1. Is there a way to set this dummy manually instead of Dataiku's default method? Want to avoid using category handling = custom preprocessing option.

Q2. Using Variable type = Categorical with Drop one dummy option on input variable of double type seems to be dropping 2 levels. For example, there are only 3 regression coefficients from a variable with 5 levels). I would of expected there would be 4 regression coefficients since 1 is used as a dummy). Does anyone know the reason for this?

Many thanks in advance.

Consequently, I am seeking a method to standardize the model training between the two projects by setting the random state. I am utilizing a random forest classifier within a visual recipe, and random forests in scikit-learn have a `random_state` attribute.

Is there a recommended approach to achieve this?

Operating system used: Redhat 8

I want to add a Column B, in which the values are forced to be integer.

I know I can do Column B = substring (Column A, -4), and Dataiku will automatically convert the values to integer, but the conversion process is a black box to me, and I don't know what's the conversion criteria/logic and when it might fail.

I thought I could add "numval" in front of the substring to force the conversion but it didn't work and returned blank.

Is there a logic I could apply to achieve this? Basically something like:

pd.to_numeric(df['Column A'].str[:4], errors='coerce')

Operating system used: win 11

I can use an Embed recipe to create an Embedding dataset / Vector Store.

Then I can setup an LLM two query the resulting object in its settings.

But, how to go from there? How can I ask a question / query to the Embedding object? Clicking on it only gives the option of a Python recipe and there's also nothing like a Visual webapp.

Operating system used: AWS Linux