Group Recipe - Min / Max calculation problem

dfwmike

Are there any tips or tricks beyond what's covered in the training for using group recipes?

I'm having problems getting the correct min and max values calculated using group recipes. I can confirm that sometimes they calculate the correct values and sometimes they don't.

This is probably user error, but the dataset is large enough to not allow me to not use a sample, making it difficult or impossible to trace the error back to a previous recipe step.

Ignacio_Toledo

Hi @dfwmike
,

I'm not sure how to help you without a more detailed example of the cases where you find wrong values. Could you share a more concrete example?

Also, are you using partitioned datasets? Just from experience, I've had some group issues when not using the partitioned datasets in the right way.

dfwmike

Thanks, @Ignacio_Toledo
!

I am using a dataset imported from Snowflake by someone else who is not reachable at the moment. Is there a way I can tell if the dataset is partitioned?

I have data that looks like this:

ID Value

A 10

A 11

B 9

A 15

B 8

When I group to get the min, I get results that look like this...

ID min_Value

A 11

B 8

When I group to get the max, I get results like this...

ID max_Value

A 11

B 9

For B, it works perfectly. But for A, it gives me the same value in both situations and the value it gives me is neither the min nor the max.

The value is calculated in a formula in the prepare recipe that immediately precedes the groupings.

Ignacio_Toledo

Hi @dfwmike
,

The easiest way to check if a dataset is partitioned is by looking at is "icon" in the flow:

As you can see it looks like a "set" of superimposed icons.

Could you share a screen shot of your group recipe? Like this example:

I'm mostly interested in the areas within the red boxes.

Cheers!

dfwmike

@Ignacio_Toledo

Thanks again! The dataset is not partitioned.

Attached is a screenshot of the group recipe where I'm calculating the ADR max by CUSTOMER_VEHICLE_VRM.

Ignacio_Toledo

Thanks for the capture. From your recipe I can't seem to find a clear indication of what might be happening. Maybe, and following your previous example, some of the IDs look similar but are actually different? Like instead of just having "A" values you also have "A " (and A plus a space), but in that case you would be seeing apparently repeating IDs (or repeated 'CUSTOMER_VEHICLE_VRM')

Just to discard a final issue, could you share, from the same recipe, what you have in the tabs "Input/output" and "Advanced", like in these examples:

 Hope that with this extra info we might get something!

dfwmike

I don't think the CUSTOMER_VEHICLE_VRM values are different because at an earlier point in the flow, I do a count by CUSTOMER_VEHICLE_VRM and I get the correct number.

Attached are screenshots from the requested tabs.

Thanks!

Ignacio_Toledo

Hi @dfwmike
,

Thanks for the screenshots. Sadly I'm running out of the ideas.

If you select both min and max to be calculated at the same time in the grouping recipe, do you still get the same min and max values?

Is there any subset of the data that you could share publicly? To try to replicate the problem.

Hope we can find the problem!

dfwmike

Yes, I've replicated the problem doing min and max in the same recipe and in separate recipes. I started with one recipe and split them out when I encountered this problem to see if that would help (and of course it didn't).

Unfortunately, I can't share the dataset at present. I'd have to mask the customer identifiers it contains to do that.

Thank you for your help! I've opened a ticket with Dataiku support.

Mike

Ignacio_Toledo

I was going to suggest next to create a support ticket, so great!

Let us know what they found, it will be interesting to find out the cause.

Good luck!

EliasH

The issue for @dfwmike
has been resolved, thank you @Ignacio_Toledo
for your assistance!

Jurre

Hi @EliasH
and @dfwmike
, Would it be possible to expand a little on the nature of the problem and it's final solution ? I use this functionality a lot so there is some interest in the reliability of it (haven't encountered any challenges with it to date..)