Hello, I am in charge of migrating some data quality control pipeline that are currently running on Dataiku to Google Cloud using SQL. I am currently on a JOIN RECIPE job and there is one step that confuse me, it is the "pre-join computed columns" as you can see over here:
This is what pre-join computed columns contains:
And if I open FCST_filter or FCSTDRAFT_filter, I am getting the same thing:
What I was expecting is to have a new column for each table (FCST_filter or FCSTDRAFT_filter,) called exists with the expect same value for every row, 1. But in fact, I do have some 0 and some 1 on my output table for the exist column. Do you have some idea where does this number come from?
Based on the information it looks like the pre-computed would be doing a "SELECT 1 FROM Table"
Which should return the constant 1 for every row of the table if the record matches your where clause and/or join. If it doesn't match it return nothing. Perhaps those are the 0's you are seeing not sure if they are directly or perhaps added later in your flow.