When reducing the columns in a dataset, identify the columns to keep, not the columns to remove

Dataiker Alumni
1 min read 5 2 2,627

Often, a source dataset includes many more columns than the ones we need for the downstream flow. In a Prepare recipe, the temptation is to use a Remove processor to delete the columns that you don’t need to carry forward in the flow.




However, if, later, more columns are added to the source dataset, you will also need to update the Remove processor adding those columns. Otherwise, you may inadvertently add the newly added columns to the downstream schemas.

Instead, the tip is to use the Keep processor instead, positively identifying the columns you want to carry forward.


In this case, when new columns are added to the source dataset, the Prepare recipe does not need to be updated, as the Keep processor automatically ignores any newly added columns, keeping downstream schemas consistent.

Now the tricky bit, when you have many columns, how do you easily select the ones to keep? It is counterintuitive, but the easiest way is to actually convert a Remove processor into a Keep processor:

  1. In your Prepare recipe, change to column view;
  2. Select the columns you want to KEEP;
  3. In the Actions menu, select Delete (trust me);




4. In the processor, at the bottom of the columns list, switch from Remove to Keep.




"Voilà”, you have a Keep processor that positively identifies the columns you need.


Level 2

I do exactly the same thing!!!

Level 1

I am finding selecting the columns difficult. When I click on the name of the field if frequently does not highlight and the step gets an orange error background. I need to click in different spots to get it to select. Is there something I am doing wrong? v11 on Windows in Chrome.