My output datasets are empty after adding condition in existing sample/filter recipe from flow.

deepakdhiman
deepakdhiman Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 10 ✭✭✭✭
edited July 16 in Academy Discussions

My output datasets are empty after adding condition of year 2017 for departures dataset in existing sample/filter recipe for "Hands On : Airport Traffic " in Visual Recipes 101 course.

After adding condition in existing filter recipe from flow, when I right click to "Build Flow outputs reachable from here", Job does execute but subsequent datasets are empty. I am not able to figure out what went wrong and what can I do?

In logs, below section was highlighted in red:

[07:40:33] [WARN] [dku.fsproviders.http] - HEAD method not working, trying GET
com.dataiku.dip.datasets.fs.HTTPSourceException: in act.download_to_departures_NP: HTTP 400: Bad Request
   at com.dataiku.dip.datasets.fs.HTTPFSProvider.execute(HTTPFSProvider.java:333)
   at com.dataiku.dip.datasets.fs.HTTPFSProvider.getResponseForHeadersWithHEADorGET(HTTPFSProvider.java:228)
   at com.dataiku.dip.datasets.fs.HTTPFSProvider.enumerateRecursive(HTTPFSProvider.java:247)
   at com.dataiku.dip.datasets.fs.HTTPFSProvider.enumerateRecursive(HTTPFSProvider.java:51)
   at com.dataiku.dip.server.recipes.KernelsDownloadRecipeService.enumerateSource(KernelsDownloadRecipeService.java:205)
   at com.dataiku.dip.server.recipes.KernelsDownloadRecipeService.copyFromSourceToFolder(KernelsDownloadRecipeService.java:419)
   at com.dataiku.dip.server.recipes.KernelsDownloadRecipeService.copyFromSourcesToFolder(KernelsDownloadRecipeService.java:321)
   at com.dataiku.dip.recipes.download.DownloadRecipeRunner.run(DownloadRecipeRunner.java:75)
   at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)

I am attaching the complete log file. Please assist.

Thank You

Deepak

Best Answer

  • Sean
    Sean Dataiker, Alpha Tester, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer Posts: 168 Dataiker
    Answer ✓

    Hi @deepakdhiman
    , when you return to the Filter recipe, 2017 should already be present. It's 2016 that you are adding-- is that correct?

    Is it possible you forgot to change the condition to keep rows that satisfy all conditions to rows that satisfy at least one of the following conditions? This would explain why the following datasets are "empty" (if by empty you mean that they have 0 rows) since no rows can satisfy both conditions at the same time.

    Screen Shot 2021-06-07 at 11.58.37 AM.png

Answers

  • deepakdhiman
    deepakdhiman Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 10 ✭✭✭✭

    Hi Sean

    Thank you for replying to help me out. I went back to the filter recipe and checked that "all the following conditions" was selected in "Keep only rows that satisfy" field.

    I changed it to "atleast one of the following condition", run and rebuild it. It worked! It was such a silly mistake.

    Thank you so much

  • Sean
    Sean Dataiker, Alpha Tester, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer Posts: 168 Dataiker

    Great! No problem at all.

Setup Info
    Tags
      Help me…