joins in spark engine

Mahesh_M
Mahesh_M Posts: 1 ✭✭✭
in Using Dataiku

when I use Left join with Join recipe using Spark engine and parquet datasets, it is not giving expected results.

I have 2M records in left table and 1 M in right table, but the result is only 2 record.

Tagged:
· Share on Facebook

Answers

  • Sarina
    Sarina Posts: 321 Dataiker

    Hi @Mahesh_M
    ,

    Can you please open a support ticket, and attach a job diagnostic for the job that results in the unexpected number of output rows? In addition, can you make sure to attach screenshots for both input datasets highlighting several rows of data that you expected to join together based on the join condition that then do not appear in the output dataset? That should help with troubleshooting!

    Thanks,
    Sarina

    · Share on Facebook
  • tgb417
    tgb417 Posts: 1,649 Neuron

    @Mahesh_M
    ,

    Definitely, followup with support.

    · Share on Facebook
  • VInayshankar
    VInayshankar Posts: 2 Partner

    Hi,

    I am facing same issue. I am using Dataiku 9.0.3 version.

    If your issue is resolved. can you please post resolution here.

    · Share on Facebook
  • Sarina
    Sarina Posts: 321 Dataiker

    Hi @vinayshankar
    ,

    The issue is likely specific to your data and join setup. If you would like us to review it, please also open a support ticket, and attach a job diagnostic for the job that results in the unexpected number of output rows? In addition, can you make sure to attach screenshots for both input datasets highlighting several rows of data that you expected to join together based on the join condition that then do not appear in the output dataset? That should help with troubleshooting!

    Thanks,
    Sarina

    · Share on Facebook
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…