Issue: Non-matching partitioning scheme for «Table Name» R recipe

Options
abhiboss25
abhiboss25 Dataiku DSS Core Designer, Registered Posts: 2 ✭✭✭✭

Hi Experts,

I have a R recipe and I am writing to data frames (which are hive tables) using below function.

dkuWriteDataset(df1,"Hive Table A") - for this I want partition pattern as %Y/%M/%D/.*

Partition_pattern_A.jpg

 dkuWriteDataset(df2,"Hive Table B") - for this I want partition pattern as %Y/%M/%D/%H/.*

Partition_pattern_B.jpg
but while running recipe it gives me error: Non-matching partitioning scheme for Hive Table B.
However, if I keep partition pattern same for both the tables ( %Y/%M/%D/.*) then there is no issue.

Also, I would like to know how can I create partition for every run?

Tagged:

Answers

  • Liev
    Liev Dataiker Alumni Posts: 176 ✭✭✭✭✭✭✭✭
    Options

    Hi @abhiboss25

    Not sure if you managed to solve the issue on your own in the end.

    When dealing with partitions, it is important to be clear on the partitions to be built and the partitions to use in order to build them.

    Please refer to our documentation on this topic.

    Good luck!

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer Posts: 753 Dataiker
    Options

    Hi,

    Specifically, what you are trying to do is impossible. Dataiku cannot write, in a single recipe, to two datasets that don't have the same partitioning scheme.

    Hence, you cannot write to a day-partitioned and to a hour-partitioned dataset at the same time.

    You will need to use two recipes for that.

Setup Info
    Tags
      Help me…