How do I specify partitions?

UserBird
UserBird Dataiker, Alpha Tester Posts: 535 Dataiker

Best Answer

  • jrouquie
    jrouquie Dataiker Alumni Posts: 87 ✭✭✭✭✭✭✭
    Answer ✓

    There are two places where you specify partitions in a recipe:

    • when running the recipe, you need to tell DSS wich partitions of the output dataset you want to build. For instance “1789-07-14/1789-07-23” would ask DSS to build ten consecutive partitions (pertaining to a long time ago).
    • when defining the recipe, you need to specify “partition dependencies”. Whenever DSS wants to build a partition of the output, it needs to know which partition of the input dataset are to be used. There are many ways to specify a dependency, for instance “from two days before to 0 days before”, in which case to build the output partition 1789-07-20 DSS would use the input partitions 1789-07-18 to 1789-07-20.

    See also http://doc.dataiku.com/dss/latest/partitions/identifiers.html

Setup Info
    Tags
      Help me…