Partitioning dataset & getting next hour partition

Bader
Level 3
Partitioning dataset & getting next hour partition

Hi Dataiku Team,  

I have dataset thats get updated every hour. This dataset is not-partitioned. I would like to load the new partition(hourly) every hour. 

I have done the following 

1-  Sync the source dataset to destination dataset

2- Partition the destination dataset by timestamp 

3- Editing the sync recipe and use redispatch partition because the source is not partitioned

4- Build the dataset.

 

Every things looks good and the dataset got partitioned correctly. The issue comes when I try to get the next hour partition. 

I have created scenario and create a build step with PREVIOUS_HOUR partition. The issue, DSS is trying to reload all the dataset again and repatriation all partition again. 

How can I get only next hour partition ? 

0 Kudos
5 Replies
Ignacio_Toledo

Hi @Bader, do you have some screenshots of the settings on the sync recipe related to input-output? Something similar to what is shown in the screenshot?

Selection_357.png

I think this could help to have some idea at what might be happening

0 Kudos
Bader
Level 3
Author

Hi @Ignacio_Toledo ,  

Thanks for your comment. I have fixed the issue by doing below steps:  

Lets assume we have source dataset called "dataset_A" (Not partition dataset)

1- Sync the source dataset to destination dataset "dataset_B"

2- Partition the destination dataset "dataset_B" by timestamp 

3- Editing the sync recipe and use redispatch partition because the source "dataset_A" is not partitioned

4- Build the dataset "dataset_B"

Now, The dataset "dataset_B" is partitioned 

5- Create sync recipe the source dataset is "dataset_B", the destination is "dataset_C" 

6- Build "dataset_C" with all partition data

7- Create scenario and create a build step with PREVIOUS_HOUR partition 

I hope the steps are clear. 

 

Thanks

Kind regards

0 Kudos
Ignacio_Toledo

The steps are clear @Bader, thanks. However you say "I've fixed the issue", does this mean that the problem is actually solved then and the scenario working as you expect?

0 Kudos
Bader
Level 3
Author

Yes 

Ignacio_Toledo

OK, good to know then! Thanks!

0 Kudos

Labels

?
Labels (1)
A banner prompting to get Dataiku