Data source change

Options
emate
emate Dataiku DSS Core Designer, Neuron 2020, Registered, Neuron 2021, Neuron 2022 Posts: 91 ✭✭✭✭✭✭

Hi Team,

Short, theorethical question - let's say I have a flow built where my datasource is connected via Amazon Redshift : Dataset -> SQL databases -> Amazon Redshift
Can I somehow easily change my input from this type connection to a file uploaded manually without rebuilding my flow?

Thanks

Mateusz

Best Answer

  • Liev
    Liev Dataiker Alumni Posts: 176 ✭✭✭✭✭✭✭✭
    Answer ✓
    Options

    You can open a recipe by double-clicking on it in the Flow screen.

    I can see you have several recipes that depend on your Redshift dataset, so the process would need to be repeated for each one.

Answers

  • Liev
    Liev Dataiker Alumni Posts: 176 ✭✭✭✭✭✭✭✭
    Options

    Hi @emate

    One way of doing this would be to:

    - Create the filesystem dataset as per your needs.

    - Open the recipes immediately after the Redshift dataset and go to Input/Output tab

    - Modify the input, replacing the Redshift dataset with the newly created one. Repeat for any other recipe where the input is the Redshift dataset.

    Assuming there are no changes in the schema or actual data, there would be no need to run it. However, when you save the recipe DSS will check whether the column types still match the expectation.

    I hope this helps!

  • emate
    emate Dataiku DSS Core Designer, Neuron 2020, Registered, Neuron 2021, Neuron 2022 Posts: 91 ✭✭✭✭✭✭
    Options

    Hi @Liev

    Thank you, I'm not sure how to "open the recipe immediately after the Redshift" part. Where should I open it?

    Let's assume that I have the same structure in my local file as on Redshift.

    Attached a screen with the 'starting part' of my flow.
    I'm just looking a way to either replace the starting datasource or maybe I can easily copy the whole flow and connect it to a new datasource?

    Thanks

  • emate
    emate Dataiku DSS Core Designer, Neuron 2020, Registered, Neuron 2021, Neuron 2022 Posts: 91 ✭✭✭✭✭✭
    Options

    Oh, I got it now sorry. I thought, I can somehow 'create' new recipe between my input and filter recipe Thanks I will try it your way.

Setup Info
    Tags
      Help me…