What is the maximum number of scenario allowed in a project

dss-slb Registered Posts: 7

Hi all,


  1. Is there any limit on the number of scenario allowed in a project?
  2. Is there any limit on how many scenario can run simultaneously?


Best Answer

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,726 Neuron
    Answer ✓



    I have an ETL workflow created in my Dataiku project's Flow where it has a source dataset, some intermediate datasets for data transformation and then write to a target dataset. Then I am going to create a scenario to build the target dataset triggered by an event when new data arrives at the source dataset. The connection settings for the target dataset are defined as scenario variables.

    Most of wht you have above are not requirements but how you implemented them. From the above I can determine that have input datasets of the same schema in different databases which need to be processed in a similar way, is this correct?


    1. The workflow can write to a single target dataset only.

    This is not a requirement. In fact the next "requirement" contradicts it since it says:


    2. The workflow must be able to run on different target datasets and be run concurrently as soon as new source data is available.

    The first part is requirement but the concurrent part is not a requirement, it's an implementation approach. Your requirement in this case could be that new data needs to be processed within 5 mins of being available. If a non-concurrent solution achieves this why make it more complicated than it needs to be? Why do you need concurrency?


    3. It should be contained in a single Dataiku project and the project will be deployed to a new Dataiku instance using a python script.

    Neither of these two part statements are requirements. Why does it need to be a single project? What requirement does that achieve? Deployment is not really something that should be covered as a requirement. The usually approach for Dataiku projects is to deploy to Automation node via project bundle. This can be automated with a Python script.

    Based on the additional information you have provided I believe at this point you should be looking at Dataiku Applications. Dataiku Applications allow you to re-use flows within multiple projects and process data in a similar way. Any changes in the Dataiku Application the project are using will result in changes in the child projects.


Setup Info
      Help me…