Using Dataiku
- Hi all! I'm currently working on a very simple project that I need to deploy as an application. I'm very, very new to the platform, so I'm not sure how to even begin. I already have a Flow that takes …
Solution by TurribeachBefore progressing further with your solution you need to understand certain limitations of the Dataiku world. In Dataiku there is no support for running a flow, a scenario or a recipe concurrently. This means that "there can only be one" running at a single point of time. In general flows/scenarios/recipes are meant to run in "batch" mode in a scheduled basis, not on demand where an adhoc user can select an input and trigger an execution. It's perfectly possible to have a Dataiku WebApp that allows a user trigger a flow/scenario/recipe but this would only work for the first user that does it. If a subsequent user attemped to run the same flow/scenario/recipe while it is already running it will most likely fail (the actual outcome of this depends on which object you are trying to re-run). So before you continue further with your design you need to consider whether your solution needs to support concurrent runs or not. Sometimes you can get away with no concurrent runs and therefore you can use a flow/scenario/recipe design. For instance if you need to support a single user or if the user will only trigger a flow/scenario/recipe once a day. Where you can't use this design approach you need to move away from flow/scenario/recipe "batch" design and look at the API node and deploy an API service. Dataiku API Services allow developers to create APIs which can be used for "realtime" use cases, where you provide a set of inputs and you want to produce a unique output for that request. Typically these are used in scoring/prediction use cases like predicting the probability of default (ie when a customer applies for a credit card). The API node supports high availability and scalability which is something you will expect for a service that's expected to run concurrently and service multiple concurrent requests/users.
Now let's talk about Dataiku Applications. Dataiku Applications allow you to design and package your project as a reusable application with customizable inputs. While Dataiku Applications can run concurrently the use case Dataiku Applications is aimed at it's not the concurrent realtime use of a flow/scenario/recipe. Dataiku Applications allow to re-use a flow design into other projects so the use case is code reusability. As such in order to achive concurrency with Dataiku Applications you have to use the Dataiku Application into different projects. This is not something that fits your use case where you want to have a single project that serves user requests. I guess you could have a kludge solution where you convert your flow into a Dataiku Application and then you create multiple projects all using the same Dataiku Application (ie Project_Slot_1, Project_Slot_2, Project_Slot_3, Project_Slot_n) which allow you to execute your flow concurrently and have your WebApp sort of handle which application is being used at the moment and choose a "free slot" to execute the user inputs. But this seems like a bad pattern to me.
So now that you understand how these solutions work you should clarify what your requirements are in terms of concurrency.
Solution by TurribeachBefore progressing further with your solution you need to understand certain limitations of the Dataiku world. In Dataiku there is no support for running a flow, a scenario or a recipe concurrently. This means that "there can only be one" running at a single point of time. In general flows/scenarios/recipes are meant to run in "batch" mode in a scheduled basis, not on demand where an adhoc user can select an input and trigger an execution. It's perfectly possible to have a Dataiku WebApp that allows a user trigger a flow/scenario/recipe but this would only work for the first user that does it. If a subsequent user attemped to run the same flow/scenario/recipe while it is already running it will most likely fail (the actual outcome of this depends on which object you are trying to re-run). So before you continue further with your design you need to consider whether your solution needs to support concurrent runs or not. Sometimes you can get away with no concurrent runs and therefore you can use a flow/scenario/recipe design. For instance if you need to support a single user or if the user will only trigger a flow/scenario/recipe once a day. Where you can't use this design approach you need to move away from flow/scenario/recipe "batch" design and look at the API node and deploy an API service. Dataiku API Services allow developers to create APIs which can be used for "realtime" use cases, where you provide a set of inputs and you want to produce a unique output for that request. Typically these are used in scoring/prediction use cases like predicting the probability of default (ie when a customer applies for a credit card). The API node supports high availability and scalability which is something you will expect for a service that's expected to run concurrently and service multiple concurrent requests/users.
Now let's talk about Dataiku Applications. Dataiku Applications allow you to design and package your project as a reusable application with customizable inputs. While Dataiku Applications can run concurrently the use case Dataiku Applications is aimed at it's not the concurrent realtime use of a flow/scenario/recipe. Dataiku Applications allow to re-use a flow design into other projects so the use case is code reusability. As such in order to achive concurrency with Dataiku Applications you have to use the Dataiku Application into different projects. This is not something that fits your use case where you want to have a single project that serves user requests. I guess you could have a kludge solution where you convert your flow into a Dataiku Application and then you create multiple projects all using the same Dataiku Application (ie Project_Slot_1, Project_Slot_2, Project_Slot_3, Project_Slot_n) which allow you to execute your flow concurrently and have your WebApp sort of handle which application is being used at the moment and choose a "free slot" to execute the user inputs. But this seems like a bad pattern to me.
So now that you understand how these solutions work you should clarify what your requirements are in terms of concurrency.
- Hi Everyone, We are building a Dataiku webapp. We want to get dynamic input from the user through UI and when the user submits the prompt then the prompts gets processed via LLM through AI recipe. We …
Last answer by tgb417Cool, So you are using the Dataiku LLM Mesh. But using the Dataiku API to get access from your Flask bases App. I can imagine that others in the community might want to learn even more about how this is going for you. Is it performant enough, reliable enough, etc.
Last answer by tgb417Cool, So you are using the Dataiku LLM Mesh. But using the Dataiku API to get access from your Flask bases App. I can imagine that others in the community might want to learn even more about how this is going for you. Is it performant enough, reliable enough, etc.
- I am connecting to QuickBase using a URL path and looking for a way to dynamically add or remove columns from the initial schema as well as rename the columns as "Simplify (and lower case)". I do not …Last answer byLast answer by Turribeach
@CurtisC
wrote:but unsure how this will interpret a new column with special characters or spaces.
It won't work as Bigquery won't accept those as valid columns names. You need to rename those columns hence why I suggested the Python recipe as the Prepare recipe can't rename columns dynamically whereas you can do that in Python. There is no other alternative.