How to create reusable plugins to work with managed folders ?
We are trying to develop a plugin which should be able to read `n` number of files from a managed folder and put `m` number of files back to same or different managed folder, here `n` is not equals to `m`.
To achieve that we have created a project and exported that as a plugin. But the issue is our flow is starting with managed folder which eventually is getting replaced in the project where it is being used. Snapshots attached below.
1st Snapshot: Flow maintained in Parent project
2nd Snapshot: Usage of plugin in Usage project.
3rd Snapshot: Flow executed in Usage project (enabled Debug application in Plugin)
Error that we are getting:
Error in Python process: At line 11: <class 'Exception'>: Managed folder D1****H* cannot be used : declare it as input or output of your recipe
Explanation:
Here, D1****H* is the original folder of the project which got converted to Plugin. Ideally, this should have been the reference to manage folder of Usage project.
Plugin Configuration:
{ "projectExportManifest": { "exportUploads": false, "exportAllInputDatasets": false, "exportAllDatasets": false, "exportManagedFS": false, "exportAnalysisModels": false, "exportSavedModels": false, "exportModelEvaluationStores": false, "exportProjectResources": false, "exportLabelingTasks": false, "exportManagedFolders": false, "exportAllInputManagedFolders": false, "exportInsightsData": false, "includedDatasetsData": [], "includedSavedModels": [], "includedManagedFolders": [], "includedModelEvaluationStores": [], "includedCodeStudios": [], "includedLabelingTasks": [], "exportGitRepository": false }, "instanceFeatures": { "showFlowNavLink": false, "showLabNavLink": false, "showCodeNavLink": false, "showSwitchToProjectViewButton": true, "showVersionControlFeatures": false }, "useAppHomepage": true, "homepageSections": [], "id": "PROJECT_POC*********", "label": "POC - *******", "shortDesc": "", "tags": [], "useAsRecipeSettings": { "icon": "icon-puzzle-piece", "category": "Custom", "inputRoles": [ { "objectId": "D1****H*", "roleLabel": "Input", "type": "MANAGED_FOLDER", "$touched.it.roleLabel": true, "$invalid": false } ], "outputRoles": [ { "objectId": "X****J*t", "roleLabel": "Output", "type": "MANAGED_FOLDER", "$touched.it.roleLabel": true, "$invalid": false } ], "variablesEditionTile": { "behavior": "MODAL", "params": [] }, "runScenarioTile": { "scenarioId": "RUNE2E", "buttonText": "Run E2E" } }, "instantiationPermission": "USE_APP_MASTER_PERMISSIONS", "limitedVisibilityEnabled": "ENABLED", "accessRequestsEnabled": "ENABLED", "allowedMissingConnections": [], "allowedMissingCodeEnvs": [] }
Summary:
Our original managed folder is getting replaced by folder of project, where the plugin is being used but we are not able to use that folder in original recipe.
> I hope there is a way to dynamically set the managed folder in custom recipes so that it could be taken up directly in custom recipes of our plugin.
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,139 Neuron
Your input and output folders will be unique and change between projects. You should therefore make a Recipe Plugin.