How to create reusable plugins to work with managed folders ?

Options
arrawat
arrawat Registered Posts: 3

We are trying to develop a plugin which should be able to read `n` number of files from a managed folder and put `m` number of files back to same or different managed folder, here `n` is not equals to `m`.
To achieve that we have created a project and exported that as a plugin. But the issue is our flow is starting with managed folder which eventually is getting replaced in the project where it is being used. Snapshots attached below.

1st Snapshot: Flow maintained in Parent project
How flow is designed in original project.

2nd Snapshot: Usage of plugin in Usage project.
2.png

3rd Snapshot: Flow executed in Usage project (enabled Debug application in Plugin)
How flow is being used in debug instance of plugin. Note: Original managed folder is being replaced by shared folder of Usage project, but custom recipe of instance still points to original folder.

Error that we are getting:

Error in Python process: At line 11: <class 'Exception'>: Managed folder D1****H* cannot be used : declare it as input or output of your recipe

Explanation:
Here, D1****H* is the original folder of the project which got converted to Plugin. Ideally, this should have been the reference to manage folder of Usage project.

Plugin Configuration:

{
   "projectExportManifest": {
      "exportUploads": false,
      "exportAllInputDatasets": false,
      "exportAllDatasets": false,
      "exportManagedFS": false,
      "exportAnalysisModels": false,
      "exportSavedModels": false,
      "exportModelEvaluationStores": false,
      "exportProjectResources": false,
      "exportLabelingTasks": false,
      "exportManagedFolders": false,
      "exportAllInputManagedFolders": false,
      "exportInsightsData": false,
      "includedDatasetsData": [],
      "includedSavedModels": [],
      "includedManagedFolders": [],
      "includedModelEvaluationStores": [],
      "includedCodeStudios": [],
      "includedLabelingTasks": [],
      "exportGitRepository": false
   },
   "instanceFeatures": {
      "showFlowNavLink": false,
      "showLabNavLink": false,
      "showCodeNavLink": false,
      "showSwitchToProjectViewButton": true,
      "showVersionControlFeatures": false
   },
   "useAppHomepage": true,
   "homepageSections": [],
   "id": "PROJECT_POC*********",
   "label": "POC - *******",
   "shortDesc": "",
   "tags": [],
   "useAsRecipeSettings": {
      "icon": "icon-puzzle-piece",
      "category": "Custom",
      "inputRoles": [
         {
            "objectId": "D1****H*",
            "roleLabel": "Input",
            "type": "MANAGED_FOLDER",
            "$touched.it.roleLabel": true,
            "$invalid": false
         }
      ],
      "outputRoles": [
         {
            "objectId": "X****J*t",
            "roleLabel": "Output",
            "type": "MANAGED_FOLDER",
            "$touched.it.roleLabel": true,
            "$invalid": false
         }
      ],
      "variablesEditionTile": {
         "behavior": "MODAL",
         "params": []
      },
      "runScenarioTile": {
         "scenarioId": "RUNE2E",
         "buttonText": "Run E2E"
      }
   },
   "instantiationPermission": "USE_APP_MASTER_PERMISSIONS",
   "limitedVisibilityEnabled": "ENABLED",
   "accessRequestsEnabled": "ENABLED",
   "allowedMissingConnections": [],
   "allowedMissingCodeEnvs": []
}

Summary:
Our original managed folder is getting replaced by folder of project, where the plugin is being used but we are not able to use that folder in original recipe.

> I hope there is a way to dynamically set the managed folder in custom recipes so that it could be taken up directly in custom recipes of our plugin.

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,725 Neuron
    Options

    Your input and output folders will be unique and change between projects. You should therefore make a Recipe Plugin.

Setup Info
    Tags
      Help me…