Irrelevant warnings during import or Automation node deployment

Turribeach

We have lots of plugins on our Designer instance since we are testing them to see if we can use them in our projects. However this means that whenever we deploy a project to the Automation node we get warnings on deployment as the plugin list in the Automation node does not match with the Designer node. This is a bad user experience. Ideally Dataiku should be able to figure out which plugins are used in each project and don't throw Warnings for plugins that are missing in the Automation node but actually are not used in the project being deployed. Dataiku is already able to determine which connections and code environments are used in a project and validate these exist in the destination Automation node being deployed to so the same should be done for Plugins.

(Topic title edited by moderator to be more descriptive. Original title "Warnings during import or Automation node")

tgb417

User Story:

As an Analyst given a project from another community member, I would like the warning during importation to be more targeted to the actual project I'm installing, so that I can more confidently exchange and receive projects from other community members.

Notes:

Here are the types of warnings that one can receive. This seems like it would be intimidating to a New Analyst on the system.

As the project, I was importing did not have any time series work in it. This warning was irrelevant. As I was not going to continue development on this project the warning about the project-developer plugin was also irrelevant to me as an Analyst. That said these warnings should be available to more advanced users.

Here is a link to a community post that details an example of these challenges for an analyst to import a project.

https://community.dataiku.com/t5/Online-Events/Converting-your-Dataiku-DSS-Project-into-a-Reusable-Application/bc-p/12431/highlight/true#M60

fsergot

Good day,

We have recorded this in the product backlog. We will let you know any progress.

As a side note, having a 100% certainty on which plugins are actually used is not trivial at all, and we don't want to give a false sense of confidence.

Tanguy

allow me to illustrate this annoying alert

Turribeach

"As a side note, having a 100% certainty on which plugins are actually used is not trivial at all, and we don't want to give a false sense of confidence." => Understood but this is not by design, so it can easily be changed. For instance Dataiku could require developers to enable plugins in the Project Settings before being able to use them, which would create a clear way to determine which plugins are "enabled" in each project. Dataiku could also maintain a registry of recipe plugins per project, so this wouldn't need to hard to find out.

Tanguy

partial duplicate of https://community.dataiku.com/t5/Product-Ideas/Warnings-on-plugins-missing-in-the-Automation-node/idi-p/28242

fsergot

Sorry for the late update but this was added in our backlog. This would indeed facilitate the sharing of projects between instances and raise more relevant warnings.

fsergot

@Turribeach
, that would indeed be an option. Note however that it would require more work on the product side to add a feature to activate a plugin by project, not counting the impact on existing usage.

I have added the idea on the original request in our backlog!

As a side note, we hope to ship a workaround for this on bundle deployment in a coming version with the ability to disable this deployment warning for non-admin users (who can't do much about it) but still keep it in the logs & for the admin.

Turribeach

Ummmm I guess that comes from this idea which is just trying to hide the problem away. If I am honest I will say that I don't really like the solution. As an Administrator what do I do? Do I enable the "disable this deployment warning for non-admin users" option so my users don't see what are most likely false alerts when deploying projects to the Automation Node but risk missing an important dependancy is some cases causing a failed pipeline run in Production? It's a tough choice and one that we will probably error on the side of caution so we will not be using this new feature. I understand that fixing this properly will take more work but it's the right thing to do when you consider that all other external project artifacts like connections, code environments, etc are traceable within the project itself. So plugins should be no different.

If you gave the choice of having the "disable this deployment warning for non-admin users" option or an option of knowing when plugins are used 99% (*) of the time (as you said it's hard to have a 100% certainty) then I will take the latter every day, as I can then deal with the 1% as an exception. The 99% option will still be much safer than the "disable this deployment warning for non-admin users" option which will basically cause failures for every project that uses a plugin until the plugin is installed.

(*) Obviously I don't know exactly in what scenarios it's difficult to know a plugin is used but I presume these are edge cases and the majority of the use cases are covered.

ClaudiusH

FYI: We have merged the idea earlier submitted by Tom which was referenced in the comments into this one as the solution to be worked out should tackle both.

tgb417

@ClaudiusH
,

This post is actually a subset of my original post. The problem that I was pointing to is I was getting errors about components that were not used in my project. Plug-ins were only one part. There were confusing bits about connections and other stuff. My original product suggestion was partially about don’t tell me things about components that have not relationship to the current project. And tell me these things in such a way that an analyst not fully conversant in the total architecture might be successful in importing a project from one design node to the next. Versions since 2020 have made some helpful changes. However, my comments are much broader than just plugins.

ClaudiusH

Thanks for pointing out the scope. I updated the idea title to reflect the review of warnings on both import or automation node deployment for relevancy

Tanguy

FYI a post related to this thread (but dealing with the topic of errors):

• https://community.dataiku.com/t5/Product-Ideas/Improve-error-message-when-deploying-updating-projects-in/idi-p/29906

Turribeach

Looks this was implemented and they have not told us!

https://doc.dataiku.com/dss/latest/release_notes/11.html#id21

Seems to be available since v11.3.0. Find it under Admin > Settings > Other > Deployer > Warn on plugin discrepancy

Note the warning on this option: Only the plugin difference between the plugins used in the project and those installed will trigger a warning when importing a project or deploying it from another node rather than the difference between all plugins in the instances (Experimental - Some rare uses of plugins may not be pointed) - (has a best-effort (Tier 2) support)

Personally I rather enable it and get some projects fail for some obscure plugin use than having false warnings all the time. So happy days!

fsergot

Hello all,

My bad, I completely forgot to update this idea.

We now check the exact usage of the plugin in the project. This allows more targeted warnings and will avoid polluting users with test/unused plugins.

As pointed out, the option needs to be activated on Automation node (and not on the Deployer node) in Administration > Deployer section - as it also applies to bundles manually imported on Automation nodes (although we tend to discourage this practice as much as possible).

fsergot

Hello dear contributors,

I am coming back to this and was wondering if you had a chance to look at the new option and have any feedback? There were also various comments on side topics, I'd be curious to hear if some are still valid concerns (that we may address as a separate request).

Thanks for your time