@HarizoR wrote:
Hi,
Do you have examples of functionalities you want to implement for this package?
As you have guessed, project libraries aren't meant to encapsulate Dataiku-specific logic, so it would not be appropriate to import dataiku inside of them. You should instead check if the functionalities you are looking for are covered by the public API client. If not, any code requiring the dataiku package should not live within a project library but directly written into the recipe/notebook you'll work on.
Hope this helps.
Best,
Harizo
I respectfully disagree. I have used project libraries or even the global library before and I found them lacking. The main issue with these is that you can only have a single version active at a single time in a single project. The only solution to this that I am aware is to clone the project which of course is not a good idea as there is no way to merge projects back after a clone. The global library is even more restrictive as there is no way around it: you want a different library version you need a different environment. In complex and large environment setups (ie lots of DSS instances) the last thing you want is to have a component which can not handle multiple versions as the same time which makes your complex environment even more complex.
The original poster however mentioned that wants to work with a package published in Github. I think this will be a great way of handling multiple versions as this package can then be installed on Python code environments. For the reasons I explained above I wouldn't go with libraries. This will allow for multiple live versions to co-exist without any issues since you can obviously have multiple code environments and Dataiku gives you all the tools to handle these with easy. In fact you could even have different package versions running side by side on the same project, try that with project libraries.
As for "encapsulate Dataiku-specific logic, so it would not be appropriate to import dataiku inside them" I see nothing wrong with this. Most Python packages depend on other packages so I don't see what the issue with this is. Every Dataiku Python code environment comes builtin with both the dataiku and the dataikuapi packages installed so it will not be an issue meeting those dependencies.
Finally I woud say that depending on the things you want to automate with your package they might be a better fit for a Dataiku macro or Dataiku plugin. Having said that while Dataiku macros or Dataiku plugins are great ways to encapsulate and automate functionality they also fail in the same way that project and global libraries do: you can only have one version of a plugin or macro at the same time. In the case of macros and plugins you can duplicate them (ie MyMacro v2.0 and MyPlugin v2.0) but the migration of objects is not as easy as a package where you just update a code environment and all your recipes will automatically use the new package.
