I have some custom Data Quality checks added to Dataiku as part of a plugin. I want to have an input variable as a multi-select dropdown list of the columns in the schema.

If I use "type": "DATASET_COLUMN", I get a dropdown list but no multi-select.

If I use "type": "DATASET_COLUMNS", I get the ability to add mutliple values but not from a list of the columns in my dataset.

Using DATASET_COLUMNS works but is a bit clunky as you have to have your dataset open in a separate window to remember the column names.

I think I can use getChoicesFromPython but the structure of the objects passed in do(payload, config, plugin_config, inputs) is unclear for a Data Quality check.

Please let me know what's best to do here!

Thanks

Dataiku version used: 14.3

"Backend died before startup complete, caused by: APIErrorException: No module named 'pydantic'"

i've download it in the default environment and configure the project code environment to be the same one, but the issue still appears and the code environment of agent hub webapp doesn't appear for me, inorder to change the environment setting that used internally.

Dataiku version used: Trial version on cloud

I'm working on cleaning up my Dataiku instance before a version upgrade.

When I use notebooks, I have a very large number of available kernels, many of which are obsolete.

Do you know where I can clean up this list?

Thank you.

Dataiku version used: 13.2.4

A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.

In my Dataiku UI, I created two Python environments under Code Envs:

• One for Python 3.12
• One for Python 3.9

Each environment installs the Python libraries listed in “packages to install” by retrieving them from repositories hosted on an Artifactory.

I imported the Artifactory certificate into the keystore of my Dataiku VM to allow communication.

In the extra options field, I specified --index-url along with the Artifactory token created for Dataiku and the repository name: one for Python 3.9, one for Python 3.12.

With the Python 3.12 environment, everything works correctly: I can rebuild the environment and it retrieves and installs the Python libraries as expected.

However, with the Python 3.9 environment I get a certificate error:

WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)'))': /artifactory/api/pypi/python-libraries-dataiku-3.9/simple/pip/

If I add --trusted-host <my-artifactory-address> in the extra options, it works.
However, I would like to avoid relying on this approach and instead configure the Python 3.9 environment in a secure way, similar to the Python 3.12 environment that works correctly.

Operating system used: Alma 9.4

Operating system used: Alma 9.4

We find our self modifying the base DSS image to accommodate some features needed in an R code environment.

And I don't know how to deal with this error:

Dev server deployment FAILED
Failed to initiate function server : <class 'Exception'> : Default project key is not specified (no DKU_CURRENT_PROJECT_KEY in env)

Can anybody please help me in solving problem " How to install Text Preparation Plugin step by step ?" including installing dependency modules ,Profibus compiler, Regex Module in Python environment. I am using Windows10 with Dataiku 13 Free Edition .

I am repeatedly failed in installing TP. when installed it was showing "Plugin Development " on Plugin page ,but didn't worked in Design Flow. I am getting below errors

Operating system used: windows 10

However, after investigating with Dataiku Support, I learned that this option is currently restricted to admins. As a result, even if a non-admin user has the necessary permissions to update a Code Studio template and its associated code environment, they cannot use this feature. Instead, they must update the code environment first and then rebuild the Code Studio templates separately, requiring two steps.

According to Dataiku Support, this limitation exists because non-admin users cannot view Code Studio template usage via the "Compute Usage" section in the code environment.

I struggle to understand why this functionality is restricted to admins. It seems logical to extend this capability to non-admin users who have the appropriate permissions to update both a code environment and a Code Studio template linked to it.

I am facing below issue for PySpark recipe.

Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

I have set the environment variables using os.environ() by providing a file path but somehow this approach only works if I run the code via Jupyter but will not work when the same run via recipe.

So i did some digging and found below community post:

PySpark Python executables — Dataiku Community

Based on above, can this issue be resolved if I set the python binary path in the code environment setting section.

(Unfortunately, I cant upload a picture so let me write it down how it looks like)

Spark

Yarn Python executable [ ] Python binary on the Yarn nodes for Pyspark (save, remove then re-install jupyter support to update in notebooks)

Operating system used: Windows 10

Operating system used: Windows 10

In our organization, we were initially excited about the feature introduced in DSS v11. However, our enthusiasm was quickly dampened by the fact that users cannot select the associated code env running behind each code studio. While we could ask our admins to build a new template with a specific version of python, we face two major issues: 1) we already have several hundred code envs on our Design instance, and 2) The Python paths used by code studios cannot be linked to our existing code envs (which users can regularly update). This situation prevents the code studio from adapting to the specific needs of the users. Although we can install packages in code studios, we can't change the Python version. As a result, no one uses them, even though we hope to utilize the new IDE and web app tools.

I am eager to hear feedback and experiences from the community.

Operating system used: RHL

Recently, we faced an issue with one of our automation nodes. New deployments were failing because there was no space left on disk. Upon investigation, we discovered that a code environment was occupying over 100 GB due to multiple versions linked to the bundles previously deployed on the instance.

After deleting some bundles, we expected the obsolete versions of the code environment to be removed automatically. However, this was not the case.

This feature proposal suggests enabling DSS to automatically delete obsolete versions of a code environment when the associated bundles are removed.

My group has dozens of code environments and use various Python packages. It would be great if Dataiku had a dedicated module that would allow the user to specify a python package and get the current version in each code environment for the code environments I have access to.

Second, it would be great if I could push a change to a package version to all code environments I have write access to. When we change python versions, it can take a couple hours to manually edit and rebuild every code environment. I would love for Dataiku to handle this in a batch job process and alert me when all are done or alert me to which ones failed.

thx

Importing serialized scikit-learn pipelines as Saved Models for MLOps - Dataiku Developer Guide

and I've been able to develop a model using the darts==0.30.0 library, having wrapped it in the standard scikit-learn pipeline. My issue is with the very last part of step 3 of this tutorial where I want to deploy_run_model but I'm unable to because my environment was missing the Visual ML packages. When I added those packages to the code env, there are many issues with mis-matching dependencies wrt the darts package, so I attempted to downgrade the darts lib but now am stuck waiting for a 2 hr environment update.

Is there any plan to update the versions of standard Visual ML packages?

Operating system used: Windows 11

I can't install the python library osgeo in a code env.

× Encountered error while trying to install package.
╰─> osgeo
note: This is an issue with the package mentioned above, not pip.

Do you have any solutions ?

Figure 1 - Python interpreter status in mid 2024

Yet, we observed that Dataiku allows upgrades of code envs that are labelled 'deprecated'.

Figure 2 - DSS provides a variety of Python interpreters, including those labeled as 'deprecated'. For these versions, DSS issues alerts indicating their potential removal in upcoming versions.

Figure 3 - For code environments utilizing a deprecated Python interpreter, DSS provides users with the option to perform an upgrade

Thus, our inquiry is whether an admin can modify the status of Python versions to 'deprecated', which would

1. inform users of a potential removal of this Python version in the ensuing DSS version
2. grant users the ability to upgrade their Python interpreter (as it appears that the 'deprecated' tag activates the 'change interpreter' button)

Is this possible?

Operating system used: RHL 8

I am just starting to use Python 3.9 and I am getting the following warnings when I import the dataiku package:

distutils Version classes are deprecated. Use packaging.version instead.
invalid escape sequence \*
invalid escape sequence \*
invalid escape sequence \s
invalid escape sequence \:
invalid escape sequence \:

Any suggestions on how to remove these warnings?

I know that warnings.filterwarnings() will suppress the display of the distutils warning (but not the others).

I'm interested in determining whether I can configure the code environment or something like that so no warnings are generated.

I get that these warnings are not a problem in terms of functionality. They will however create questions and diminish confidence when we roll this out to our user base. That's why I'd like to try to stop these from occurring.

Thanks,

Marlan

Operating system used: RHEL

I'm having a problem importing the "dataiku" package in python in the following environments since update to 11.2

python 3.6

python 3.7

python 3.9

# Example: load a DSS dataset as a Pandas dataframe
mydataset = dataiku.Dataset("PWHRn6UM")

import dataiku

/data1/dss/dataiku-dss-11.2.0/python/dataiku/core/schema_handling.py:17: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.  (This may have returned Python scalars in past versions.
  'boolean': np.bool

Appears at first to be a soft warning but any attempt to use the dataiku package fails

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-3-4bdc522e44af> in <cell line: 2>()
      1 # Example: load a DSS dataset as a Pandas dataframe
----> 2 mydataset = dataiku.Dataset("PWHRn6UM")

NameError: name 'dataiku' is not defined

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-3acb3cd0f9dd> in <cell line: 1>()
----> 1 import dataiku

/data1/dss/dataiku-dss-11.2.0/python/dataiku/__init__.py in <module>
     11 from .base import remoterun
     12 from .core.base import is_container_exec
---> 13 from .core.dataset import Dataset, _dataset_writer_atexit_handler
     14 from .core.schema_handling import get_schema_from_df
     15 

/data1/dss/dataiku-dss-11.2.0/python/dataiku/core/dataset.py in <module>
     33 
     34 # Module code
---> 35 from dataiku.core import flow, base, schema_handling, dkuio
     36 from dataiku.core.platform_exec import read_dku_json
     37 from dataiku.core.dkujson import dump_to_filepath, load_from_filepath

/data1/dss/dataiku-dss-11.2.0/python/dataiku/core/schema_handling.py in <module>
     15     'float': np.float32,
     16     'double': np.float64,
---> 17     'boolean': np.bool
     18 }
     19 

/data1/dss/dss_data/code-envs/python/GC_dbs_39/lib64/python3.9/site-packages/numpy/__init__.py in __getattr__(attr)
    282             return Tester
    283 
--> 284         raise AttributeError("module {!r} has no attribute "    285                              "{!r}".format(__name__, attr))
    286 

AttributeError: module 'numpy' has no attribute 'bool'

.

Operating system used: Windows

Operating system used: Windows

I am aiming at running Jupyter notebooks with STATA interpretation capability. I wonder if there has been any similar efforts so far. It would be great to give me some hints.

Useful links:
https://www.stata.com/python/pystata18/

https://github.com/ticoneva/pystata-kernel

I could install STATA on the DSS VM. However, I could not run the examples of STATA code it in the Jupyter notebook:

%%stata

UsageError: Cell magic `%%stata` not found.

How would you consider the installation directory of STATA? And how to resolve Cell magic error, assuming its a IPython kernel issue?

If new environment is needed, how do you recommend creating a new environment? maybe by conda?!

Much appreciate it!

Cheers, Farshad

Operating system used: Azure

I set a Python env in order to install specific packages, but the following recipe is failing when executed in such an env:

import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
df_train_asis = dataiku.Dataset("df_train_asis")
df_train_asis_df = df_train_asis.get_dataframe()
df_test_asis = dataiku.Dataset("df_test_asis")
df_test_asis_df = df_test_asis.get_dataframe()


# Compute recipe outputs
# TODO: Write here your actual code that computes the outputs
# NB: DSS supports several kinds of APIs for reading and writing data. Please see doc.

df_results_asis_df = df_train_asis_df # Compute a Pandas dataframe to write into df_results_asis


# Write recipe outputs
df_results_asis = dataiku.Dataset("df_results_asis")
df_results_asis.write_with_schema(df_results_asis_df)

The error message:

Job failed: The Python process failed (exit code: 1)

Error type:com.dataiku.dip.exceptions.ProcessDiedException

The script works fine when run in the builtin DSS env. Any idea?

Operating system used: MacOS

Operating system used: MacOS

TL;DR

I want to use this model llama-2-7b.ggmlv3.q4_0.bin inside a code env, to do this I am using python port llama-cpp-python

My problem is that I need to compile llama.cpp and install shared lib. How can I do this?

PD: This code env will be containerized

Saul

I am having some trouble setting Dataiku code environments with any Python other than 2.7 & 3.6.

Further more, I can't seem to get the "Use Conda" option either.
I know anaconda is considered a tier 2 support item, but it's required for my current project.

Does anyone know how to get these features working?

Thank you in advance!

Operating system used: Red Hat Enterprise Linux

Operating system used: Red Hat Enterprise Linux 8.5

I am using the Forecasting Plugin to train a forecasting model for a time series commodities dataset.

However, I have a few issues with regards to setting up a functional code environment for this plugin. In particular, DSS tells me that I need gluonts and mxnet in my environment in order to start the training process.

Therefore I tried adding mxnet-cu100==1.7.0 and gluonts==0.10.4 in the general >> extra options settings of the plugin managed environment but I received this errror:

I also created a new python environment and added the set of packages for forecasting in "packages to install". This worked with gluonts and mxnet now included in the environment.

However, when I now try and dissociate the previous environment in the plugin page, I cannot see this new python environment in the dropdown list (the only ones available are those created and managed by DSS.)

Many thanks for any advice.

When I do, I get the following error:

                               ^
      In file included from /usr/local/include/python3.9/unicodeobject.h:1026,
                       from /usr/local/include/python3.9/Python.h:93,
                       from pmdarima/arima/_arima.c:39:
      /usr/local/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
       Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                   ^~~~~~~~~~~~~~~~~~~
      pmdarima/arima/_arima.c:27634:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                               (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                               ^
      In file included from /usr/local/include/python3.9/unicodeobject.h:1026,
                       from /usr/local/include/python3.9/Python.h:93,
                       from pmdarima/arima/_arima.c:39:
      /usr/local/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
       static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~
      pmdarima/arima/_arima.c:27634:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                               (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                               ^
      In file included from /usr/local/include/python3.9/unicodeobject.h:1026,
                       from /usr/local/include/python3.9/Python.h:93,
                       from pmdarima/arima/_arima.c:39:
      /usr/local/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
       static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~
      pmdarima/arima/_arima.c:27634:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
                               (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                               ^
      In file included from /usr/local/include/python3.9/unicodeobject.h:1026,
                       from /usr/local/include/python3.9/Python.h:93,
                       from pmdarima/arima/_arima.c:39:
      /usr/local/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
       Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
                                                   ^~~~~~~~~~~~~~~~~~~
      pmdarima/arima/_arima.c:27634:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
                               (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 :
                               ^
      In file included from /usr/local/include/python3.9/unicodeobject.h:1026,
                       from /usr/local/include/python3.9/Python.h:93,
                       from pmdarima/arima/_arima.c:39:
      /usr/local/include/python3.9/cpython/unicodeobject.h:446:26: note: declared here
       static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~
      pmdarima/arima/_arima.c: In function ‘__Pyx_decode_c_string’:
      pmdarima/arima/_arima.c:29855:9: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations]
               return PyUnicode_FromUnicode(NULL, 0);
               ^~~~~~
      In file included from /usr/local/include/python3.9/unicodeobject.h:1026,
                       from /usr/local/include/python3.9/Python.h:93,
                       from pmdarima/arima/_arima.c:39:
      /usr/local/include/python3.9/cpython/unicodeobject.h:551:42: note: declared here
       Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode(
                                                ^~~~~~~~~~~~~~~~~~~~~
      error: Command "gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_CBLAS -I/home/dss/dataiku/data/code-envs/python/EGAS_Timeseries/lib/python3.9/site-packages/numpy/core/include -Ipmdarima/arima/_arima_fast_helpers.h -I/home/dss/dataiku/data/code-envs/python/EGAS_Timeseries/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/home/dss/dataiku/data/code-envs/python/EGAS_Timeseries/include -I/usr/local/include/python3.9 -c pmdarima/arima/_arima.c -o build/temp.linux-x86_64-3.9/pmdarima/arima/_arima.o -MMD -MF build/temp.linux-x86_64-3.9/pmdarima/arima/_arima.o.d -msse -msse2 -msse3" failed with exit status 1
      INFO:
      ########### EXT COMPILER OPTIMIZATION ###########
      INFO: Platform      :
        Architecture: x64
        Compiler    : gcc
      
      CPU baseline  :
        Requested   : 'min'
        Enabled     : SSE SSE2 SSE3
        Flags       : -msse -msse2 -msse3
        Extra checks: none
      
      CPU dispatch  :
        Requested   : 'max -xop -fma4'
        Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
        Generated   : none
      INFO: CCompilerOpt.cache_flush[857] : write cache to path -> /tmp/pip-install-0aio0cc8/pmdarima_efe601bcd4004abfbac3fd1bfd109d9d/build/temp.linux-x86_64-3.9/ccompiler_opt_cache_ext.py
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
  Running setup.py install for pmdarima: finished with status 'error'
× Encountered error while trying to install package.
╰─> pmdarima
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Can someone help?

Operating system used: Red Hat 8 VM
Dataiku version: Dataiku 11 Enterprise
Browser used: Chrome

[2018/06/17-18:42:07.394] [MRT-164] [ERROR] [dku.analysis.ml.python]  - Processing failed
com.dataiku.dip.io.SocketBlockLink$SecretKernelTimeoutException: Subprocess failed to connect, it probably crashed at startup. Check the logs.
  at com.dataiku.dip.io.SocketBlockLink.waitForConnection(SocketBlockLink.java:82)
  at com.dataiku.dip.io.SecretProtectedKernelLink.waitForProcess(SecretProtectedKernelLink.java:38)
 at com.dataiku.dip.io.PythonSecretProtectedKernel.start(PythonSecretProtectedKernel.java:84)
  at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java:87)
Caused by: java.net.SocketException: Socket closed
    at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
  at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at com.dataiku.dip.io.SocketBlockLink.waitForConnection(SocketBlockLink.java:78)
  ... 3 more

DatalakeRESTException: HTTP error: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='aicpadlsdev.azuredatalakestore.net', port=443): Max retries exceeded with url: /webhdfs/v1/curated/cornerstone/d2_data/DataEngineeringLab/ADF_User_ConfigFile.csv?OP=GETFILESTATUS&api-version=2018-09-01 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa490a435f8>: Failed to establish a new connection: [Errno -2] Name or service not known',))",),)

For reference I am linking one article which i had referred to make the connection -

https://medium.com/azure-data-lake/using-jupyter-notebooks-and-pandas-with-azure-data-lake-store-48737fbad305

Operating system used: Windows

Operating system used: Windows

Operating system used: Windows

I have the instance installed on a Red Hat 8 VM where I am the admin of the OS with root access and the admin of the Dataiku Instance.

This is on the free version of Dataiku

I go to the dataiku bin folder and start the dssadmin with the option install-julia-integration.
I get the following errors in the attached images.

--------------------------------------------------------------------------------------------------------------------

Submitted GitHub Issue:

https://github.com/dataiku/dss-plugin-julia/issues/2

Operating system used: Red Hat 8 VM