Are user-defined meanings strictly global?
UserBird
Dataiker, Alpha Tester Posts: 535 Dataiker
As far as I can tell, user-defined meanings are global i.e. if a meaning is created for use in one project useful, but I also could see this being cumbersome if projects have a lot of custom meanings.
Our use case is we are reading primarily from an external system via a REST API. I have been using a custom plugin to read data from the other system. I like that I can set the schema via a plugin because the other system allows metadata to be queried. That metadata includes data value mappings (e.g. 1 - yes, 0 - no, 2 - not sure, -1 - N/A). I have been trying to come up with a good way of importing those mappings into a project when a dataset is imported. The values-mapping type of user-defined meanings are a very useful concept because they separate storage type from the labels (usually ints to string) and allow someone viewing the data to easily see what number means what.
I imagine us having many projects with different users having access to each project. I could keep using meanings and use prefixes to distinguish them (although I could see the odd case where having the meanings visible to the wrong users might be a problem). I could also import the value mappings as datasets and use joins to fill in the values but this is not very automated and will make the projects quite messy.
Any tips? Anyone else looking into a similar use-case?
Our use case is we are reading primarily from an external system via a REST API. I have been using a custom plugin to read data from the other system. I like that I can set the schema via a plugin because the other system allows metadata to be queried. That metadata includes data value mappings (e.g. 1 - yes, 0 - no, 2 - not sure, -1 - N/A). I have been trying to come up with a good way of importing those mappings into a project when a dataset is imported. The values-mapping type of user-defined meanings are a very useful concept because they separate storage type from the labels (usually ints to string) and allow someone viewing the data to easily see what number means what.
I imagine us having many projects with different users having access to each project. I could keep using meanings and use prefixes to distinguish them (although I could see the odd case where having the meanings visible to the wrong users might be a problem). I could also import the value mappings as datasets and use joins to fill in the values but this is not very automated and will make the projects quite messy.
Any tips? Anyone else looking into a similar use-case?
Tagged:
Answers
-
Hi,
At the moment, UDMs are indeed only global. We are thinking about the possibility to add per-project-UDMs but do not have a timeframe for that yet.
From your description, couldn't you perform the replacement of values directly in your plugin ? Read the metadata mapping and then read the data and transform it before returning from the plugin code ? -
I think that is a reasonable work-around. Ideally, I would like to use functionality that is meant for what I am trying to accomplish. UDMs are appealing because they explicitly represent the idea of a key-value mapping for a datatype.
-
importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuronbumping this oldie to just say project level or even user level meanings would be wonderful. they have proven fantastic for validation of upload files and our user base is all over them - global meanings can get out of hand quickly!
-
I am a new Dataiku user and was searching for ways to create non-global UDMs. If I create a UDM, is it always seen globally or configurable to restrict at project-level?
