Using Dataiku

Thank you for your answer!

I'll try it and let you know if it worked

Thank you for your answer!

I'll try it and let you know if it worked

You can get the information you want using the Dataiku Python API:

https://developer.dataiku.com/latest/concepts-and-examples/flow.html#working-with-flow-zones

You can get the information you want using the Dataiku Python API:

https://developer.dataiku.com/latest/concepts-and-examples/flow.html#working-with-flow-zones

Hi Turribeach, thanks again!

2a. I feel like you aren't hearing my issue here. I'm not asking why the schema tab exists, rather, why are the fields editable. I'm asking you why can you change the types manually if it doesn't do anything. Don't you see how that could be confusing for a user?

2b. I feel like you aren't hearing my issue here. I'm not asking for a workaround so that my NVARCHAR values are never cut off. I'm asking if it's possible to change to the default settings of NVARCHAR schema processing in SQL, since I think many people would agree that setting it to the max string length in the column is not very practical.

This page covers your needs:

https://developer.dataiku.com/latest/tutorials/webapps/common/resources/index.html

There is no magic logic to decide which compute engine will be better for each recipe. Generally speaking Spark should be better for larger datasets but it will depend on many other factors. If using a visual recipe sometimes the Spark translation is not very efficient and DSS engine is better. Sometimes custom Spark code can improve on what DSS does. There is also an overhead to using Spark on small datasets so it might not be worth doing so. The best approach to determine the best engine is to run on both engines and see which one is the fastest. You can multi-select recipes in the flow holding the Ctrl key and change the engine for all of them. You can also use the Recipe Engines flow view to see where each recipe is running. and in v13 you can even combine these two views at the same time to get something like this which shows me the last build duration for all DSS Engine recipes:

So if you were to flip all these to Spark, re-run and then compare the two views and figure which one runs faster than the other.

In v12 there isn't much you can do. But v13 has a new Column-level Data Lineage so another good reason to upgrade:

https://doc.dataiku.com/dss/latest/data-catalog/data-lineage/index.html

After trying everything, none of the parameters really work. The module still checks inside /home/dataiku/ directory and if you don't have access, you will still get an error.

This is a quick not-so-clean fix I made to get things done. You can put it in a function inside libraries to declutter your code.

I understand this is out of the scope of this community, I just hope this helps somebody.

#great-expectations==1.3.5
import pathlib
import great_expectations as gx
from great_expectations.data_context import AbstractDataContext

YOUR_CONF_PATH = pathlib.Path("SET/YOUR/PATH/HERE…") # use a managed folder for example
AbstractDataContext._ROOT_CONF_DIR = YOUR_CONF_PATH / ".great_expectations"

AbstractDataContext._ROOT_CONF_FILE = AbstractDataContext._ROOT_CONF_DIR / "great_expectations.conf"

context = gx.get_context(mode="file")

Hi,
The error suggests you are missing a code env

dataikuapi.utils.DataikuException: java.lang.IllegalArgumentException: No PYTHON code env for Python_39_MLF

Can you try creating or mapping to an existing code when trying to import the project?

Thanks

Hi @samuel_acr_96 ,

If the issue still persists, can you please open support along with the instance diagnostics taken immediately after your tests? The logs may provide more information on the exact exception

Thanks