Running unit tests and dealing with project paths

rnorm

Hi all !

I'm fairly new when it comes to unit testing, but I've started reading more and more and creating my own test for a specific Dataiku project with a fairly well-furnished custom python lib.

As our project is hosted on a distant repo, I'd like to be able to implement a way for the tests of lib/tests/ to run everytime we merge a branch. That requires the remote VM to be able to run the tests, so I'm working on a framework to mock the dataiku API calls we use in our scripts.

One thing I'm struggling with though is how to deal with the paths of the lib modules. When I'm inside DSS, the lib is loaded and act as root directory of all modules. So the following statement works:

from foo.bar import fun

But outside of DSS, say on my local machine, this will throw me a ModuleNotFoundError and the correct way to load the above would be to do the following :

from lib.python.foo.bar import fun

Is there a way to implement in my tests some kind of patch to how the root path behaves so that the first statement will always work no matter what ?

Hope that was clear enough

Cheers,
Pierre.

---

Operating system used: Win10

Sarina

Hi @rnorm
,

The difference is likely due to differences in your PYTHONPATH environment variable while inside DSS compared to while outside of DSS. From inside DSS, your PYTHONPATH variable will include the path to <DATADIR>/lib/python, which is why you don't need to include "lib.python" within DSS. You can see this more clearly if you run the following within a notebook in DSS

import os 
os.environ['PYTHONPATH']

For example, with a setup similar to yours, you'll see that the PYTHONPATH os variable includes the full path to <DATADIR>/lib/python within DSS:

If you go ahead and add the full path to <DATADIR>/lib/python to your PYTHONPATH variable (via the export line), then you should then be able to import the packages the same way, outside of DSS. For example:

ssinick@DKU-MBP-ssinick ~/Downloads/dataiku-versions/dataiku-dss-10.0.5/lib/python/foo (master) $ export PYTHONPATH=$PYTHONPATH:/Users/ssinick/Downloads/dataiku-versions/dataiku-dss-10.0.5/lib/python

ssinick@DKU-MBP-ssinick ~/Downloads/dataiku-versions/dataiku-dss-10.0.5/lib/python/foo (master) $ /usr/bin/python3
Python 3.8.2 (default, Oct  2 2020, 10:45:42) 
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from foo.bar import some_function
>>> some_function()
maybe i'm testing something

So you should just need to set the PYTHONPATH prior to running your tests, to allow for the same behavior in and outside of DSS.

Let me know if you have any questions about that!

Thank you,
Sarina