Running unit tests and dealing with project paths

Options
rnorm
rnorm Registered Posts: 9 ✭✭✭✭
edited July 16 in Using Dataiku

Hi all !

I'm fairly new when it comes to unit testing, but I've started reading more and more and creating my own test for a specific Dataiku project with a fairly well-furnished custom python lib.

As our project is hosted on a distant repo, I'd like to be able to implement a way for the tests of lib/tests/ to run everytime we merge a branch. That requires the remote VM to be able to run the tests, so I'm working on a framework to mock the dataiku API calls we use in our scripts.

One thing I'm struggling with though is how to deal with the paths of the lib modules. When I'm inside DSS, the lib is loaded and act as root directory of all modules. So the following statement works:

from foo.bar import fun

But outside of DSS, say on my local machine, this will throw me a ModuleNotFoundError and the correct way to load the above would be to do the following :

from lib.python.foo.bar import fun

Is there a way to implement in my tests some kind of patch to how the root path behaves so that the first statement will always work no matter what ?

Hope that was clear enough

Cheers,
Pierre.


Operating system used: Win10

Best Answer

  • Sarina
    Sarina Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 315 Dataiker
    edited July 17 Answer ✓
    Options

    Hi @rnorm
    ,

    The difference is likely due to differences in your PYTHONPATH environment variable while inside DSS compared to while outside of DSS. From inside DSS, your PYTHONPATH variable will include the path to <DATADIR>/lib/python, which is why you don't need to include "lib.python" within DSS. You can see this more clearly if you run the following within a notebook in DSS

    import os 
    os.environ['PYTHONPATH']


    For example, with a setup similar to yours, you'll see that the PYTHONPATH os variable includes the full path to <DATADIR>/lib/python within DSS:

    Screen Shot 2022-04-06 at 5.25.42 PM.png

    If you go ahead and add the full path to <DATADIR>/lib/python to your PYTHONPATH variable (via the export line), then you should then be able to import the packages the same way, outside of DSS. For example:

    ssinick@DKU-MBP-ssinick ~/Downloads/dataiku-versions/dataiku-dss-10.0.5/lib/python/foo (master) $ export PYTHONPATH=$PYTHONPATH:/Users/ssinick/Downloads/dataiku-versions/dataiku-dss-10.0.5/lib/python
    
    ssinick@DKU-MBP-ssinick ~/Downloads/dataiku-versions/dataiku-dss-10.0.5/lib/python/foo (master) $ /usr/bin/python3
    Python 3.8.2 (default, Oct  2 2020, 10:45:42) 
    [Clang 12.0.0 (clang-1200.0.32.27)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from foo.bar import some_function
    >>> some_function()
    maybe i'm testing something
    


    So you should just need to set the PYTHONPATH prior to running your tests, to allow for the same behavior in and outside of DSS.

    Let me know if you have any questions about that!

    Thank you,
    Sarina

Setup Info
    Tags
      Help me…