Tab to auto-complete isn't working with created code envs?

Options
ak12
ak12 Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2022, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant Posts: 7 Neuron

Hi,

When writing code I am able to use tab to auto-complete when I am using the default Python 2 environment in DSS, but when I change the kernel of my notebook to be any of the python envs I have created I lose the ability to use tab to auto-complete?

I have looked in the setting and the documentation and I cannot see any way of making sure it is enabled. If it helps I am on Dataiku version 7

Thank you

Tagged:

Best Answer

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Answer ✓
    Options

    Hi @ak12

    We've seen this before. It is likely that your impersonated end-user does not have a valid home directory. The mechanism that IPython uses to perform the tab autocompletion when running python 3 makes use of a disk-based cache. Such disk-based cache requires a valid home directory.
    Generally speaking, we strongly recommend that every impersonated end-user have a valid and writable home directory. Failure to have this is known to cause a number of issues such as this one, alongside with Jupyter exports, matplotlib, xgboost, and others.

Answers

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @ak12

    +1, I think that I've seen the same thing.

    @sergeyd

    I'm not clear that I know exactly what you mean by an impersonated user.

    That leaves me unclear on how to resolve this issue on a local Macintosh design node install?

    What needs to be done to resolve this:

    1. The User name in DSS security must match the name of the short macintosh login name? (Same as login directory)
    2. Or does one have to create a Dataiku account on the computer?
    3. Or something else...

    For the community edition where you can not change the login name is there a way to resolve these issues? Thanks for your help.

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Options

    Hi @tgb417
    , @ak12

    OK, so impersonated user is the Unix user that runs the jupyter notebook when UIF is enabled:

    https://doc.dataiku.com/dss/latest/user-isolation/index.html

    While there is still a requirement to have a valid home directory for the user if UIF is enabled to make autocomplete working, there is also one thing that breaks the autocompletion feature in Jupyter notebooks.
    The built-in code-env contains hard-coded ipython==5.5.0 (ipython>=5.5,<5.6) package while managed code-envs don't have that limitation and install the latest available for corresponding Python version:
    - for Python3.6 -> 7.16.0,
    - for Python3.5 -> 7.9.0
    - for Python2.7 -> 5.10.0
    I haven't checked the latest versions of this package for all Python versions, but installing ipython==5.5.0 to the code-env solves the autocomplete issue.
    Please note, you will need to reload the kernel after installing the new package.
  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @sergeyd
    ,

    Based on your instructions I made the following changes to one of the code environments I use with Jupyter Notebooks. Is this what you had intended? I do have a tab-based auto-complete.

    iPython Library.jpg

    Is there a way to get an auto-complete popup like MS Intelesense to come up?

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Options

    Hi @tgb417

    I mentioned v7.16.0 as the version that is being installed by default for Python3.6 code envs. During my tests, autocomplete and autosuggestion wasn't working with this ipython version so I installed 5.5.0 (the same one as we have in built-in code env) and get both features working as expected:

    Screenshot 2021-02-13 at 23.36.11.png

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    edited July 17
    Options

    @Sergey

    On the first attempt, I was not able to downgrade to this version. I ended up with some sort of problem with the fbprophet library that was previously installed in this code environment.

     ERROR: Failed building wheel for fbprophet

    And ended up with some other problem that kept me from Python all together. Not clear why. And I now have a broken code environment. I don't have time to sort this out at the moment.

    However, the experience left me with a question:

    That seems like a fairly large downgrade...

    Do you have any sense of anything important we might loose in terms of features, or increased security vulnerabilities?

    Do you have any sense when more recent version will be usable with DSS?

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Options

    Of course, every code-env has a specific set of packages and one package change can break others. I cannot say that fbprophet is a relatively easy-to-install package. So if you have deps issues with python 5.5.0 you will need to revert back it to 7.16.0 (original version) and your env should return back to normal. Sure, no guarantee that autocomplete feature will work in this case.

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    The following seems to have worked. I had to completely rebuild the code environment.

    iPython Library V2.jpg

    Still testing to see if all is working OK.

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Options

    @tgb417

    I just run a test to check fbprophet package compatibility with ipython==5.5.0 and had no issues installing it:

    Screenshot 2021-02-14 at 00.08.35.png

    From your screenshot, you are using conda while I have used regular pip env. That might be the root cause of the issue for your specific env.

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    It does appear that one has to hit the tab key to activate. This does not automatically pop up on the screen as you type like SSMS or VS Code might do.

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Options

    Well The topic is called "Tab to autocomplete" so using Tab to activate is expected here.

  • Suman
    Suman Partner, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered Posts: 8 Partner
    Options

    @sergeyd

    We are using conda (only) for DSS 7.0

    and by default when creating python3.6 code environment ipython==7.6.1 is getting installed.

    How can we include ipython>=5.5,<5.6 part of conda?

    Will it affect any other inbuilt environment?

    Thanks & regards,

    Suman

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Options

    Hi @Suman

    You should have no issues installing ipython v5.5.0 using conda:

    Screenshot 2021-02-18 at 20.18.39.png

    Result:

    Screenshot 2021-02-18 at 20.18.48.png

    Other code envs are not affected. You apply the change only to the one where you install the package.

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @sergeyd
    ,

    Sorry agreed. Sorry for cluttering up this thread.

    So, the tool that I think I'm thinking about that allows the IntelliSense-like auto-completion seems to be hinterland a nbextensions. I'm not clear if it is supported at some level by DSS.

    And I eventually found this thread as well that seems to reference a way back to auto-complete experience. (w/o tab)

    https://community.dataiku.com/t5/Using-Dataiku-DSS/How-to-enable-auto-completion-in-Jupyter-Notebook/m-p/2436

  • Suman
    Suman Partner, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered Posts: 8 Partner
    Options
  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Options

    Based on https://github.com/ipython/ipython/issues/12740 to get autocomplete to work with the latest available ipython for Py3.6 (7.16.0) we need to downgrade jedi from 0.18.0 to 0.17.2.

  • Suman
    Suman Partner, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered Posts: 8 Partner
    Options

    Thanks @sergeyd

    Is there any reference link which can be followed like which version support what.... for example if I want to use Python3.6 with ipython 7.9.0 what version of jedi or any other package need to upgrade/downgrade to make jupyter autocomplete work.

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Options

    I am not aware of such a compatibility matrix.

  • Mattsco
    Mattsco Administrator, Dataiker Posts: 125 Administrator
    Options
    For anyone still struggling with tab autocomplete inside of Jupyter I add the following magic at the top of my notebook and it fixed the issue :
    %config Completer.use_jedi = False
Setup Info
    Tags
      Help me…