Tab to auto-complete isn't working with created code envs?

[ak12]

Hi,

When writing code I am able to use tab to auto-complete when I am using the default Python 2 environment in DSS, but when I change the kernel of my notebook to be any of the python envs I have created I lose the ability to use tab to auto-complete?

I have looked in the setting and the documentation and I cannot see any way of making sure it is enabled. If it helps I am on Dataiku version 7

Thank you

[sergeyd · Accepted solution]

Hi @ak12 

We've seen this before. It is likely that your impersonated end-user does not have a valid home directory. The mechanism that IPython uses to perform the tab autocompletion when running python 3 makes use of a disk-based cache. Such disk-based cache requires a valid home directory.

Generally speaking, we strongly recommend that every impersonated end-user have a valid and writable home directory. Failure to have this is known to cause a number of issues such as this one, alongside with Jupyter exports, matplotlib, xgboost, and others.

[tgb417]

@ak12 

+1,  I think that I've seen the same thing.

@sergeyd 

I'm not clear that I know exactly what you mean by an impersonated user. 

That leaves me unclear on how to resolve this issue on a local Macintosh design node install?

What needs to be done to resolve this:

1. The User name in DSS security must match the name of the short macintosh login name? (Same as login directory)
2. Or does one have to create a Dataiku account on the computer?
3. Or something else...

For the community edition where you can not change the login name is there a way to resolve these issues?  Thanks for your help.

[sergeyd]

Hi @tgb417, @ak12 

OK, so impersonated user is the Unix user that runs the jupyter notebook when UIF is enabled: 

https://doc.dataiku.com/dss/latest/user-isolation/index.html

While there is still a requirement to have a valid home directory for the user if UIF is enabled to make autocomplete working, there is also one thing that breaks the autocompletion feature in Jupyter notebooks. 

The built-in code-env contains hard-coded ipython==5.5.0 (ipython>=5.5,<5.6) package while managed code-envs don't have that limitation and install the latest available for corresponding Python version:

- for Python3.6 -> 7.16.0, 

- for Python3.5 -> 7.9.0

- for Python2.7 -> 5.10.0

I haven't checked the latest versions of this package for all Python versions, but installing ipython==5.5.0 to the code-env solves the autocomplete issue. 

Please note, you will need to reload the kernel after installing the new package. 

[tgb417]

@sergeyd ,

Based on your instructions I made the following changes to one of the code environments I use with Jupyter Notebooks.  Is this what you had intended?  I do have a tab-based auto-complete.

The screen used to setup an DSS Python Code environment. In this case for a Python 3.6 environment. Highlighting the inclusion of a request packages python line " ipython>=7.16.0 ". With a speech cloud that says "Added this line. Then Updated"

Is there a way to get an auto-complete popup like MS Intelesense to come up?

[sergeyd]

Hi @tgb417 

I mentioned v7.16.0 as the version that is being installed by default for Python3.6 code envs. During my tests, autocomplete and autosuggestion wasn't working with this ipython version so I installed 5.5.0 (the same one as we have in built-in code env) and get both features working as expected: 

[tgb417]

@Sergey 

On the first attempt, I was not able to downgrade to this version.  I ended up with some sort of problem with the fbprophet library that was previously installed in this code environment.

 ERROR: Failed building wheel for fbprophet

And ended up with some other problem that kept me from Python all together.  Not clear why.  And I now have a broken code environment.  I don't have time to sort this out at the moment.

However, the experience left me with a question:

That seems like a fairly large downgrade...

Do you have any sense of anything important we might loose in terms of features, or increased security vulnerabilities?

Do you have any sense when more recent version will be usable with DSS?

[sergeyd]

Of course, every code-env has a specific set of packages and one package change can break others. I cannot say that fbprophet is a relatively easy-to-install package. So if you have deps issues with python 5.5.0 you will need to revert back it to 7.16.0 (original version) and your env should return back to normal. Sure, no guarantee that autocomplete feature will work in this case. 

[sergeyd]

@tgb417 

I just run a test to check fbprophet package compatibility with ipython==5.5.0 and had no issues installing it:

From your screenshot, you are using conda while I have used regular pip env. That might be the root cause of the issue for your specific env. 

[tgb417]

The following seems to have worked.  I had to completely rebuild the code environment.

DSS Code environment showing the downgrade of ipython to 5.5 and the rebuild of the environment, and fbprofit being included.

Still testing to see if all is working OK.

[tgb417]

It does appear that one has to hit the tab key to activate.  This does not automatically pop up on the screen as you type like SSMS or VS Code might do.

[sergeyd]

Well 🙂 The topic is called "Tab to autocomplete" so using Tab to activate is expected here. 

[tgb417]

@sergeyd ,

Sorry agreed.  Sorry for cluttering up this thread.

So, the tool that I think I'm thinking about that allows the IntelliSense-like auto-completion seems to be hinterland a nbextensions.  I'm not clear if it is supported at some level by DSS.

And I eventually found this thread as well that seems to reference a way back to auto-complete experience. (w/o tab)

https://community.dataiku.com/t5/Using-Dataiku-DSS/How-to-enable-auto-completion-in-Jupyter-Notebook...

[Suman]

@sergeyd 

We are using conda (only) for DSS 7.0

and by default when creating python3.6 code environment ipython==7.6.1 is getting installed.

How can we include ipython>=5.5,<5.6 part of conda? 

Will it affect any other inbuilt environment?

Thanks & regards,

Suman

[sergeyd]

Hi @Suman 

You should have no issues installing ipython v5.5.0 using conda:

Result: 

Other code envs are not affected. You apply the change only to the one where you install the package. 

[Suman]

Thanks @Sergey 

It worked. 

[sergeyd]

Based on https://github.com/ipython/ipython/issues/12740 to get autocomplete to work with the latest available ipython for Py3.6 (7.16.0) we need to downgrade jedi from 0.18.0 to 0.17.2.

[Suman]

Thanks @sergeyd 

Is there any reference link which can be followed like which version support what.... for example if I want to use Python3.6 with ipython 7.9.0 what version of jedi or any other package need to upgrade/downgrade to make jupyter autocomplete work.

[sergeyd]

I am not aware of such a compatibility matrix. 

[Mattsco]

For anyone still struggling with tab autocomplete inside of Jupyter I add the following magic at the top of my notebook and it fixed the issue :

%config Completer.use_jedi = False