Greetings fellow Linguists,
You can start by installing NLTK (Natural Language Toolkit) as any other Python package in DSS, by creating a code environment and adding "nltk" to your package requirements. To do so, follow this documentation.
However, some functionalities of NLTK such as text corpora and language-specific models rely on resources which are not bundled in the library itself. The full list of available resources is available here.
To use these models, you need an additional download step. Typically, this can create issues on shared DSS nodes where users do not have write access to shared locations on the server (see User Isolation Framework).
1. Download NLTK Data for all users (recommended)
pip install nltk sudo python -m nltk.downloader -d /usr/share/nltk_data all
from nltk.corpus import brown print(brown.words())
python -m nltk.downloader -d /home/<yourLinuxUserName>/nltk_data all
import os os.environ['NLTK_DATA'] = /home/<yourLinuxUserName/nltk_data
How can I download the NLTK Data if I install the 'nltk' package in a (Dataiku controlled) virtual environment? If I just use the 'sudo python -m nltk.download ...' from the command-line, the nltk-package is not found.
The procedure highlighted above is indeed designed for use in Dataiku-managed code environment. The command you used is correct (see NLTK doc). Have you checked that `python` is not pointing to Python 2? NLTK dropped its support of Python 2 recently.
If that does not work, we will need the full output of your command to diagnose.
the problem also occurs when I run the command (in a terminal) with python3; I get an error: (ModuleNotFoundError: No module named 'nltk'). So, I think I need to run the command in the Dataiku code environment in which I installed the nltk. How can I do that? Should I just navigate in a terminal to the folder containing the code environment and run the command?
Thanks, that's clearer. You are just missing the nltk package in your main Python environment.
Can you try:
- pip install nltk
- sudo python -m nltk.downloader -d /usr/local/share/nltk_data all
No need to navigate to the Dataiku code-env folder. It's a central installation, it should be picked up by the code-env automatically.
If that doesn't work, please send us the full output of the commands you executed.
I missed the installation of nltk in the main Python environment. That indeed solved the problem.
No problem, I will add that to the original article for completeness.
I installed NLTK using pip in code evnu, while using wordnet in python recipe its pointing to different location. so I am getting wordnet not installed
How to change path in python recipe so that it can point to location where it downloaded.
Hi @gmanoj_reddy ,
Have you applied the step "1. Download NLTK Data for all users (recommended)" from the article?