Try your hand at analyzing royal sentiment in Dataiku DSS! Learn more

How to use NLTK in DSS

How to use NLTK in DSS

Greetings fellow Linguists,

You can start by installing NLTK (Natural Language Toolkit) as any other Python package in DSS, by creating a code environment and adding "nltk" to your package requirements. To do so, follow this documentation.

However, some functionalities of NLTK such as text corpora and language-specific models rely on resources which are not bundled in the library itself. The full list of available resources is available here.

To use these models, you need an additional download step. Typically, this can create issues on shared DSS nodes where users do not have write access to shared locations on the server (see User Isolation Framework).

1. Download NLTK Data for all users (recommended)

WARNING: this procedure needs command-line access and administrative privileges on the machine hosting DSS. You may need to speak to your DSS admin and/or Linux admin. Assuming you are on a Linux machine and have administrative privileges, run:
sudo python -m nltk.downloader -d /usr/share/nltk_data all
 
For macOS, the path is slightly different: /usr/local/share/nltk_data.
 
To test that it worked correctly, run the following code in a notebook using your code environment with nltk.
from nltk.corpus import brown
print(brown.words())
 
For further details, please refer to this NLTK documentation.
 
2. Download NLTK Data for yourself
 
WARNING: This code will not work for other users if your DSS node is configured with the User Isolation Framework. Run this command without sudo, pointing to your Linux home directory:
python -m nltk.downloader -d /home/<yourLinuxUserName>/nltk_data all
 
In your Python code, you will then need to set the variable "NLTK_DATA" before running code requiring it.
import os
os.environ['NLTK_DATA'] = /home/<yourLinuxUserName/nltk_data
 
Happy natural language processing!
Labels (4)
Version history
Revision #:
6 of 6
Last update:
‎05-18-2020 11:56 AM
Updated by:
 
Contributors