Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

Removing (209) specific words/strings/sp characters from string column

Solved!
SuzanneD
Level 2
Removing (209) specific words/strings/sp characters from string column

I am preparing a column containing 'Comments/narrative' (string) for a word cloud data set. I have a list of 209 specific words i'd like to remove from the column ('A', 'AN', 'THE', etc).

I'd rather not use the 'find and replace' recipe, for obvious reasons.

Can anyone recommend a more efficient solution?  Thank you!

0 Kudos
1 Solution
MiguelangelC
Dataiker

Hi,

From the example words it seems the focus is on the so-called 'stop-words'. If so, an option would be to use a Prepare recipe and apply the 'Simplify text' processor. Enable the 'Clear stop words' checkbox. There is a tutorial about this in the documentation: https://knowledge.dataiku.com/latest/ml-analytics/nlp/concept-text-data-cleaning.html

If you want to build your own specific solution, there are some Python packages that can be of use, such as NLTK or Gensim.

View solution in original post

7 Replies
MiguelangelC
Dataiker

Hi,

From the example words it seems the focus is on the so-called 'stop-words'. If so, an option would be to use a Prepare recipe and apply the 'Simplify text' processor. Enable the 'Clear stop words' checkbox. There is a tutorial about this in the documentation: https://knowledge.dataiku.com/latest/ml-analytics/nlp/concept-text-data-cleaning.html

If you want to build your own specific solution, there are some Python packages that can be of use, such as NLTK or Gensim.

SuzanneD
Level 2
Author

Thank you! I will give this a try and post back result. I appreciate the response.

0 Kudos
jp1
Level 2

@MiguelangelC  Can I look at those predefined stopwords list? I want to extract them!! Can you pls suggest me on how can I them?

0 Kudos
SuzanneD
Level 2
Author

Hi jp1, they are specific to automobile damages - in addition to the 'stop words' in the DSS prepare recipie.

0 Kudos
jp1
Level 2

I understand, But I want to get the list of stop words which DataIKU prepare recipe processing!! That's what I was asking in the last comment!! 

0 Kudos
SuzanneD
Level 2
Author

So sorry i thought you were asking me : ) 

0 Kudos
jp1
Level 2

If we can look at those stop word list could you pls suggest me on that? 

0 Kudos