Dashboards & Visualizations

how to define stop words in splunk

mhqssyh
Explorer

Hi, I am doing some text mining research by splunk. Though the popular words can be detected, there are so much stop words, such as the, a, you, I, which means mining is nonsense for me. Does anybody know how to handle stop words in splunk?

Tags (2)
0 Karma
1 Solution

jimodonald
Contributor

Chapter 12 of "Big Data Analytics Using Splunk" covers the details of what you are looking to do. In the book, the author is analyzing a Twitter feed.

Some quick outtakes:
Most popular words (page 220)

* | head 200 | makemv text | mvexpand text | top text

and filtering out the stop words (page 221)

lang=en | makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

Hope that points you in the right direction.

View solution in original post

yserrano
New Member

My case example:

eval text=lower(NOTES)|
eval text=urldecode(text)|
makemv text |
mvexpand text |top limit=50000 text|
search NOT ( [ |inputlookup StopWords_inc.csv|rename Word as text ] )|
head 100

PD: The NOTES field is used in my work to record Ticket data (service center)

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If there's a question hiding in there please post it as a separate question.

0 Karma

jimodonald
Contributor

Chapter 12 of "Big Data Analytics Using Splunk" covers the details of what you are looking to do. In the book, the author is analyzing a Twitter feed.

Some quick outtakes:
Most popular words (page 220)

* | head 200 | makemv text | mvexpand text | top text

and filtering out the stop words (page 221)

lang=en | makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

Hope that points you in the right direction.

mhqssyh
Explorer

OK, Thank u. Then, I have another question how to search some word from both field Word in word.csv and field text.
I tried follow search command, but it turned out wrong

search text= [|inputlookup word.csv| fields word]
search text AND[|inputlookup word.csv| fields word]
0 Karma

lavanya_gurrapu
New Member

Hi,
makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

is not working for me. I am getting no results. pls help.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...