Splunk Search

How to get certain fields from the logs

xvxt006
Contributor

Hi, I am brand new to splunk, sorry if i am asking very basic questions. i have data in the below format (I have put 3 sample requests)

i would like to know how many times each command is being called from the logs in a tabular format. For example from the first request below, i need to extract "search"
and display the count.

For the first request the pattern is - the command will always be preceded by /Company/directory and ends with .shtml.

For the 2nd request the pattern is - always preceded by /typeahead (as i need to capture TypeaheadQueryResponder).

3rd request is SEO url - after /Company i would like to capture till ? (URI).

34.234.42.184 - - [26/Sep/2012:12:01:21 -0500] "GET
/Company/directory/search.shtml?searchQuery=desk+lights&op=search&btr=desk+lights&N=0&GlobalSearch=true HTTP/1.1" 200

237.189.83.254 - - [26/Sep/2012:12:01:21 -0500] "POST /typeahead/TypeaheadQueryResponder HTTP/1.1"
200

55.242.45.133 - - [26/Sep/2012:12:01:21 -0500] "GET
/Company/hand-protection/safety/ironclad/category/werwerre/No-48/WORK+GLOVES?Ner=textsearchesinbase%2Btrue HTTP/1.1"

Can someone help me with this. Thank you for your help in advance.

Tags (2)
0 Karma

lguinn2
Legend

You have several choices:

  • Create fields by editing props.conf. This will create a "permanent" search-time field that everyone can use
  • Use the Interactive Field Extractor to create the same fields as editing props.conf
  • Use the rex command to create fields "on-the-fly"

Extract and add new fields describes all of these options.

For all of them, it will be helpful to know regular expressions. Also, is this log indexed as sourcetype access_combined or access_combined_wcookie? If so, you have some existing fields that may help.

Following are some regular expressions that may work. I have shown them with the rex command.

1 - Extract the command field

 yoursearchhere | rex "/Company/directory/(?<command>.*?).shtml"

2 - Extract TypeaheadQueryResponder

yoursearchhere | rex "/typeahead/(?<TypeaheadQueryResponder>.*?)\s"

3 - Extract file

yoursearchhere | rex "/Company/(?<file>.*?)\?" 

xvxt006
Contributor

Hi,

To consolidate all the results, i have used the 2 rex commands in the same search (consolidated the first and 2nd ).

my search | rex "/typeahead/(?.*?)[\s|/]" | rex

"/Company/directory/(?.*?).shtml" | top limit=5000 command

If i do that, i am getting results but when i do the search individually (having one rex command only), the search results are different. Can someone help me how to get the results consistently whether i do the search separately or having the all rex commands in the same search.

0 Karma

lguinn2
Legend

You could do it all in one search like this:

yoursearchhere
| rex "/Company/directory/(?<command>.*?).shtml"
| rex "/typeahead/(?<TypeaheadQueryResponder>.*?)\s"
| rex "/Company/(?<file>.*?)\?"
| yourstatisticshere

You might be able to put it all into one giant regular expression. But when I thought about that, it made my head hurt.

0 Karma

xvxt006
Contributor

Thank you. They are giving results individually. Do you guys know how to combine all of these results? i have tried to have multiple rex (in the same search) and it is complaining. Any clue?

0 Karma

xvxt006
Contributor

Thanks to both of you. Response to my question is blazing fast. I will try one of these solutions and let you guys know how it goes. Thx again.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...