Splunk Search

How do you display the # of requests per 24 hours of log access (Between 00:00:00 and 23:59:59) for set of keywords?

saifullakhalid
Explorer

I have a set of keywords which I need to search and generate the output similar to
https://answers.splunk.com/answers/588828/how-do-you-display-the-of-requests-per-24-hours-of.html#co...

keywords:

DCBClaimSearch
CountryCode=&PRODUCT_LINE_CD=&REGION_CD=&LOB=&LOB_SUB_CD=&Count=
etc

log format:

1x.xx.xxx.xxx - - 1xxxx4 [04/Oct/2017:00:03:44 -0400] - /xxx/Gxxxt/ibxx_xxxxxxxxxxxxxxxxxxxxxxx[[/!TABTHREAD1 HTTP/1.1 xxxxxxxxxxx.net TIME:0/49626 "POST /pxxxb/Gxxxxt/ixxxU_xxxxxxxxxxxxxxxxxxxxxxxxx/!TABTHREAD1?pyActivity=xxxxxxxxxxxxxxxxxxxxxxxx%20&Request_Type=&xxxxxxxxxxxx_CD=Nxxx_xxxxEL&Exxxxxxxxx_CD=&Lxxxxxxxxxe=&CountryCode=&PRODUCT_LINE_CD=&REGION_CD=&LOB=&LOB_SUB_CD=&Count= HTTP/1.1" 200 737

Tags (1)
0 Karma

saifullakhalid
Explorer

?pyActivity=...............................................PreActivity=DCBClaimSearch&HeaderButtonSectionName.................HTTP/1.1" 200 4502
?pyActivity=...........................LanguageCode=&CountryCode=&PRODUCT_LINE_CD=&REGION_CD=&LOB=&LOB_SUB_CD=&Count=..........HTTP/1.1" 200 3402

Every keyword has different pattern and it lies in middle of requests but all will start with ?pyActivity=

For example:
?pyActivity=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx%20&Request_Type=&xxxxxxxxxxxxxx_xxxxxxxxxxxxxxx&ELEMENT_CD=&LanguageCode=&CountryCode=&PRODUCT_LINE_CD=&REGION_CD=&LOB=&LOB_SUB_CD=&Count= HTTP/1.1" 200 5092

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi @saifullakhalid,

Can you please try this search?

YOUR_SEARCH
| rex field=_raw Count=(.*)\s.*\s(?<COUNT>.*) 
| stats count AS "Count" sum(COUNT) as "Sum of COUNT" min(_time) AS start_time max(_time) as end_time by date_month,date_mday,date_hour
| eval "Start Time"=strftime(start_time,"%d/%m/%Y %I:%M:%S:%p") 
| eval "End Time"=strftime(end_time,"%d/%m/%Y %I:%M:%S:%p") 
| table "Start Time" "End Time" "Count" "Sum of COUNT"

Thanks

0 Karma

saifullakhalid
Explorer

This is displaying me start time , end time Count and total count, How should I display the keys words below along with these. I tried adding COUNT, but it was empty. I tried adding like below but it is displaying the full request, I just wanted keywords to display.

index="prod_4oct_508" pyActivity="" | rex field=_raw Count=(.)\s.\s(?.)
| stats count AS "Count" sum(COUNT) as "Sum of COUNT" min(_time) AS start_time max(_time) as end_time by date_month,date_mday,date_hour,pyActivity
| eval "Start Time"=strftime(start_time,"%d/%m/%Y %I:%M:%S:%p")
| eval "End Time"=strftime(end_time,"%d/%m/%Y %I:%M:%S:%p") | dedup pyActivity | sort –count
| table "Start Time" "End Time" "pyActivity" "Count" "Sum of COUNT"

keywords:

DCBClaimSearch
CountryCode=&PRODUCT_LINE_CD=®ION_CD=&LOB=&LOB_SUB_CD=&Count=

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi @saifullakhalid,

Do you want to extract these keywords?

DCBClaimSearch
CountryCode=&PRODUCT_LINE_CD=®ION_CD=&LOB=&LOB_SUB_CD=&Count=

Below search will extract "CountryCode=&PRODUCT_LINE_CD=®ION_CD=&LOB=&LOB_SUB_CD=&Count="

index="prod_4oct_508" pyActivity="" 
| rex field=_raw "&CountryCode=(?<FIELD1>.*)\s(.*)\s(.*)\s(?<COUNT>.*)"
| table _time FIELD1 COUNT

Can you please let me know the pattern of keyword "DCBClaimSearch" ?

Thanks

0 Karma

saifullakhalid
Explorer

I have written a java program which reads the list of keywords (61 keywords )from csv and parse the logs and generate 24 hours reports like below for each keyword. Can we do this in splunk if so then please suggest.

First report format:

User Time Protocol Url Elapsed Time (Seconds) call Status Size logname
1432029 17/Oct/04 05:19:05 HTTP/1.1 xxxxxxxxxxx.net 0.150266 POST /xxxx/xxxxt/xxxxxxxxxxxxxx*/!TABTHREAD4?pyActivity=Reloadxxxx&pzxxxxxxxx=xxxxxxxxx&pzFromFrame=xxxxxxxxx&pzxxxxe=pyxxxe&pzxxxxx=false&StxxxxName=xxxxxe&xxxx&inStandardsMode=false&AJAXTrackID=3&pzHarnessID=xxx36749 HTTP/1.1 200 2207 \507\access_log_10_04_2017

second report format:

Transaction keyword Start Time End Time Total # of executions Avg # of Executions per Hour Min Resp Time Max Resp Time Avg Resp Time 90th percentile Resp Time Std Dev Of Resp Time Min Size of Response Max Size of Response Avg Size of Response 90th percentile Size of Response Std Dev Of Size of Response

TC01 DCBClaimSearch 17/Oct/04 00:11:46 17/Oct/04 23:24:05 2398 104 0.029761 27.35804 0.108725955 0.1099671 0.594170951 82 10342 4302.940784 4543 424.216474

3rd report:
Transaction keyword Start Time End Time Total # of executions Avg # of Executions per Hour Min Resp Time Max Resp Time Avg Resp Time 90th percentile Resp Time Std Dev Of Resp Time Min Size of Response Max Size of Response Avg Size of Response 90th percentile Size of Response Std Dev Of Size of Response

TC01 DCBClaimSearch 17/Oct/04 00:00:00 17/Oct/04 00:59:59 4 4 0.056125 0.070999 0.0613225 0.070999 0.00671778 3617 4533 3886.75 4533 437.5083809
TC01 DCBClaimSearch 17/Oct/04 01:00:00 17/Oct/04 01:59:59 3 3 0.058215 0.080105 0.066264 0.080105 0.012039662 3780 4548 4036 4548 443.4050067

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi @saifullakhalid,

Can you please try this?

 index="prod_4oct_508" pyActivity="*" 
| rex field=_raw "&CountryCode=(?<FIELD1>.*)\s(.*)\s(.*)\s(?<COUNT>.*)"
| rex field=_raw "pyActivity=(.*)PreActivity=(?<PreActivity>.*)&" 
| table _time PreActivity _time FIELD1 COUNT
0 Karma

saifullakhalid
Explorer

looks like it is extracting values which has the pattern "&CountryCode" and c
But is it possible to read these keywords values which needs to be extracted using csv. because as I said the pattern is every line in the code starts with ?pyActivity= , but the keywords falls in middle of this request. like in above comment "CountryCode=&PRODUCT_LINE_CD=®ION_CD=&LOB=&LOB_SUB_CD=&Count=" is part of the line ?pyActivity=

0 Karma

saifullakhalid
Explorer
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...