Splunk Search

regex help with existing regex

fisuser1
Contributor

have a business area that changed some of their log format which broke my existing regex and having a hard time matching response code. seems my existing regex is pulling two matches from each event. (matching "fromIndex=150" or

*"fromIndex=100"* also in the event) any suggestions to edit this? 

Existing regex: (working before log format change)
\b(?<**http_status**>\d{3}) \d

_raw data:
2019-05-30 17:52:15 127.0.0.1 GET /api/accounts/19006/account-history timePeriod=&type=&amount=&check=&description=&*startfromIndex=100* 1123 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"061120534","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:67.0)+Gecko/20100101+Firefox/67.0 https://my.bankingsite.com/ORUI/ **200** 0 0 3531 127.0.0.1 

2019-05-30 17:52:05 127.0.0.1 GET /api/accounts/67343/account-history timePeriod=7&type=1&amount=&check=&description=&*startfromIndex=100* 1124 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"061120686","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 875 127.0.0.1 

2019-05-30 17:52:03 127.0.0.1 GET /api/accounts/46850/account-history timePeriod=&type=&amount=&check=&description=&*startfromIndex=100* 1120 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"091302966","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Linux;+Android+7.0;+SM-G928V)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/73.0.3683.90+Mobile+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 3578 127.0.0.1 

2019-05-30 17:51:51 127.0.0.1 GET /103103985/api/accounts/33098/account-history timePeriod=&type=&amount=&check=&description=&*startfromIndex=100* 80 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"103103985","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+10.0;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko https://my.bankingsite.com/103103985/ORUI/ **200** 0 0 562 127.0.0.1 

2019-05-30 17:51:50 127.0.0.1 GET /api/accounts/14342/account-history timePeriod=03/22/2019,05/30/2019&type=1&amount=&check=&description=&*startfromIndex=100* 1111 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"061120806","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:61.0)+Gecko/20100101+Firefox/61.0 https://my.bankingsite.com/ORUI/index.html 200 0 0 1718 127.0.0.1 

2019-05-30 17:51:47 127.0.0.1 GET /api/accounts/128235/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1101 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"053102586","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 484 127.0.0.1 

2019-05-30 17:51:43 127.0.0.1 GET /api/accounts/57435/account-history timePeriod=7&type=1&amount=&check=&description=&*startfromIndex=100* 1112 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"064106775","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 1125 127.0.0.1 

2019-05-30 17:51:41 127.0.0.1 GET /api/accounts/66752/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1150 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"221971015","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 187 127.0.0.1 

2019-05-30 17:51:35 127.0.0.1 GET /api/accounts/290903/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1127 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"102000966","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 1562 127.0.0.1 

2019-05-30 17:51:32 127.0.0.1 GET /api/accounts/36874/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1107 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"063114030","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 875 127.0.0.1 

2019-05-30 17:51:30 127.0.0.1 GET /api/accounts/24299/account-history timePeriod=&type=&amount=&check=&description=&*startfromIndex=100* 1135 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"111908965","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+6.1;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko https://my.bankingsite.com/ORUI/ **200** 0 0 6703 127.0.0.1 

2019-05-30 17:51:17 127.0.0.1 GET /api/accounts/389912/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1127 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"102000966","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 765 127.0.0.1 

2019-05-30 17:51:01 127.0.0.1 GET /pahranagatvalleyfcu/api/accounts/4058/account-history timePeriod=09/27/2016,05/30/2019&type=1&amount=&check=&description=&*startfromIndex=100* 1101 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"003381.MERCURY","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:61.0)+Gecko/20100101+Firefox/61.0 https://my.bankingsite.com/PahranagatValleyFCU/ORUI/index.html **200** 0 0 6546 127.0.0.1 

2019-05-30 18:03:26 127.0.0.1 GET /api/system/logoff - 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 0 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/personalfinance/ widgetName=mini_spending_widget&sync=true 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 0 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/system/logoff - 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 15 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/system/logoff - 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 0 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/personalfinance/ widgetName=mini_spending_widget&sync=true 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 15 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/system/logoff - 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 0 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /CommCUofNewMilford/api/users/22/getholdamount accountId=2175 1101 {"UserId”:crazy_carl,”Username”:”foo”,”Realm":"003042.MERCURY","sessionTimeout":"20","ipAddress":"107.77.224.32"} 127.0.0.1 Mozilla/5.0+(iPhone;+CPU+iPhone+OS+12_2+like+Mac+OS+X)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Mobile/15E148+Safari/604.1 https://my.bankingsite.com/CommCUofNewMilford/ORUI/ **200** 0 0 109 127.0.0.1

2019-05-30 18:03:25 127.0.0.1 GET /api/users/22/accounts assetSortOrder=ASC&assetDefaultSortColumn=accountName&investmentSortOrder=ASC&investmentDefaultSortColumn=accountName&liabilitiesortOrder=ASC&liabilitiesDefaultSortColumn=accountName&externalSortOrder=ASC&externalDefaultSortColumn=accountName&ccSortOrder=ASC&ccDefaultSortColumn=accountName 1101 {"UserId”:crazy_carl,”Username”:”foo”,”Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(Linux;+Android+9;+SM-G955U)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.157+Mobile+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 140 127.0.0.1

2019-05-30 18:03:25 127.0.0.1 GET /login.aspx ReturnUrl=%2fORUI%2f 1101 - 127.0.0.1 Mozilla/5.0+(iPhone;+CPU+iPhone+OS+12_3_1+like+Mac+OS+X)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1.1+Mobile/15E148+Safari/604.1 - **200** 0 0 46 172.58.99.130

2019-05-30 18:03:25 127.0.0.1 GET /api/users/22/scorecardrewards - 1101 {"UserId”:crazy_carl,”Username”:”foo”,”Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(Linux;+Android+9;+SM-G955U)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.157+Mobile+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 46 127.0.0.1

2019-05-30 18:03:25 127.0.0.1 GET /ORUI/index.html - 1101 {"UserId”:crazy_carl,”Username”:”foo”,”Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1 "} 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.169+Safari/537.36 https://my.bankingsite.com/PassMarkRecognizedAdv.aspx?qs=b2TU7c2wr8E0dfptJiVc6osqODJNVVaa6UZusUsRdZI... **200** 0 0 125 127.0.0.1

2019-05-30 18:03:25 127.0.0.1 GET /OOBChallenge.aspx qs=cUlHrH9oGju91rnRg%2bOmkzrLhg8Oc0ZMSvVHZDwpaoNAhY0dPG6o3WXkCMUZMARx61iChJjKqmntkcKCmNEX9oD2KDmMage%2fb2TU7c2wr8E0dfptJiVc6osqODJNVVaan3drqxuh3WzZy%2fva1rs6RM9OaC59wRrRZ2yA%2bDcWz7lmJSZPd2bHwWdceDRZ9bE2QNZL%2f5%2fuohrofoZvlVaPJA%3d%3d 1101 - 127.0.0.1 Mozilla/5.0+(Windows+NT+6.1;+Trident/7.0;+rv:11.0)+like+Gecko https://my.bankingsite.com/SignOn.aspx?qs=nLl1ZWczEjHofoZvlVaPJA%3d%3d **200** 0 0 453 127.0.0.1
Tags (2)
0 Karma
1 Solution

martinpu
Communicator

Having asterisks around the staus complicated things by a bit, it is not good practice to use asterisks like that.

This works :

[\*]{2}(?<http_status>\d{3})[\*]{2}

it doesnt work on line 15 as that is formatted differently than others, is this a mistake or will these events also be in the sample? In that case the regex will need to be slightly different

For confirming and working on regex, I'd recommend this site:

https://rubular.com/

paste the sample events into the test string area and the regex into the reg expression editor and see the magic 🙂

View solution in original post

0 Karma

martinpu
Communicator

Having asterisks around the staus complicated things by a bit, it is not good practice to use asterisks like that.

This works :

[\*]{2}(?<http_status>\d{3})[\*]{2}

it doesnt work on line 15 as that is formatted differently than others, is this a mistake or will these events also be in the sample? In that case the regex will need to be slightly different

For confirming and working on regex, I'd recommend this site:

https://rubular.com/

paste the sample events into the test string area and the regex into the reg expression editor and see the magic 🙂

0 Karma

fisuser1
Contributor

Thanks for the response @martinpu. actually, for whatever reason, the answers.splunk.com format added that. there are no ** around the status codes in the logs. These are IIS logs. Not sure why the format of this page added them. Pasting a few examples again.

2019-05-30 11:26:48 127.0.0.1 GET /javascript/MenuDropItems.js v=1801 1101 - 10.237.0.43 Mozilla/5.0+(Linux;+Android+8.0.0;+SM-G930T)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.157+Mobile+Safari/537.36 https://online.com/PassMarkRecognizedAdv.aspx?qs=b2TU7c2wr8E0dfptJiVc6osqODJNVVaa6UZusUsRdZI%3d 200 0 0 0 127.0.0.1

2019-05-30 10:56:06 127.0.0.1 GET /api/accounts/3448122/account-history timePeriod=1&type=1&amount=&check=&description=&startfromIndex=150 1101 {"UserId":carl,"Username":"mr_blah","Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://online.com/ORUI/ 200 0 0 250 127.0.0.1

2019-05-30 10:55:57 127.0.0.1 GET /064107994/api/accounts/17004/account-history timePeriod=03/01/2019,05/30/2019&type=1&amount=&check=&description=&startfromIndex=150 80 {"UserId":carl,"Username":"mr_blah","Realm":"064107994","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:61.0)+Gecko/20100101+Firefox/61.0 https://www.banking.com/064107994/ORUI/index.html 200 0 0 1062 127.0.0.1

2019-05-30 10:54:59 127.0.0.1 GET /api/accounts/55233/account-history timePeriod=1&type=1&amount=&check=&description=&startfromIndex=150 1120 {"UserId":carl,"Username":"mr_blah","Realm":"091302966","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://choicefinancialgroup.ebanking-services.com/ORUI/ 200 0 0 2375 127.0.0.1

2019-05-30 10:54:54 10.237.66.52 GET /api/accounts/69173/account-history timePeriod=1&type=1&amount=&check=&description=&startfromIndex=150 1150 {"UserId":carl,"Username":"mr_blah","Realm":"221971015","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://onlinebanking.rhinebeckbank.com/ORUI/ 200 0 0 734 127.0.0.1

0 Karma

fisuser1
Contributor
0 Karma

martinpu
Communicator

Try this:

http([^\s]+) (?<**http_status**>\d{3}) \d

After http matches any word until space then takes 3 digits if those are followed by a space and a digit, I think it should cover everything

0 Karma

fisuser1
Contributor

thank you @martinpu, i tested this and seemed to extract properly. I'll continue monitor the update I made (very inconsistent logs which the dev team is addressing) and refer to your extraction if necessary. thanks again for all the help!

0 Karma

martinpu
Communicator

You're very welcome 🙂

Happy Splunking

0 Karma

fisuser1
Contributor

this was the working regex before they changed format

\b(?<http_status>\d{3}) \d

0 Karma

mydog8it
Builder

I think this is what you want....

        \*\*(?<http_status>\d{3})\*\*
0 Karma

fisuser1
Contributor

thx for the response, but this doesn't seem to extract any field, http_status, in from the raw data

0 Karma

mydog8it
Builder

How are you trying to use the regex? Are you trying to use it in SPL or as part of a conf? Can you better define what you are trying to collect? The question describes trying to collect the "fromIndex=" value but the regex looks like it is trying to extract the http status value. My suggestion was for the http status code.

0 Karma

fisuser1
Contributor

this will be done via props. trying to match on the http_status value only, ie 200 or 401 in the raw data provided. the current regex I provided is matching on both "fromIndex=" AND http status fields after the application team changed the log format.

0 Karma

fisuser1
Contributor

another example:

2019-05-30 10:56:06 127.0.0.1 GET /api/accounts/3448122/account-history timePeriod=1&type=1&amount=&check=&description=&startfromIndex=150 1101 {"UserId":9999999,"Username":"mr_blah","Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://online.com/ORUI/ 200 0 0 250 127.0.0.1

0 Karma

fisuser1
Contributor

working regex before log format change, which was extracting http_status

\b(?<http_status>\d{3}) \d

0 Karma

fisuser1
Contributor
0 Karma

mydog8it
Builder
0 Karma

mydog8it
Builder
\b(?<http_status>\d{3})\s\d+\s\d+\s\d+\s\d+\.\d+\.\d+\.\d+
0 Karma

mydog8it
Builder

That checks for all the fields to the end of the message, its ugly but got rid of false matches.

0 Karma

fisuser1
Contributor

thank you @mydog8it , it looks like this may have worked. the log format is pretty inconsistent, so I do have the developers/admins fixing this, but it seems this is matching more accurately now. thank you again for all the help!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...