Preventing rex from automatically retrieving the value from the next field when the Wi-Fi country name is empty

[sateesh250795]

Hi all,

rex "WifiCountryDetails\W+(?<WifiCountryDetails>[\w*\s*]+)"

We r using the above Rex for getting the Wi-Fi country details... But the problem is while fetching the data, if the Wi-Fi country name is empty it automatically gathers the next field value in it..

But if the wificountrydetails are empty it has to show empty in the data,

please let me know how to achieve it.

[inventsekar]

Hi @sateesh250795 ..

1)Can you provide some sample logs..

2) After the "WifiCountryDetails" field, do you have a "," or ":", etc ?!?!?

let me assume it is a comma (",")...then, lets try this rex:

rex "WifiCountryDetails\W+(?<WifiCountryDetails>[\w*\s*]+)\,"

[sateesh250795]

178.197.202.134 SECURITY_MONITORING_AUTOMATION CAMERA_SYSTEMS VML4 VML4030 1697109816450 INFO 51D29C77A022A KU5TFBKH-1700-316-114351851 - statusInfo {"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable","ChargerTech":"QuickCharger","ChargingState":"Off","Bat1Volt":7.931,"Temperature":24,"Battery1CaliVoltage":7.931,"Streamed":53395,"UserStreamed":1650,"MotionStreamed":7561,"IRLEDsOn":21054,"PoweredOn":11029385,"CameraOnline":11029337,"CameraOffline":48,"WifiConnectionCount":2,"WifiConnectionAttempts":2,"PIREvents":66,"FailedStreams":1,"FailedUpgrades":0,"SnapshotCount":0,"LogFrequency":2,"CriticalBatStatus":0,"ISPOn":57564,"TimeAtPlug":11025490,"TimeAtUnPlug":10979418,"PercentAtPlug":95,"PercentAtUnPlug":100,"ISPWatchdogCount":0,"ISPWatchdogCount2":0,"SecsPerPercentCurr":0,"SecsPerPercentAvg":0,"PirOorEvents":0,"DdrFailCnt":0,"carrierFw":"Generic","SignalStrengthIndicator":2,"SystemVersionString":"1.096.3.2_1.8.65.0_22","ID":534,"SystemFirmwareVersion":"1.096.3.2_30_0bad0df","HardwareVersion":"H6","SystemSerialNumber":"51D29C77A022A","TimeStamp":"2023-10-12T11:23:36+0000"}

[inventsekar]

Hi @sateesh250795 .... may we know if you got answer for your question please, thanks. 

[yuanliu]

As I always warn people, do not treat structured data as text.  Using Splunk's built-in function with JSON, XML, and so on is much more robust and saves you tons of headaches in future maintenance.

All you need to do is to extract the conformant JSON into its own field.  Then apply spath. (An alternative new syntax is fromjson.)

| rex "^[^{]+\s*(?<json>{.+})"
| spath input=json

It's that simple.  Your sample log will give you everything in that JSON.

fieldname	fieldvalue
Bat1Volt	7.931
BatPercent	100
BatTech	Rechargeable
Battery1CaliVoltage	7.931
CameraOffline	48
CameraOnline	11029337
ChargerTech	QuickCharger
ChargingState	Off
CriticalBatStatus	0
DdrFailCnt	0
FailedStreams	1
FailedUpgrades	0
HardwareRevision	H6
HardwareVersion	H6
ID	534
IRLEDsOn	21054
ISPOn	57564
ISPWatchdogCount	0
ISPWatchdogCount2	0
LogFrequency	2
MotionStreamed	7561
PIREvents	66
PercentAtPlug	95
PercentAtUnPlug	100
PirOorEvents	0
PoweredOn	11029385
SecsPerPercentAvg	0
SecsPerPercentCurr	0
SignalStrengthIndicator	2
SnapshotCount	0
Streamed	53395
SystemFirmwareVersion	1.096.3.2_30_0bad0df
SystemSerialNumber	51D29C77A022A
SystemVersionString	1.096.3.2_1.8.65.0_22
Temperature	24
TimeAtPlug	11025490
TimeAtUnPlug	10979418
TimeStamp	2023-10-12T11:23:36+0000
Type	status
UserStreamed	1650
WifiConnectionAttempts	2
WifiConnectionCount	2
WifiCountryDetails
carrierFw	Generic
json	{"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable","ChargerTech":"QuickCharger","ChargingState":"Off","Bat1Volt":7.931,"Temperature":24,"Battery1CaliVoltage":7.931,"Streamed":53395,"UserStreamed":1650,"MotionStreamed":7561,"IRLEDsOn":21054,"PoweredOn":11029385,"CameraOnline":11029337,"CameraOffline":48,"WifiConnectionCount":2,"WifiConnectionAttempts":2,"PIREvents":66,"FailedStreams":1,"FailedUpgrades":0,"SnapshotCount":0,"LogFrequency":2,"CriticalBatStatus":0,"ISPOn":57564,"TimeAtPlug":11025490,"TimeAtUnPlug":10979418,"PercentAtPlug":95,"PercentAtUnPlug":100,"ISPWatchdogCount":0,"ISPWatchdogCount2":0,"SecsPerPercentCurr":0,"SecsPerPercentAvg":0,"PirOorEvents":0,"DdrFailCnt":0,"carrierFw":"Generic","SignalStrengthIndicator":2,"SystemVersionString":"1.096.3.2_1.8.65.0_22","ID":534,"SystemFirmwareVersion":"1.096.3.2_30_0bad0df","HardwareVersion":"H6","SystemSerialNumber":"51D29C77A022A","TimeStamp":"2023-10-12T11:23:36+0000"}

Here is an emulation you can play with and compare with real data

| makeresults
| eval _raw="178.197.202.134 SECURITY_MONITORING_AUTOMATION CAMERA_SYSTEMS VML4 VML4030 1697109816450 INFO 51D29C77A022A KU5TFBKH-1700-316-114351851 - statusInfo {\"Type\":\"status\",\"HardwareRevision\":\"H6\",\"WifiCountryDetails\":\"\",\"BatPercent\":100,\"BatTech\":\"Rechargeable\",\"ChargerTech\":\"QuickCharger\",\"ChargingState\":\"Off\",\"Bat1Volt\":7.931,\"Temperature\":24,\"Battery1CaliVoltage\":7.931,\"Streamed\":53395,\"UserStreamed\":1650,\"MotionStreamed\":7561,\"IRLEDsOn\":21054,\"PoweredOn\":11029385,\"CameraOnline\":11029337,\"CameraOffline\":48,\"WifiConnectionCount\":2,\"WifiConnectionAttempts\":2,\"PIREvents\":66,\"FailedStreams\":1,\"FailedUpgrades\":0,\"SnapshotCount\":0,\"LogFrequency\":2,\"CriticalBatStatus\":0,\"ISPOn\":57564,\"TimeAtPlug\":11025490,\"TimeAtUnPlug\":10979418,\"PercentAtPlug\":95,\"PercentAtUnPlug\":100,\"ISPWatchdogCount\":0,\"ISPWatchdogCount2\":0,\"SecsPerPercentCurr\":0,\"SecsPerPercentAvg\":0,\"PirOorEvents\":0,\"DdrFailCnt\":0,\"carrierFw\":\"Generic\",\"SignalStrengthIndicator\":2,\"SystemVersionString\":\"1.096.3.2_1.8.65.0_22\",\"ID\":534,\"SystemFirmwareVersion\":\"1.096.3.2_30_0bad0df\",\"HardwareVersion\":\"H6\",\"SystemSerialNumber\":\"51D29C77A022A\",\"TimeStamp\":\"2023-10-12T11:23:36+0000\"}"
``` data emulation above ```

[PickleRick]

While I wholehaeartedly agree about not using text-based tools for structured data, the problem is that you can't use spath for defining extractions - you can only use it in search. And json or xml kv-extractions work only if the whole event is a well-formed structure. You can't use it if you have - for example - a json event with a syslog header.

There is an open idea about it somewhere on ideas.splunk.com

[yuanliu]

---

extractions work only if the whole event is a well-formed structure. You can't use it if you have - for example - a json event with a syslog header.

There is an open idea about it somewhere on ideas.splunk.com

---

I remember seeing you posting a link in Slack but can't find it in a hurry.  Maybe more people could vote for such.  Be honest, though, seeing all kinds of "smart" schemes developers come up to mix JSON with plain text in this board, it can be tricky to implement a generally application solution.

One workaround for situations like this is to make a data-specific regex field extraction to extract the compliant part for each sourcetype so this regex doesn't have to be included in search commands.  This makes easier code maintenance.

[PickleRick]

Found it.

https://ideas.splunk.com/ideas/EID-I-208

And I don't think it even needs to cover so many use cases. So far we can't do the one, most popular - a structured data blob forwarded wthin a syslog message.

I wiuldn't expect such feature to be too automatic. It could just go with "if the sourcetype has KV_MODE set the STRUCTURED_PART (by default (.*) ) defines which part of the raw event is subjected to KV extraction". And that's it. You could do your own regex to capture everything after the syslog header or whatever you need. Simple yet flexible. Like LINE_BREAKER or TOKENIZER

[yuanliu]

That STRUCTURED_PART could be entered as a comment.  I think it is more practical than a general solution.

[inventsekar]

the rex command, in my previous reply.. works perfect. pls check it. 

As you are  a new member, karma points / upvotes are appreciated, pls help those who help you, with your karma points. if any reply solved your issue, pls accept it as "Accept as solution", thanks. 

[inventsekar]

Lets try this (i have tested this on my laptop, its working fine)

source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty" 
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>\w+)\"" | table _raw WifiCountryDetails

[sateesh250795]

If we r doing it for stats its not working.... giving the same results as before .... can u pls try for stats count by wificountrydetails instead of table  

[inventsekar]

source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty" 
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>\w+)\"" 
| stats count by WifiCountryDetails

working fine, as suggested previously, check screenshot below..  (pls note the rex command closely.. i have edited ur rex little bit.. i added the ("\"\:\") before rex and (\") after the rex). 

[sateesh250795]

I did exactly same but the again its fetching the next field.. pls look into the screenshot

[inventsekar]

please take the screenshot along with the SPL command also seen inside the image (zoom down or keep the browser window small, so big portion can be copied to the image)

[sateesh250795]

pls have a look

[inventsekar]

source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty" 
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>(\w+|\S))\"" 
| stats count by WifiCountryDetails

Pls check the above SPL.. i checked it and its working fine. 

to troubleshoot further.. can you pls copy paste two sample logs, one with country and one without country. 

(pls remove ip address, important details from the sample logs.. just copy paste the line like these..

{"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"","BatPercent":100,"BatTech":"Rechargeable"

{"Type":"status","HardwareRevision":"H6","WifiCountryDetails":"india","BatPercent":100,"BatTech":"Rechargeable"

)

[inventsekar]

Hi Splunk New beginners:
For absolute beginners...Splunk newbie learning videos..
https://www.youtube.com/@SiemNewbies101/playlists

I have added 25 small videos of rex... Completely for Splunk newbies and beginners.
hope this helps somebody, thanks.

[inventsekar]

Pls check this SPL:

source="rex-empty-string.txt" host="laptop" sourcetype="rexEmpty" 
| rex "WifiCountryDetails\"\:\"(?<WifiCountryDetails>(\w+|\S))\"" 
| eval WifiCountryDetails = if(isnull(WifiCountryDetails) OR len(WifiCountryDetails)==0, "No Country Name", WifiCountryDetails)
| stats count by WifiCountryDetails

[sateesh250795]

No. its not working😑. again its taking the next field value... and we r gathering the count by using stats count by wificountrydetails... 

[sateesh250795]

there is : in the place of wificountry details ..... its has to fetch as it is.. but it is fetching next field value batpercent in the place of wificountry details.