- Splunk Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- rex field limit
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
rex field limit
Hi,
I got the ff script working but putting in more rex field hangs splunk
index=xxx
| rex field=_raw "tel:001001(?
| rex field=_raw "tel:001001(?
| lookup roaming_db.csv HLR_ADDR OUTPUT Destination Operator_Name
| eval HLR_ADDR = HLR_ADDR + " - " + Destination + " - " + Operator_Name
| stats count by HLR_ADDR
Is there other ways to approach this?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
New Answer: I am still using a lookup, but to simplify things I have included a macro. Will my first answer work? Sure, but you will need to create hlr_addr4 hlr_addr5 hlr_addr6 hlr_addr7 hlr_addr8 - where will the madness end!!
The new solution still includes a lookup, defined as follows in transforms.conf
[roaming_lookup]
filename = roaming_db.csv
min_matches = 1
default_match =Unknown
It adds a macro, which is defined in macros.conf (or the GUI)
[check_hrl_addr(1)]
args = len
definition = "rex "tel:001001(?<HRL_ADDR$len$>\d{$len$}) | lookup roaming_lookup HLR_ADDR OUTPUT Destination$len$ Operator_Name$len$"
validation =isnum(len)
errormsg = len must be a number
Now the search is still pretty long, but not quite so bad.
index=xxx
| `check_hrl_addr(4)` | `check_hrl_addr(5)` | `check_hrl_addr(6)` | `check_hrl_addr(7)` | `check_hrl_addr(8)`
| eval Destination = case(Destination4!="Unknown", Destination4, Destination5!="Unknown", Destination5,
Destination6!="Unknown", Destination6, Destination7!="Unknown", Destination7,
Destination8!="Unknown", Destination8, 1==1, "Unknown")
| eval Operator_Name = case(Operator_Name4!="Unknown", Operator_Name4, Operator_Name5!="Unknown",
Operator_Name5, Operator_Name6!="Unknown", Operator_Name6,
Operator_Name7!="Unknown", Operator_Name7, Operator_Name8!="Unknown", Operator_Name8, 1==1, "Unknown")
| eval HRL_ADDR = case(HRL_ADDR4!="Unknown", HRL_ADDR4, HRL_ADDR5!="Unknown", HRL_ADDR5,
HRL_ADDR6!="Unknown", HRL_ADDR6, HRL_ADDR7!="Unknown", HRL_ADDR7,
HRL_ADDR8!="Unknown", HRL_ADDR8, 1==1, "Unknown")
| fields - HRL_ADDR4 HRL_ADDR5 HRL_ADDR6 HRL_ADDR7 HRL_ADDR8 Destination4 Destination5 Destination6 Destination7
Destination8 Operator_Name4 Operator_Name5 Operator_Name6 Operator_Name7 Operator_Name8
| stats count by HLR_ADDR Destination Operator_Name
Still, I think that there might be a nicer way to write the rex and do it just once! But without seeing the actual events, this is the best I could do.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
UPDATE: It was pointed out that the hrl_addr can be 4 to 8 digits, not just 4 or 5. Therefore, I have updated this answer as a separate answer - it was just too hard to rewrite this one...
I would consider doing it this way - use your lookup table to validate your hrl_addr like this
index=xxx
| rex "tel:001001(?<hlr_addr4>\d{4})"
| lookup roaming_db.csv HLR_ADDR as hlr_addr4 OUTPUT Destination as Destination4 Operator_Name as Operator_Name4
| rex "tel:001001(?<hlr_addr5>\d{5})"
| lookup roaming_lookup HLR_ADDR as hlr_addr5 HR OUTPUT Destination as Destination4 Operator_Name as Operator_Name4
| eval Destination = if(Destination4=="Unknown", Destination5, Destination4)
| eval Operator_Name = if(Destination4=="Unknown",Operator_Name5,Operator_Name4)
| stats count by HLR_ADDR Destination Operator_Name
In order for this to work, you will need to setup your CSV file as a lookup. If you do it manually in transforms.conf, it will look like this:
[roaming_lookup]
filename = roaming_db.csv
min_matches = 1
default_match =Unknown
(This transforms.conf goes in the same app as the csv file.) Or you can set up the lookup using the GUI. Just be sure to set the minimum number of matches to 1, and the default match value to Unknown.
The nice thing about this solution is that it will behave consistently, even with a large CSV file.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
rex "tel:001001(?<hrl_addr>\d{4,8})" will not work because the regular expression has nothing after the {4,8} - how would the regex know whether to choose 4, 5, 6, 7 or 8 digits?
IF we know that the number ended with a space, we could do this
rex "tel:001001(?<hrl_addr>\d+) "
which says "pick off the one or more digits up to the next space" - but we don't know that. We don't know how many digits we need until we check to see if it matches in the lookup.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Right, but what I don't get is why you need separate fields for different lengths.
rex "tel:001001(?<hrl_addr>\d{4,8})"
Why would this not work?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ouch - I didn't notice the ones with length 6-8! Still, I like my solution better than having to maintain these awful lists...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Doesn't the hlr_addr field vary in length more than that?
The second rex checks for values of length 4 to 8:
33660|33751|68977|59669690|59035|24105|24107|22077|2206|49151...919802|919806
I don't get the need to split it into 4 or 5 digit numbers, but I might be missing something.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Would you mind providing some data samples? This will help if there are any alternative rex commands.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
this is not a good approach, rather than writing this you can count the number of "|" pipes and group them in rex command e.g."tel:001001(?
But i agree splunk will be slow with this much regex processing.
Introducing the 2024 SplunkTrust!
Introducing the 2024 Splunk MVPs!
Splunk Custom Visualizations App End of Life
