Getting Data In

Select the contents of the key-values fields from the string

mbabakov
Explorer

Hi!

There is a log with such records:

Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SSH/6/SSH_LOGIN(l): -DevIP=10.52.137.1; STEL user monitor (IP: 192.168.181.94) logged in successfully.
Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=ACCOUNT-AAAScheme= local-Service=login-UserName=monitor@system; AAA is successful.
Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=AUTHEN-AAAScheme= hwtacacs-scheme tacacs-Service=login-UserName=monitor@system; AAA is successful.
Dec 17 10:08:13 10.98.171.65 Jan 20 00:00:17 2011 MSR954-RTR-LTE-5686 %%10CELLULAR/5/CELLULAR: -DevIP=10.98.171.65;   Controller Cellular1/0: The network connection switched to 3G.
Dec 17 10:08:04 10.199.69.26 May 23 21:50:30 2012 930-RTR-14815 %%10SSH/4/TrapLogoff(t):  1.3.6.1.4.1.25506.2.22.1.3.0.4 SSH user logoff trap information

It is necessary to parse the fields between the characters "-" and ";". For example:

-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;

The difficulty is that the symbol "-" is used as a separator between key-value pairs, which is also present in the value: "AAAScheme = hwtacacs-scheme tacacs".

I used EVAL to replace "-" with "|":

EVAL-cmd_params_src = replace (cmd_params_src, "- (\ p {Lu})", "| \ 1")

And I see:

cmd_params_src = "DevIP = 10.52.137.1 | AAAType = AUTHEN | AAAScheme = hwtacacs-scheme tacacs | Service = login | UserName = monitor @ system"

But fields are not translated to key value.

props.conf

[hp_routers]
EVAL-vendor = "HP"
KV_MODE = none

REPORT-fields_general = extract_hp_route_general
REPORT-fields_cmd_parameters = extract_hp_route_cmd_parameters
REPORT-fields_cmd_message = extract_hp_route_cmd_message
EVAL-cmd_params_src = replace(cmd_params_src, "-(\p{Lu})", "|\1")
REPORT-field_params = extracet_field_from_params

transforms.conf

[extract_hp_route_general]
REGEX = ^(?P<Date>\w+\s+\d+\s+\d+:\d+:\d+)\s+(?P<device_ip>[^ ]+)\s+(?P<device_time>\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)[^ \n]* (?P<hostname>[^ ]+)[^ \n]* \%\%10(?<cmd_grp>\w+)\/$

[extract_hp_route_cmd_parameters]
REGEX = (?:\s-)(?<cmd_params_src>.+)(?:;)
disabled = 0

[extract_hp_route_cmd_message]
REGEX = (?:;\s+)(?<cmd_msg>.+)$

[extracet_field_from_params]
REGEX = (\w+)=(\s?[.\-\s@\w]+)
FORMAT = $1::$2
SOURCE_KEY = cmd_params_src
#DELIMS = "|", "="
REPEAT_MATCH = True
CLEAN_KEYS = 1

What am I doing wrong, how to fix the situation?

regards
Michael

0 Karma
1 Solution

woodcock
Esteemed Legend

To set this up to be automatic, like this:

In props.conf:

[hp_routers]
KV_MODE = none
REPORT-HP_KVPs = HP_KVPs

In transforms.conf:

[HP_KVPs]
REGEX = ([^-=]+)=([^=]*?)(?=$|-[^-=]+=)
FORMAT = $1::$2
REPEAT_MATCH = true

The key is to do your own KVP decoding and to use the RegEx positive lookahead feature to ensure that no keys (only values) can contain hyphens.

View solution in original post

woodcock
Esteemed Legend

To set this up to be automatic, like this:

In props.conf:

[hp_routers]
KV_MODE = none
REPORT-HP_KVPs = HP_KVPs

In transforms.conf:

[HP_KVPs]
REGEX = ([^-=]+)=([^=]*?)(?=$|-[^-=]+=)
FORMAT = $1::$2
REPEAT_MATCH = true

The key is to do your own KVP decoding and to use the RegEx positive lookahead feature to ensure that no keys (only values) can contain hyphens.

mbabakov
Explorer

Thank you!

0 Karma

woodcock
Esteemed Legend

Like this:

| makeresults 
| eval raw="Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SSH/6/SSH_LOGIN(l): -DevIP=10.52.137.1; STEL user monitor (IP: 192.168.181.94) logged in successfully.:::Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=ACCOUNT-AAAScheme= local-Service=login-UserName=monitor@system; AAA is successful.:::Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=AUTHEN-AAAScheme= hwtacacs-scheme tacacs-Service=login-UserName=monitor@system; AAA is successful.:::Dec 17 10:08:13 10.98.171.65 Jan 20 00:00:17 2011 MSR954-RTR-LTE-5686 %%10CELLULAR/5/CELLULAR: -DevIP=10.98.171.65;   Controller Cellular1/0: The network connection switched to 3G.:::Dec 17 10:08:04 10.199.69.26 May 23 21:50:30 2012 930-RTR-14815 %%10SSH/4/TrapLogoff(t):  1.3.6.1.4.1.25506.2.22.1.3.0.4 SSH user logoff trap information" 
| makemv delim=":::" raw 
| mvexpand raw 
| rename raw AS _raw
| eval _time = strptime(_raw, "%b %d %H:%M:%S")

| rename COMMENT AS "Everything above generates sample events; everything below is your solution"

| rex max_match=0 "(?<key>[^-=]+)=(?<value>[^=]*?)(?=$|-[^-=]+=)"
| eval _raw=mvzip(key, value, "=")
| fields - key value
| kv

woodcock
Esteemed Legend

The key is to do your own KVP decoding and to use the RegEx positive lookahead feature to ensure that no keys (only values) can contain hyphens.

0 Karma

mbabakov
Explorer

Thanks for your solution!
I wanted the analysis to go at the indexing stage. But in the end agreed to parse the search stage. 🙂

0 Karma

woodcock
Esteemed Legend

Right, I forgot that. See my other answer.

0 Karma

woodcock
Esteemed Legend

I think you clicked Accept on the wrong one, right?

0 Karma

mbabakov
Explorer

Sorry, yes. Corrected

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi mbabakov,
can you use "DevIP" that is after "-"?
If yes, you can try (see on regex101.com at https://regex101.com/r/2cwhUx/1 )

-(?<my_field>DevIP[^;]*)

If instead you're not sure about the word "DevIP" but you're sure that there's the equal after the first word, you could use something like this

-(?<my_field>[^\=]*\=[^;]*)

Bye.
Giuseppe

0 Karma

mbabakov
Explorer

Sorry, but I need to work out all possible lines between "-" and ";". For example:
-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;

A special case of "-DevIP = 10.52.137.1;", this is very simple. 🙂

0 Karma

gcusello
SplunkTrust
SplunkTrust

infact, as you can see at https://regex101.com/r/2cwhUx/1 using this regex, you take all between "-" and the first ";".
If you have more ";", you can use a similar regex.
Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Sorry for an additional information:
if you have "-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;" do you want separately each field?
something like

DevIP = 10.52.137.1
AAAType = ACCOUNT
AAAScheme = local
Service = login
UserName = monitor@system

if this is your requirement, see https://regex101.com/r/2cwhUx/2 and try

-(?<my_field>\w*\=[^;-]*)

Bye.
Giuseppe

0 Karma

mbabakov
Explorer

Yes, I need to separate each field separately. And I do not know how many of these fields will be in different records and how they will be called.
Therefore, I need a construction of the form (? <_ KEY_1> [^ =] [\ p {Lu} \ p {Lt}]. +) = (? <_ VAL_1> [^ |]. +). But, when using the separator "-" between the pairs, the regular expression becomes very complicated, since the "-" symbol can be used in the Value field.
I used EVAL to replace "-" with "|", but I don’t know how in this case to force splunk to parse all key-value pairs.

For example https://regex101.com/r/tlaRxc/1.

0 Karma

mbabakov
Explorer

To clarify, I am interested in the analysis of all possible fields in rows in the log, for example:
-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;

Those. for the first line is DevIP. For the second, these are DevIP, AAAType, AAAScheme, Service, UserName, etc.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...