Explain this transform

[cvajs]

v4.3 sles 11.1

can you explain for me this transform

[csafields]
REGEX = ^[^\|]+\|([^\|]+)\|([^\|]+)\|([^\|]+)\|([^\|]+)\|([^\|]+)\|([^\|]+)
FORMAT = nbtname::$1 ip::$2 ruleid::$3 code::$4 remotetime::$5 alert::$6

i get the FORMAT part (which doesnt work correctly with my event data), but what is this regex? what does the \| mean?

here's the raw event data

2012-04-05 15:38:29 syslog.prod.org [UDP: [10.219.0.134]:1086->[10.222.1.253]]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (124678725) 14 days, 10:19:47.25       SNMPv2-MIB::snmpTrapOID.0 = OID: CSAMC-SNMPv2-MIB::csaTraps.1        CSAMC-SNMPv2-MIB::eventID = INTEGER: 10329635   CSAMC-SNMPv2-MIB::ruleID = Wrong Type (should be INTEGER): NULL      CSAMC-SNMPv2-MIB::hostName = STRING: "hostB.prod.org"   CSAMC-SNMPv2-MIB::eventTime = STRING: "2012-04-05 15:38:29.617"      CSAMC-SNMPv2-MIB::severityCode = INTEGER: 2     CSAMC-SNMPv2-MIB::eventCode = INTEGER: 164   CSAMC-SNMPv2-MIB::processName = STRING: "<remote application>"  CSAMC-SNMPv2-MIB::fileName = Wrong Type (should be OCTET STRING): NULL       CSAMC-SNMPv2-MIB::sourceIPAddress = Wrong Type (should be IpAddress): NULL      CSAMC-SNMPv2-MIB::destinationIPAddress = Wrong Type (should be IpAddress): NULL      CSAMC-SNMPv2-MIB::eventText = STRING: "The process '<remote application>' has triggered too many log records in the last few minutes. Further messages will be logged at a decreased rate for 10 minutes."        CSAMC-SNMPv2-MIB::hostID = INTEGER: 209 CSAMC-SNMPv2-MIB::currentHostIPAddress = Wrong Type (should be IpAddress): STRING: "10.132.194.158"  CSAMC-SNMPv2-MIB::hostOSType = STRING: "W"      CSAMC-SNMPv2-MIB::sourcePort = Wrong Type (should be INTEGER): NULL  CSAMC-SNMPv2-MIB::destinationPort = Wrong Type (should be INTEGER): NULL        CSAMC-SNMPv2-MIB::eventType = STRING: "Administrative"       CSAMC-SNMPv2-MIB::ruleDescription = Wrong Type (should be OCTET STRING): NULLCSAMC-SNMPv2-MIB::ruleModuleID = Wrong Type (should be INTEGER): NULL   CSAMC-SNMPv2-MIB::ruleModuleName = Wrong Type (should be OCTET STRING): NULL CSAMC-SNMPv2-MIB::buttonCode = Wrong Type (should be INTEGER): NULL     CSAMC-SNMPv2-MIB::userName = STRING: "myDOMAIN\\WSecGat_Px"   CSAMC-SNMPv2-MIB::flags = INTEGER: 0
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (124679186) 14 days, 10:19:51.86       SNMPv2-MIB::snmpTrapOID.0 = OID: CSAMC-SNMPv2-MIB::csaTraps.1        CSAMC-SNMPv2-MIB::eventID = INTEGER: 10329637   CSAMC-SNMPv2-MIB::ruleID = INTEGER: 1374    CSAMC-SNMPv2-MIB::hostName = STRING: "hostA.prod.org"    CSAMC-SNMPv2-MIB::eventTime = STRING: "2012-04-05 15:38:29.999"      CSAMC-SNMPv2-MIB::severityCode = INTEGER: 2     CSAMC-SNMPv2-MIB::eventCode = INTEGER: 179      CSAMC-SNMPv2-MIB::processName = Wrong Type (should be OCTET STRING): NULL    CSAMC-SNMPv2-MIB::fileName = Wrong Type (should be OCTET STRING): NULL       CSAMC-SNMPv2-MIB::sourceIPAddress = Wrong Type (should be IpAddress): NULL      CSAMC-SNMPv2-MIB::destinationIPAddress = Wrong Type (should be IpAddress): NULL      CSAMC-SNMPv2-MIB::eventText = STRING: "The 'Service Control Manager' service logged event code 7036 into the system event log: The LiveUpdate service entered the running state. "   CSAMC-SNMPv2-MIB::hostID = INTEGER: 2206     CSAMC-SNMPv2-MIB::currentHostIPAddress = Wrong Type (should be IpAddress): STRING: "10.10.10.10"   CSAMC-SNMPv2-MIB::hostOSType = STRING: "W"      CSAMC-SNMPv2-MIB::sourcePort = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::destinationPort = Wrong Type (should be INTEGER): NULL CSAMC-SNMPv2-MIB::eventType = STRING: "NT Event log"CSAMC-SNMPv2-MIB::ruleDescription = ""   CSAMC-SNMPv2-MIB::ruleModuleID = INTEGER: 280   CSAMC-SNMPv2-MIB::ruleModuleName = STRING: "CSA Service Monitoring"  CSAMC-SNMPv2-MIB::buttonCode = Wrong Type (should be INTEGER): NULL     CSAMC-SNMPv2-MIB::userName = Wrong Type (should be OCTET STRING): NULL       CSAMC-SNMPv2-MIB::flags = INTEGER:

[Drainy]

Just wrote a massive reply and it doesn't fit in the box! Read my re-re-edited answer

[Ayn]

I don't think this is the transform that's applied to the sample data you provided, for that reason - I don't see any pipes in there either.

[cvajs]

sorry, i am on 4.3.1, my keyboard didnt get the .1 in there.

and hmmm, i dont see any pipe chars in my raw data that can be used for the transform, however, when i do a open search on my csa index i get lots of fields on the left with data (albeit the values for each field have more data in them then needed, but it is extracting fields somehow). the delimiter in my case looks like it should be a tab \t or \s{2,10}

[Drainy]

Also, if you are on v4.3 still you should consider upgrading to v4.3.1. There were some reasonably major issues in 4.3 that are resolved in 4.3.1; http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/4.3.1

[Drainy]

the \ is an escape character so the regex is saying;
^[^\|]+ <-- while character is not | keep eating up all characters. \| means | the character and not OR.

Ok, so if you look at this part;

([^\|]+)\|([^\|]+)\|([^\|]+)

Each part within brackets is a "group". The $1, $2 refers to group 1, group 2 etc. So everything within those brackets is what will be assigned to that group. Lets take the above snippet as an example. Lets say the data looked like this;

blah blah | foo | monkey!

Then the above regex would assign "blah blah " (note the whitespace caught at the end too) to the fieldname for $1. " foo " to the fieldname for $2 and so on. Anything outside of brackets is just to progress through an event, it may be that the last 3 sets of values enclosed by |'s are interesting so you skip through the previous ones without grouping.

EDIT AGAIN:
Hmm, just re-read your comment.. There must be a pipe character or something differentiating. Could you perhaps paste some example data? (Best way would be to edit your original question to paste it clearly as code)

EDIT EDIT AGAIN:
How well is the data being extracted? I just tried it in a test instance and found that Splunk automatically extracted some values (although not very good ones, just lots of Wrong and INTEGER). If thats what you're seeing then thats just Splunk trying to be clever and pull out Key/Value pairs. If you are sure something is extracting it then try the command, ./splunk cmd btool props list and also try transforms instead of props too to check for any other extractions that might be taking effect that look like they fit. (Saves looking through all the config files). If you stick a --debug on the end of that command it will also tell you what app is applying it

[cvajs]

i'll look into this further as suggested, but i did search all other transforms and props files looking for CSA related entries and only found related items in the CSA app folder, etc. and yes, the extractions i have do have "Wrong" and "INTEGER" included in the values, which leads me to believe i need to write my own transform regex/format to map the event data to variables. should not be too hard to do.

[Drainy]

Ah right, look at my edited answer for a fuller explanation (although give me a moment if you look now 🙂 )

[cvajs]

but how does the regex actually group real matches for the FORMAT statement? the groupings have a \| between thme, but my data has no | char so how does the groupings fall into matches?