I applied the below settings to parse data from 47 equallogic hosts, but I stopped get any logs altogether. Whereas, prior to these settings, I was getting logs but they were getting parsed into cisco:ios sourcetype.
transforms.conf
[set_sourcetype_equal_log_for_Equallogic_hosts]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?).\w+\s+)[?(195.235.128.16\d|195.165.203.\d\d|195.165.201.21|10.40.40.\d)[\w.-]]?\s
FORMAT = sourcetype::equal_log
DEST_KEY = MetaData:Sourcetype
props.conf
[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts
Additionally, I even successfully tested the regex from regex101.com as well as by using the below command to check in Splunk
[index=* | head 1 | eval testdata="123456a" | regex testdata="^.{5,6}(?
The best way to set the sourcetype is in inputs.conf
The best way to do regex on the host field is a transforms with SOURCE_KEY = MetaData:Host
The best way to set the host field is in inputs with a reference to a directory where the host information is stored.
Complicated regex is never recommended. And when regex can't be avoided, it can usually be simplified a lot. If the first three points have received due diligence, a link to the regex101 test plus the primary objectives of the regex should be provided. Currently, it appears that regex is not necessary.
Hi @landen99,
Thanks for your input.
considering 1st option, would the inputs.conf be in \etc\system\local and look like below ?
[udp://X.X.X.X:514]
index = main
sourcetype = equal_log
disabled = 0
in above case, would specifying all 47 IPs work ? I cant use wildcards because the IP range is very discontiguous in nature.
Would the 2nd option look something like this ?
transform.conf
[set_sourcetype_equal_log_for_Equallogic_hosts]
REGEX = 199.299.129.19\d|199.169.205.\d\d\d|199.165.200.22|20.90.90.\d
SOURCE_KEY = MetaData:Host
FORMAT = sourcetype::equal_log
DEST_KEY = MetaData:Sourcetype
I am not sure how the 3rd option is implemented. Can you please provide an example ?
Your inputs can be there, but it is better in an app for organization/management.
Your transform extracts nothing and tries to use a list of matches, which is always bad without a good anchor.
I don't understand what you mean about my 3rd option. I was providing guidance not options.
Hi @landen99, can you please help clarify the above solution ?
Thanks
So you want to match all of those events and set them as a different source type?
That regex does not match those events at all, I would say the best way to do that would be to send the syslogs on a different port and setup a new input to a new sourcetype.
rather than that you could make a different regex.
It's a little messy, but something like this should match everything starting with that format:
REGEX = \w+\s\d+\s\d+:\d+:\d+\s\d+.\d+.\d+.\d+\s\d+:\d+:\w+:\d+-\w+-\d+\s\d+:\d{2}:\d{2}.\d{6}:
Like i said, changing the UDP port for the syslog would be the best bet, like port 4500 or something and then make a new input for UDP 4500 that goes to the new sourcetype.
Try
\w+\s\d+\s\d+\d+:\d+:\d+\s(23.34.35.80|27.34.35.80)\s\d+:\d+:\w+:\d+-\w+-\d+\s\d+:\d{2}:\d{2}.\d{6}:
you could replace the addresses.
Maybe there could be a better solution than that though.
Hi @agarrison,
I cannot set a new udp port as Equallogic has no provision to specify a particular port. It sends on 514 by default. No other option than sourctype overriding.
I tried your regex but it parsed unexpected event as well. Example below,
Oct 20 06:53:26 x.x.x.x 59368:8696:agent:20-Oct-2017 06:53:25.653704:agent.c:1866:AUDIT:Unknown:22.7.1:SNMP packet validation failed, request received from x.x.x.x
Oct 20 06:00:37 x.x.x.x 2130:127:VolExec:20-Oct-2017 06:00:36.190088:VE_VolSetWorker.hh:151:WARNING::43.3.5:Volume MAIL has used 1 percent of its local replication reserve. If the in-use space exceeds the local replication reserve (set to 5 percent of the volume reserve), the group will cancel any in-progress replication for the volume.
The easiest solution would be to call the built-in "syslog-host" TRANSFORM, in addition to your "changesourcetype" TRANSFORM.
Hi @sowings,
did you mean the below way ?
[set_sourcetype_syslog_for_Equallogic_hosts]
REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?).\w+\s+)[?(host1|host2|host3|host4|host5|host6)[\w.-]]?\s
FORMAT = sourcetype::equal_log
DEST_KEY = MetaData:Sourcetype
[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts
TRANSFORMS-changesourcetype = syslog-host
No; by having two stanzas with the same LHS (left-hand side) you introduce a collision, and Splunk must choose one of them to pick. IIRC it picks the first one it sees. In my case I was referring to something like this:
[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts, syslog-host
Transforms listed on the RHS (right-hand side) of a props.conf stanza are called in the order they appear, left to right. This means that your custom transform would be called first, then syslog-host after. Depending upon the structure of your logs, you may wish to reverse the order.
Close: The syntax you've provided would actually overwrite the settings (because the "TRANSFORMS-changesourctype" string is the same). Try this instead:
[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_equal_log_for_Equallogic_hosts, syslog-host
I tested it, it started parsing even the cisco:ios logs into equal_log sourcetype, which I guess, is due to including the default syslog-host transforms.
No, the "syslog-host" transform only attempts to change the 'host' field to the value found from the syslog preamble. If the logs are changing to the equal_log sourcetype for Cisco events, too, then your regex might need some tweaking.
So I created a regex and successfully tested it on regex101.com.
Regex = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?).\w+\s+)[?(x.x.x.2\d|x.x.x.1\d)[\w.-]]?\s
I used the above regex, and raw logs from Splunk. However, I noticed 2 main things that
there were some logs that that had no host IP in it and Splunk split the same event into two.
Example:
10/19/17
12:36:05.000 PM Oct 19 12:36:04 x.x.x.x 344237:342551:MgmtExec:19-Oct-2017 12:36:02.821184:targetAttr.cc:582:ERROR::7.4.3:iSCSI login to target 'x.x.x.x:3260, abc.2000-05.com.equallogic:8-jhduyc76-cjdbhc-jhd6hjs-dj78-iscsi-raid50-datastore8' from initiator 'x.x.x.x:49171, aaa.1000-05.com.microsoft:abc-netvault.mail.com' failed for the following reason:
10/19/17 Requested target not found.
12:36:05.000 PM
Do you have some sample events?
Something like:
\w+\s\d+\s\d+\d+:\d+:\d+\s(23.34.35.80|27.34.35.80)\s\d+:\d+:\w+:\d+-\w+-\d+\s\d+:\d{2}:\d{2}.\d{6}:
with a list of addresses, 23.34.35.80 and 27.34.35.80 being placeholders, you could add more. This is really not a great solution however.
Oct 18 08:49:31 X.X.X.80 251188:63:netmgtd:18-Oct-2017 08:49:29.341224:rca_ocptcp.c:655:AUDIT:grpadmin:25.7.4:GUI: Account grpadmin from X.X.X.X to X.X.X.8 logged out.
Sourcetype =equal_log
Oct 18 07:43:01 X.X.X.80 251003:61:netmgtd:18-Oct-2017 07:42:59.604422:rca_ocp.c:1372:AUDIT:grpadmin:25.7.3:GUI: Account grpadmin logged in from X.X.X.X to X.X.X.8, using local authentication. User privilege is group-admin.
Sourcetype =equal_log
Oct 18 06:00:52 X.X.X.62 2109:125:VolExec:18-Oct-2017 06:00:50.382206:VE_VolSetWorker.hh:151:WARNING::43.3.5:Volume MAIL has used 1 percent of its local replication reserve. If the in-use space exceeds the local replication reserve (set to 5 percent of the volume reserve), the group will cancel any in-progress replication for the volume.
Sourcetype =cisco:ios
Above is the latest sample event.
From the above events, I have noticed that its parsing correctly the event from equallogic, however, weird thing is I had put a host which ends in x.x.x.8, however, this log from host x.x.x.80 also seems to have parsed in sourcetype :equal_log. How did this happen ? I didnt event mention any wildcard.