Getting Data In

props & transforms not taking effect

ltrand
Contributor

I am in desperate need to figure out what I'm doing wrong with this props config. Currently I am bringing in logs via syslog-ng that then get written to disk on a heavy forwarder. I push several configs to the HF's, in one of them I've specified the following:

App: global_hfs_props/local/

props.conf:

[source::/syslog/AIRWATCH/*/*.log]
LINE_BREAKER = (|)
TRANSFORMS-airwatch= airwatch_parsing

transforms.conf:

[airwatch_parsing]
# Attempt to grab all fields in one regex
REGEX = ((Event Type:)\s)(?<Event_Type>([^ ]*))(\s(Event:)\s)(?<Event>([^ ]*))(\s(User:)\s)(?<User>([^ ]*))(\s(Event Source:)\s)(?<Event_Source>([^ ]*))(\s(Event Module:)\s)(?<Event_Module>([^ ]*))(\s(Event Category:)\s)(?<Event_Category([^ ]*))(\s(Event Data:)\s)(?\s)(?<Event_Data>([^ ]*))

When on the HF I run the following command:

/opt/splunk $ bin/splunk test sourcetype /syslog/AIRWATCH/IPADDR/IPADDR.log
Using logging configuration at /opt/splunk/etc/log-cmdline.cfg.
PROPERTIES OF /syslog/AIRWATCH/IPADDR/IPADDR.log
PropertiesMap: {ANNOTATE_PUNCT -> 'True' BREAK_ONLY_BEFORE -> '' BREAK_ONLY_BEFORE_DATE -> 'True' CHARSET -> 'UTF-8' DATETIME_CONFIG -> 'CURRENT' HEADER_MODE -> '' LEARN_SOURCETYPE -> 'true' LINE_BREAKER_LOOKBEHIND -> '100' MAX_DAYS_AGO -> '2000' MAX_DAYS_HENCE -> '2' MAX_DIFF_SECS_AGO -> '3600' MAX_DIFF_SECS_HENCE -> '604800' MAX_EVENTS -> '256' MAX_TIMESTAMP_LOOKAHEAD -> '32625' MUST_BREAK_AFTER -> '' MUST_NOT_BREAK_AFTER -> '' MUST_NOT_BREAK_BEFORE -> '' SEGMENTATION -> 'indexing' SEGMENTATION-all -> 'full' SEGMENTATION-inner -> 'inner' SEGMENTATION-outer -> 'outer' SEGMENTATION-raw -> 'none' SEGMENTATION-standard -> 'standard' SHOULD_LINEMERGE -> 'False' TRANSFORMS -> '' TRANSFORMS-force-sourcetype_for_cisco_devices -> 'force_sourcetype_for_cisco_pix, force_sourcetype_for_cisco_asa, force_sourcetype_for_cisco_wap, force_sourcetype_for_cisco_fwsm, force_sourcetype_for_cisco_acs, force_sourcetype_for_cisco_ios, force_sourcetype_for_cisco_catchall' TRUNCATE -> '10000' detect_trailing_nulls -> 'false' is_valid -> 'True' maxDist -> '500' sourcetype -> 'IPADDR'}

I know test has been deprecated, but I don't know a better command to check to see if props or transforms are being applied to a specific file. Am I perhaps doing this wrong? I see that it's getting transforms from a props that doesn't specify a source to apply the transforms to, but I don't see when I specify the source that it gets applied.

I'm hoping that this gets me the fields I need & I want to have the "|" used as a end of message character since the logs sometimes run together.

-Thanks!

0 Karma
1 Solution

lguinn2
Legend

Okay, you are sending data to a heavy forwarder (HF) and you expect it to parse your data. So far, so good. BUT

You shouldn't be doing field extractions at index time. Use search time field extractions. Search time field extraction specifications belong on the indexer (or the search head, if you have one).

Next, you can do the field extractions as you have it by just using props.conf:

[source::/syslog/AIRWATCH/*/*.log]
EXTRACT-everything = ((Event Type:)\s)(?<Event_Type>([^ ]*))(\s(Event:)\s)(?<Event>([^ ]*))(\s(User:)\s)(?<User>([^ ]*))(\s(Event Source:)\s)(?<Event_Source>([^ ]*))(\s(Event Module:)\s)(?<Event_Module>([^ ]*))(\s(Event Category:)\s)(?<Event_Category([^ ]*))(\s(Event Data:)\s)(?\s)(?<Event_Data>([^ ]*))

But I think there is an easier way to do it! On the indexer, put the following in props.conf

[source::/syslog/AIRWATCH/*/*.log]
REPORT-aw=extract_airwatch

and in transforms.conf

[extract_airwatch]
DELIMS=":"
FORMAT = $1::$2

Now you don't have to write such a complicated regular expression. Finally, the LINE_BREAKER needs to stay on the HF.

Finally, when you change props.conf and transforms.conf on a HF, you need to restart it for the changes to take effect. Search time field extractions do not require a restart. However, it is a good idea to explicitly request a reload of the conf files by doing this

http://yourservername:8000/debug/refresh

assuming that your SplunkWeb UI is running on port 8000...

View solution in original post

lguinn2
Legend

Okay, you are sending data to a heavy forwarder (HF) and you expect it to parse your data. So far, so good. BUT

You shouldn't be doing field extractions at index time. Use search time field extractions. Search time field extraction specifications belong on the indexer (or the search head, if you have one).

Next, you can do the field extractions as you have it by just using props.conf:

[source::/syslog/AIRWATCH/*/*.log]
EXTRACT-everything = ((Event Type:)\s)(?<Event_Type>([^ ]*))(\s(Event:)\s)(?<Event>([^ ]*))(\s(User:)\s)(?<User>([^ ]*))(\s(Event Source:)\s)(?<Event_Source>([^ ]*))(\s(Event Module:)\s)(?<Event_Module>([^ ]*))(\s(Event Category:)\s)(?<Event_Category([^ ]*))(\s(Event Data:)\s)(?\s)(?<Event_Data>([^ ]*))

But I think there is an easier way to do it! On the indexer, put the following in props.conf

[source::/syslog/AIRWATCH/*/*.log]
REPORT-aw=extract_airwatch

and in transforms.conf

[extract_airwatch]
DELIMS=":"
FORMAT = $1::$2

Now you don't have to write such a complicated regular expression. Finally, the LINE_BREAKER needs to stay on the HF.

Finally, when you change props.conf and transforms.conf on a HF, you need to restart it for the changes to take effect. Search time field extractions do not require a restart. However, it is a good idea to explicitly request a reload of the conf files by doing this

http://yourservername:8000/debug/refresh

assuming that your SplunkWeb UI is running on port 8000...

ltrand
Contributor

Thanks for the help! Moving it to the search head absolutely soved my problem. The $1::$2 breaks because of the multiword key values, so I get weird entries. So I stuck with the ugly regex I wrote.

The only issue I am still having though is that the line breaker is still not applying, but I am trying to work through it.

0 Karma

nychawk
Communicator

ltrand;

I ran across this posting while searching for AirWatch TA's and/or apps.

Can you share what you've done?

Thank you in advance.

0 Karma

j0shrice
Path Finder

Also looking for AirWatch TA with CIM mapping

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...