Getting Data In

Why is our props.conf configuration for our universal forwarder and clustered indexers not breaking events properly?

burras
Communicator

We have a few access log files from our SecureMedia application that we are attempting to ingest. I've been able to get most of them to ingest properly, but one particular sort of access log keeps giving me problems. The entries in the log look something like this:

Jun23-18:27:52.213 ESAM
<rpksmsresp><rc>0</rc><msg>OK</msg><type>Live</type></rpksmsresp>
Jun23-18:27:52.214 ESAM: sn=xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx add: --oatvra del: wait: 27 sec
Jun23-18:27:52.258 ESAM: v=3&sname=AUTOKEY&sn=xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx&huid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx&maxkeylen=2048&...

There is a Splunk Universal Fowarder installed on the system sending to our clustered indexers, all running 6.2.3.

When ingested into Splunk, all of these are combined into a single event instead of broken out into separate events. We have the following props.conf in place on both the forwarder and on the indexer cluster:

[sm-esam-access]
TIME_FORMAT=%b%d-%H:%M:%S.%N
TIME_PREFIX=^
TZ=UTC
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
disabled = false

I've played around with various applications of the props.conf stanza for this sourcetype - including and not including statements like BREAK_ONLY_BEFORE=^\w+\d\d-\d\d:\d\d:\d\d with no success. I've even pulled the log data up into a standalone instance of 6.2.3 and played with the "Add Data" section to try to get it to break properly. Problem is, when I do it this way it appears to break properly but the props.conf from that app doesn't work when put into production.

Any thoughts on what might be going on here or how to fix it?

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi burras,

these are only hints to give: You're doing linebreaking which is a parsing operation and always happens on the indexer or on a heavy weight forwarder (see the wiki on this topic http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F )
To troubleshoot this you can check at first if the stanza name in the props.conf exactly matches; is your sourcetype really called sm-esam-access?
Next check if any other props.conf is taking precedences over yours or if your is applied at all with btool:

$SPLUNK_HOME/bin/cmd btool props list --debug

or

$SPLUNK_HOME/bin/cmd btool props list sm-esam-access --debug

Next would be to check if any defined regex is matching or not and last but not least remember this props setting is only valid for any new incoming events.

Hope that helps ...

cheers, MuS

burras
Communicator

Thanks MuS. I verified that the stanza name matches up directly with the sourcetype. And yes, it is actually called sm-esam-access 🙂

Thanks for the link to the Wiki - I checked that out previously but hadn't had any luck getting the breaks to work correctly so that's how I ended up with props.conf in different locations.

I tried to run the btool commands you list to check props.conf precedences but it kept coming back with the following error: /opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory

0 Karma

MuS
SplunkTrust
SplunkTrust

run it as the user splunk or run this first source /opt/splunk/bin/setSplunkEnv to set all needing environment settings.

0 Karma

burras
Communicator

Okay, that fixed the run problem. The only props.conf I show on the indexers that contains the sourcetypes that I'm working on are the correct one:

[root@resvasplindex06 ~]# btool props list --debug |grep -v default | grep esam |more
/opt/splunk/etc/slave-apps/_cluster/local/props.conf [sm-esam-access]
/opt/splunk/etc/slave-apps/_cluster/local/props.conf [sm-esam-audit]
/opt/splunk/etc/slave-apps/_cluster/local/props.conf [sm-esam-error]

From what I can tell the one specifically for sm-esam-access looks okay as well (everything is either from cluster props.conf or from the default):

[root@resvasplindex06 ~]# btool props list sm-esam-access --debug
/opt/splunk/etc/slave-apps/_cluster/local/props.conf [sm-esam-access]
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/slave-apps/_cluster/local/props.conf BREAK_ONLY_BEFORE = ^\w+\d\d-\d\d:\d\d:\d\d
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/slave-apps/_cluster/local/props.conf NO_BINARY_CHECK = true
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/slave-apps/_cluster/local/props.conf SHOULD_LINEMERGE = false
/opt/splunk/etc/slave-apps/_cluster/local/props.conf TIME_FORMAT = %b%d-%H:%M:%S.%N
/opt/splunk/etc/slave-apps/_cluster/local/props.conf TIME_PREFIX = ^
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/slave-apps/_cluster/local/props.conf TZ = UTC
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/slave-apps/_cluster/local/props.conf disabled = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf sourcetype =

I also reverified that the stanzas in props.conf match up to the stanzas in inputs.conf:
[root@revaapp01 local]# cat inputs.conf |grep esam
[monitor:///opt/securemedia/var/app-8082/logs/esam-access.log]
sourcetype = sm-esam-access
[monitor:///opt/securemedia/var/app-8082/logs/esam-audit.log]
sourcetype = sm-esam-audit
[monitor:///opt/securemedia/var/app-8082/logs/esam-error.log]
sourcetype = sm-esam-error

0 Karma

MuS
SplunkTrust
SplunkTrust

props.conf and transforms.conf troubleshooting is hard I know and mostly like the hardest thing is that you must figure out what's wrong, because I can only provide some basic hints like these:

http://answers.splunk.com/answers/4075/whats-the-best-way-to-track-down-props-conf-problems.html
http://docs.splunk.com/Documentation/Splunk/6.2.3/Troubleshooting/Enabledebuglogging

0 Karma

burras
Communicator

Thanks MuS - I'll continue to dig into it and let everyone know what I find...

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...