Splunk Search

Why is my custom sourcetype configuration not extracting the host field?

lisaac
Path Finder

I have a file with data similar to the following:

2015:09:01:15:00:00.005 sl200services007 3:INFO SERVER NOFMT 4327963992431091696812 Saving to client_request_map table

I have a custom sourcetype called services:client, and I have set up an index-time extracttion for the host field.

inputs.conf

[monitor://C:\temp\sample_logs2.txt]
disabled = false
index = test
sourcetype = services:client

props.conf

[services:client]
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = trans services
pulldown_type = true
TRANSFORMS-host = hostoverride2

transforms.conf

[hostoverride2]
DEST_KEY = MetaData:Host
REGEX = ^[^\s]+\s([^\s]+)
FORMAT = host::$1

For testing, I stop splunk, append data to the monitored file, and I start splunk. Splunk is indexing the data fine, but the host field isn't set based on the regex result. I tested the regex on https://regex101.com/, and I followed the override syntax from http://blogs.splunk.com/2008/04/16/overriding-default-syslog-host-extraction/. Any thoughts on why this isn't working to set the hostname?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Try this in transforms.conf instead:

[hostoverride2]
REGEX = ^[^\s]+\s([^\s]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

I'm not certain the order maters but it's worth a shot. Everything else looks good to me.

0 Karma

lisaac
Path Finder

I tried the suggestion to no avail.

[hostoverride2]
REGEX = ^[^\s]+\s([^\s]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

0 Karma

tskinnerivsec
Contributor

Also, are both the props.conf and transforms.conf contained in the same configuration app with no weird default.meta stanzas applied?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Usually yes, but they dont have to be in the same app.

I tend to do apps like this

/ta-appnameInputs/
/ta-appnameProps/
/ta-appnameTransforms/

which is a bit overkill and usually i combine props & transforms into just "props". Reason being inputs can go on forwarders, props and transforms can be on indexers, or heavy forwarders, some usage on universal forwarders, etc. I do this in distributed environments that employ a deployment server.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You could go crazy and have both search and index time transformations in different apps, because search configs should only be on search heads and index time configs should only be on ingestion.

0 Karma

lisaac
Path Finder

I think a ticket with support might be in order, or a test on a Linux host vs. Windows. Debug output shows the source correctly settting the transforms-host correctly.

C:\Program Files\Splunk\etc\apps\search\local\props.conf [source::...sample_logs2.txt]
C:\Program Files\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
C:\Program Files\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True
C:\Program Files\Splunk\etc\system\default\props.conf CHARSET = AUTO
C:\Program Files\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml
C:\Program Files\Splunk\etc\system\default\props.conf HEADER_MODE =
C:\Program Files\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
C:\Program Files\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_AGO = 2000
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
C:\Program Files\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
C:\Program Files\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
C:\Program Files\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\system\default\props.conf SHOULD_LINEMERGE = True
C:\Program Files\Splunk\etc\system\default\props.conf TRANSFORMS =
C:\Program Files\Splunk\etc\apps\search\local\props.conf TRANSFORMS-host = hostoverride2
C:\Program Files\Splunk\etc\system\default\props.conf TRUNCATE = 10000
C:\Program Files\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
C:\Program Files\Splunk\etc\system\default\props.conf maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf priority =
C:\Program Files\Splunk\etc\system\default\props.conf sourcetype =

0 Karma

tskinnerivsec
Contributor

Didn't realize this was windows indexer, I may definitely be worth checking with support . . ., this sort of stuff just seems to work on linux splunk servers.

0 Karma

jkat54
SplunkTrust
SplunkTrust

you might also try to change some small details...

like TRANSFORMS-host to TRANSFORMS-customhost

to rule out any "internal" stuffs related to "host".

0 Karma

tskinnerivsec
Contributor

I like the idea of changing the TRANSFORMS name, I have definitely gotten bitten in the past buy using reserved words in stanza names before and "host" has a high probability of that.

0 Karma

jkat54
SplunkTrust
SplunkTrust

also look for errors under index=_internal source=splunkd

0 Karma

tskinnerivsec
Contributor

Since you are trying to over write an indexed field, you probably need to catch it on the indexer at that source level. So try something like this in your props.conf:

[source::…sample_logs2.txt]
TRANSFORMS-host = hostoverride2

instead of :

[services:client]
TRANSFORMS-host = hostoverride2

0 Karma

lisaac
Path Finder

Still no go, I removed transforms reference from the sourcetype and added the source based on the example. The ...sample_logs2.txt will catch any file ending in sample_logs2.txt.

My props.conf value has the following:

[cybs:trans_svs]
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
description = cybs trans services
pulldown_type = true

[source::...sample_logs2.txt]
TRANSFORMS-host = hostoverride2

My splunk instance is a single windows 7 host running 6.2.5.

0 Karma

lisaac
Path Finder

I have a file on the indexer, since this is just a test situation. This is from props and transforms on the single test indexer.

0 Karma

tskinnerivsec
Contributor

Is that props.conf and transforms.conf deployed on your indexers? (These should be on your indexers, or heavy forwarders, if data is being sent through them first. )

If so, you may also want to run this on your indexers to verify what props stanzas are being applied to your indexers:

./splunk btool list props

you can pipe this to a file and review it to verify what the indexer thinks the total applied configuration is.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...