Deployment Architecture

sourcetype override on indexer cluster

eddiet
Explorer

Hi guys,
We have a clustered 6.5.3 deployment and following this doc to re-assign a new sourcetype to squid access logs but to no success.
The squid log is in the default log format as per https://wiki.squid-cache.org/Features/LogFormat

The sourcetype is currently access-too_small.

I don't have control of the UFs to update its inputs.conf so taking this approach for now.

Re-assigning all events so regex is .*. I have seen . as well .*. Which is the correct syntax to match all events?

Here are the props and transforms on the indexers:

/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf       [squid_override_sourcetype]
/opt/splunk/etc/system/default/transforms.conf                         CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf                         CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf                         DEFAULT_VALUE = 
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf       DEST_KEY = MetaData:Sourcetype
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf       FORMAT = sourcetype::squid
/opt/splunk/etc/system/default/transforms.conf                         KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf                         LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf                         MV_ADD = False
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/transforms.conf       REGEX = .*
/opt/splunk/etc/system/default/transforms.conf                         SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf                         WRITE_META = False


/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/props.conf       [source::...squid/access.log]
/opt/splunk/etc/system/default/props.conf                         ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf                         AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE = 
/opt/splunk/etc/system/default/props.conf                         BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf                         CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf                         DATETIME_CONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf                         HEADER_MODE = 
/opt/splunk/etc/system/default/props.conf                         LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf                         LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf                         LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf                         MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf                         MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf                         MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf                         MAX_EVENTS = 256
/opt/splunk/etc/system/default/props.conf                         MAX_TIMESTAMP_LOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf                         MUST_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_AFTER = 
/opt/splunk/etc/system/default/props.conf                         MUST_NOT_BREAK_BEFORE = 
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf                         SEGMENTATION-standard = standard
/opt/splunk/etc/system/default/props.conf                         SHOULD_LINEMERGE = True
/opt/splunk/etc/system/default/props.conf                         TRANSFORMS = 
/opt/splunk/etc/slave-apps/Splunk_TA_squid/local/props.conf       TRANSFORMS-sourcetype = squid_override_sourcetype
/opt/splunk/etc/system/default/props.conf                         TRUNCATE = 10000
/opt/splunk/etc/system/default/props.conf                         detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf                         maxDist = 100
/opt/splunk/etc/system/default/props.conf                         priority = 
/opt/splunk/etc/system/default/props.conf                         sourcetype = 

I have also tried using the [host::redacted_hostname] and [access-too_small] stanzas in props but to no avail.

Cluster automatically did a rolling restart on initial app push, but subsequent updates to props/transforms didn't. Did a rolling restart just in case as well.

Having read a lot of related posts here I see people having success so I know its an oversight on my side so any pointers would be much appreciated.

The settings shown above are on the first indexers receiving the events from the UF where logs are collected.
The instance collecting the logs is a true UF, not a full splunk enterprise install

0 Karma

woodcock
Esteemed Legend

To do a true sourcetype override like you think you need, you need to deploy the changes to the first Heavy Forwarder or Indexers that receives the events. If you installed full Splunk Enterprise on your "UF", then it is not a UF, it is HF and you need to deploy there. But maybe you do not need to do this at all. Have you considered a sourcetype rename (happens on the Search Heads at search time)?

https://docs.splunk.com/Documentation/Splunk/latest/Data/Renamesourcetypes

0 Karma

eddiet
Explorer

The settings shown in my first post are on the first indexers receiving the events.
The instance collecting the logs is a true UF, not a full splunk enterprise install.

0 Karma

thambisetty
SplunkTrust
SplunkTrust

Hi,

Thanks for the link. just got to know that there is an option to rename sourcetype at search time.

————————————
If this helps, give a like below.
0 Karma

thambisetty
SplunkTrust
SplunkTrust

Hi @eddiet,

Try something like below,

in local folder create props.conf and use host/source to identify events & override sourcetype as mentioned below,

props.conf
[host::redacted_hostname]
sourcetype = squid_override_sourcetype
————————————
If this helps, give a like below.
0 Karma

eddiet
Explorer

Thanks but I did try this, didn't work then realised that it only takes effect on the forwarding instance, not on the receiver.

0 Karma

thambisetty
SplunkTrust
SplunkTrust

props.conf and transforms.conf can't be configured in Universal forwarder. so it should take effect on either Heavy forwarder/Indexer.

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...