Hello
I have a syslog server which is being used to collect various network oriented data. For example if its a Aruba device then its location is /var/log/splunk/<deviceType>/<deviceName>/<devicename>.log
and for Cisco devices its /var/log/splunk/<deviceType>/<deviceName>/<devicename>.log
. What I have now is
host_segment=5
sourcetype =<deviceType>
and
host_segment=5
sourcetype = <deviceType>.
So if I have data from about 20-30 different devices I need to write the same number of stanza's in inputs.conf. To avoid this can I do segmentation based on sourcetype and host so that I can use just one stanza? Something like
host_segment=5
soucetype_segment=4
Any ideas?
Update: Below config testing is for index along with sourcetype extraction.
transforms.conf
[replace_index_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = index::$1
DEST_KEY = _MetaData:Index
[replace_sourcetype_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){4}/([^/]+)/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
props.conf
[replace_sourcetype_with_segment_5_from_source]
TRANSFORMS-replace = replace_sourcetype_with_segment_5_from_source
[replace_sourcetype_with_segment_5_from_source]
TRANSFORMS-replaceindex = replace_index_with_segment_4_from_source
inputs.conf
[monitor:///home/K23780/splunk/.../.../.../*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_5_from_source
#index= replace_index_with_segment_4_from_source
You could do something like this (amended to include index override):
inputs.conf
[monitor:///var/log/splunk/*/*/*/*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_4_and_index_with_segment_5_from_source
props.conf
[replace_sourcetype_with_segment_4_and_index_with_segment_5_from_source]
TRANSFORMS-replaceSourcetype = replace_sourcetype_with_segment_4_from_source
TRANSFORMS-replaceIndex = replace_index_with_segment_5_from_source
transforms.conf
[replace_sourcetype_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
[replace_index_with_segment_5_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){4}/([^/]+)/
FORMAT = $1
DEST_KEY = _MetaData:Index
That should take a look at your source's fourth segment and write it into your sourcetype, and write segment 5 into the index.
Looks like I have another issue in the same case. Now that the host_segment is extracting the host name, I am trying to modify the host name. As of now we are getting the hostname as host.company.com for all the devices. I am trying to just get the host value. I know I can do just the extract-host in props.conf which goes to the HF(where indexing is happening) but that doesn't seem to work.
My config is
[source::/var/log/splunk////.log]
EXTRACT-host = (?P
as I want it to be applied to any source under it. Any idea on where I am missing it
Works now. Had to send it to HF and Indexers for it to work. Thanks @martin_mueller for all your help.
It still send it to default @martin. Looks like it can't be done for index.
testing it. Its the other way around. index as 4,sourcetype as 5 and host as 6
I've updated my answer to load the sourcetype from segment 4, the index from segment 5, and the host from segment 6.
Any index you put into the inputs.conf is going to be overwritten by the transforms.conf. Your issue right now appears to be that the transforms.conf stanza isn't being executed.
It still fails for the index. Goes into default. Should I mention anything for the index in inputs.conf?
That updated props.conf defines TRANSFORMS-replace
twice for sourcetype replace_sourcetype_with_segment_5_from_source
, change one to TRANSFORMS-replaceIndex
Updated the original question with the configs.
So how does the inputs and props look? I shouldn't mention the index in inputs.conf file ?
By writing [foo]
in props.conf you specify properties for sourcetype "foo". As a result, your TRANSFORMS-replace = ...
is only applied to events with a sourcetype of "replace_index_with_segment_4_from_source". That doesn't match your events because their sourcetype is "replace_sourcetype_with_segment_5_from_source"
Isn't the replace just a stanza name
Or is it a function which is mentioning the source to extract the values. Both source type and index are from the source. Can you please share any documentation on how the replace function is working.
Yeah, that's not going to match your incoming events because that's looking for a sourcetype called replace_index_with_segment_4_from_source.
If you always want to grab the index from segment 4 for sourcetype replace_sourcetype_with_segment_5_from_source than you can drop that TRANSFORMS-replace into its transforms.conf stanza.
as [replace_index_with_segment_4_from_source]
TRANSFORMS-replace = replace_index_with_segment_4_from_source
How do you trigger the transforms.conf [replace_index_with_segment_4_from_source]
stanza from your props.conf?
[monitor:///var/log/splunk/.../.../.../*.log]
disabled = false
followTail = 0
host_segment=6
sourcetype = replace_sourcetype_with_segment_5_from_source
index= replace_index_with_segment_4_from_source
Do post the corresponding inputs.conf, I suspect you set index=replace... there while your stanza in transforms.conf is looking for sourcetype=...
I am doing it the same way as martin mentioned it for the sourcetype. But I get this error
Search peer aabbcc has the following message: received event for unconfigured/disabled/deleted index='replace_index_with_segment_4_from_source' with source='source::/var/log/splunk/test_web/Cisco/.../newtest01.log' host='host::abc' sourcetype='sourcetype::Cisco'
@martin_mueller, Can I do the same for the Index too. In case I add index at /var/log/splunk/test-index/.../.../.log as the path name. Would it be possible to even extract the index from it?
[replace_index_with_segment_4_from_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(?:/[^/]+){3}/([^/]+)/
FORMAT = index::$1
DEST_KEY = _MetaData:Index