Changing sourcetype of logs coming in as syslog

[slee75]

Hello, we currently have Apache logs coming into our Splunk instance via rsyslog. As such, the sourcetype of the apache logs is "syslog". Is there a way to change the sourcetype of just the apache logs to access_common and/or access_combined?

[skoelpin · Accepted solution]

I'm assuming your logging the logs and forwarding them with a UF on the syslog server? If so then you need to change the sourcetype in inputs.conf on the remote forwarder and restart it.. Keep in mind, you will lose all fields you created under the current sourcetype since fields are relative to sourcetype..

Whats the reason for wanting to change sourcetypes?

[slee75]

Thank you! We want to change sourcetypes so we can extract the httpd access log fields, ie source IP, etc. Is it possible to configure props.conf to change sourcetype based on process? I'm assuming that's created by the syslog field extraction, so I don't know if props.conf applies before or after the logs are indexed? I know the apache logs get the field "process" set as httpd, otherwise I'm not sure how to differentiate these logs from other linux syslog files (like from /var/log/messages)

[skoelpin]

Sorry about that, I had a typo.. The sourcetype is defined in inputs.conf on the forwarder.. Are you using the deployment server? If so then it will most likely be under $SPLUNK_HOME/etc/deployment-apps/<APP-NAME>/local/ on the DS.. It could also be under $SPLUNK_HOME/etc/system/local on the UF which is against Splunks best practices..

How long have you been indexing this data for? You could delete all the data from that previous sourcetype and re-index on the new sourcetype if it hasn't been a while.. If you have a ton of data, then you should probably not re-index and wait for the previous sourcetype to drop off.. You could also create custom field extractions under that sourcetype which can capture the source IP.. How many fields do you want to capture under your current sourcetype?

[slee75]

We're actually using rsyslogd to forward the logs on port 514, which is what makes it a bit tougher. I suppose we can create custom field extractions, those would apply to every log wouldn't they? Can we make it conditional? I come from an ELK background, and you could use logstash to parse based on conditions (if programname=httpd { mapping => mapping.json } for example)

[skoelpin]

Ah I see.. I would suggest you log the data and forward it via Universal Forwarder to Splunk. You're going to run into issues down the road if you forward the traffic directly to Splunk (You will lose data if need to restart Splunk)

You probably can assign the sourcetype conditionally, but it will require some tricky conf changes.. But I would suggest assigning sourcetype based on the "shape" of the data.. If both sourcetypes have the same "shape" then your only adding more admin work on your end to maintain it.. Also remember that line breaking and timestamp identification depends on sourcetype so you will need to account for this along with extracting fields.

In your case, I would suggest keeping your current sourcetype and apply extractions on your sourcetype

[skoelpin]

Did this answer solve your issue? If so can you accept it to close it out?