Getting Data In

how can I use log filename to set "host" "date"

indeed_2000
Motivator

Hi I have log files like this:

log.machine03.20200310.bz2
log.machine04.20200310.bz2
log.machine05.20200310.bz2

these files copy each day to /opt , and splunk continuously monitor /opt.
now how can I use log filename to set "host" "date" of logs in splunk?

FYI: in every line of log file only store time NOT date! that's why I need to use date that exist in file name.
e.g.
file name = log.machine05.20200310.bz2
01:00:00 info logmessage
02:00:00 info logmessage
03:00:00 info logmessage
...

Thanks,

0 Karma

manjunathmeti
Champion

You can set host from file name in inputs.conf on forwarder.
inputs.conf

[monitor://<path>]
sourcetype = sourcetype_name
host_regex = [a-zA-Z]+\d+(?!$)

For setting _time from file name use INGET-EVAL. Add below configurations on indexer server.

props.conf

[sourcetype_name]
TRANSFORMS = timestamp_eval

trsansforms.conf

[timestamp_eval]
INGEST_EVAL = _time=strptime(replace(source, "/opt/log\.[\w]+\.", ""), "%Y%m%d.bz2")

Note: INGEST_EVAL is supported in splunk version >= 7.2.0.

0 Karma

indeed_2000
Motivator

Unfortunately I can’t use forwarders on Operation and logs copy manually to logserver.

0 Karma

manjunathmeti
Champion

You wrote splunk continuously monitor /opt. Which server is this? You need to put inputs.conf here.

0 Karma

indeed_2000
Motivator

On Splunk server I add /opt to index continuously.
Logs coming from other machines without Forwarders.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...