Getting Data In

Help with wildcard inputs issues

a212830
Champion

Hi,

I have some new inputs configured with wildcards and whitelists, but they aren't pulling in the data.

The path to the files is: /xt112qdal0/log/app/XTRAC/6.9.1.1/XTRAC6.9.1.1_ClusterMbr1/xtrac.error.log

(Note that this paticular file can exist in many different sub-directories off the ..log parent, hence the wildcard)

I've confirmed that the filed is readable. I can see the connection being made to the indexer, so, I'm assuming that it's my inputs:

[monitor:///xt*qa*/log/]
recursive = Yes
index = Xtrac_ceops_qa_pm_logs
sourcetype = xtrac_error
followTail = 0
disabled = 0
whitelist = xtrac.error.log
crcSalt =

Is something wrong with this setup?

Tags (2)
0 Karma

lguinn2
Legend

Aha - this is the problem. You cannot have multiple stanzas with the same monitor. Do this instead:

inputs.conf

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = xtrac\.(error|trace|perf)\.log$

props.conf

[source::.../xtrac.error.log]
sourcetype = xtrac_error

[source::.../xtrac.perf.log]
sourcetype = xtrac_perf

[source::.../xtrac.trace.log]
sourcetype = xtrac_trace

Here is a reference in the Getting Data In manual for more information.

0 Karma

a212830
Champion

Thank you, this is fantastic. I'm noticing that the splunkd is recommending that I set a crcsalt setting, because the error log is small. In this case, since they are all using the same inputs stanza, how would I do that?

0 Karma

lguinn2
Legend

To do that, add the following line to your inputs.conf stanza:

crcSalt=<SOURCE>

This tells Splunk to consider the full path name of your file, as well as the contents of the file, to determine if a file is unique. (Splunk looks at the contents of each file to determine if it has already indexed the data.)

0 Karma

a212830
Champion

Thanks. I was aware of crcSalt, but I'm never quite sure if it should be applied to all sources, or just certain ones.

That said, is this valid? I'm going to have a lot of files for this feed, so I'm concerned that the whitelist will reach a size limit.

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = xtrac.(error|trace|perf).log$

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = xwb.(error|trace|perf).log$

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = fuse.log$

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = TimerManager.log$

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = System(Err|Out).log$

0 Karma

lguinn2
Legend

Wait!! You CANNOT have monitor stanzas that overlap. You have 5 identical monitor stanzas - this will not work!

Don't worry about the length of the whitelist...

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = (xtrac|xwb).(error|trace|perf).log$|fuse.log$|TimeManager.log$|System(Err|Out).log$

is not that long. But you could add more information into the monitor stanza as well:

[monitor:///xt*q*/log/.../fuse.log]

for example.

0 Karma

a212830
Champion

Thanks - yes, I realized that wouldn't work.

The whitelist could get quite large - hence my concern. Would the example above work for a fuse.log that exists in any directory off ../log? The number and level of subdirectories varies - could be 2, could be 5... - want to make sure it would work.

0 Karma

lguinn2
Legend

Try this

[monitor:///xt*qa*/log/]
index = Xtrac_ceops_qa_pm_logs
sourcetype = xtrac_error
whitelist = xtrac.error.log

You don't need most of the settings, as you are using the defaults. Plus recursive = Yes is not a valid setting.

Finally, in your path example, the first directory name is tx112qdal0 and in the regular expression, you are missing the d between the q and the a. This may be the real problem...

0 Karma

a212830
Champion

Thanks. Picked up the typo, and it found one of the files(trace), but not the others, even after making your recommended changes. The full inputs is the following:

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
sourcetype = xtrac_error
whitelist = xtrac.error.log

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
sourcetype = xtrac_perf
whitelist = xtrac.perf.log

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
sourcetype = xtrac_trace
whitelist = xtrac.trace.log

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...