Getting Data In

Routing and Filtering question

yongly
Path Finder

Hi all,

I've come across a strange problem that I can't seem to figure out how to fix or troubleshoot. My problem is that for some reason, I can't seem to get my source or host recognised in the filter. I have a default discard_all rule that discards all logs sent to my filter server unless I define another stanza or rule to specifically handle those log files:

props.conf

[default]
TRANSFORMS-drop_all=discard_all

For some reason it ignores my source and host stanzas
[source::/var/log/nginx/access.log]
TRANSFORMS-ccp=allow_all

I have a filter set up with these entries in transforms.conf

[discard_all]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

Use this transform to allow and forward all entries from log file to indexer
[allow_all]
REGEX=.
DEST_KEY=queue
FORMAT=indexQueue

I know that when I change my default rule to allow_all. the file comes through to the indexer. I'm stumped because other log files seem to work fine.

Any ideas?

0 Karma

kristian_kolb
Ultra Champion

Hm..why even monitor the files if you're going to discard most of them..?

Well, perhaps you have to specify both transforms on the same line, like;

[source::/var/log/nginx/access.log] 
TRANSFORMS-ccp = discard_all, allow_all

In this case it seems pretty silly, but perhaps you have more clever filters elsewhere.

/K

0 Karma

yongly
Path Finder

Yeh after some testing, I found that I had to remove it to get it to recognise the [source:..] stanza.

What I don't understand is why it worked with other sources and sourcetypes but not with this one?

0 Karma

kristian_kolb
Ultra Champion

did you remove the [default] discard transform?

0 Karma

yongly
Path Finder

Well this is an intermediate server that we've been using for filtering. The idea is to keep control of what gets passed onto the indexer to avoid big files getting through and exceeding our licence. Hence a default discard and an explicit allow 🙂

I did wonder if another filter or stanza was picking it up and taking precedence but when I change the [default] to allow_all, the file comes through no problems.. this kind of suggests that for some reason it's not linking the access.log file and the stanza in props.conf.

I did try your suggestion anyway, but no luck. Any other ideas as to how I might troubleshoot this?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...