Getting Data In

stop splunk from splitting a file.

jstillwell
Explorer

I'm trying to read some config files into splunk, ala change management. I'm not using fschange, I'm using a tcp monitor, and pushing them into splunk from different hosts.

I use a ***SPLUNK*** header to set source, sourcetype, host properly. As that info isn't in the files anywhere. Those values get picked up fine.

Some of the files are left as one huge event, as they should be. Other files are getting split by line, which is what I don't want. How can I tell splunk I don't want a sourcetype split up at all. One tcp connection = one event.

I've tried various combinations of things.

[database_config]
LINE_BREAKER = disable_linebreaking_with_impossible_regex
BREAK_ONLY_BEFORE = disable_linebreaking_with_impossible_regex
LEARN_MODEL = false
SHOULD_LINEMERGE = false

addition info.

I tried rsyncing the files over and using fschange (for the fullEvent parameter), but that was worse, I got the same line breaking results, but couldn't override the sourcetype using conf files. And was left with 20 random ones splunk invented, for the same file type. I tried a large volume of different configurations on multiple different OSes.

btool reports nothing is overriding my sourcetype definition.

I don't trust what splunk test sourcetype says. When I give it a file it ends up creating stuff in the 'learned' app, which doesn't happen when I import via tcp.

splunk is doing splitting fine for the logs I push into it.

Tags (3)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You need to increase/set TRUNCATE and MAX_EVENTS to be large enough to fit your entire file.

Also, a regex that will be guaranteed to never match is (?!) that is open-paren, question-mark, exclamation-mark, close-paren. An equivalent is (*FAIL)

jstillwell
Explorer

plain TCP input port. no forwarders anywhere. I just have a [tcp://:port] stanza in inputs.conf, and homebrew scripts that connect and send the data into that port.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

And if using forwarders, are these light or heavy forwarders?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Can you please clarify, are you using Splunk forwarders to monitor files, or are you sending data in to a plain TCP input port? Or really, what do you mean by "using a tcp monitor"?

0 Karma

jstillwell
Explorer

This doesn't work. I suspect splunk is ignoring my sourcetype stanza in props.conf. I don't know how to verify this.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...