Getting Data In

stop splunk from splitting a file.

jstillwell
Explorer

I'm trying to read some config files into splunk, ala change management. I'm not using fschange, I'm using a tcp monitor, and pushing them into splunk from different hosts.

I use a ***SPLUNK*** header to set source, sourcetype, host properly. As that info isn't in the files anywhere. Those values get picked up fine.

Some of the files are left as one huge event, as they should be. Other files are getting split by line, which is what I don't want. How can I tell splunk I don't want a sourcetype split up at all. One tcp connection = one event.

I've tried various combinations of things.

[database_config]
LINE_BREAKER = disable_linebreaking_with_impossible_regex
BREAK_ONLY_BEFORE = disable_linebreaking_with_impossible_regex
LEARN_MODEL = false
SHOULD_LINEMERGE = false

addition info.

I tried rsyncing the files over and using fschange (for the fullEvent parameter), but that was worse, I got the same line breaking results, but couldn't override the sourcetype using conf files. And was left with 20 random ones splunk invented, for the same file type. I tried a large volume of different configurations on multiple different OSes.

btool reports nothing is overriding my sourcetype definition.

I don't trust what splunk test sourcetype says. When I give it a file it ends up creating stuff in the 'learned' app, which doesn't happen when I import via tcp.

splunk is doing splitting fine for the logs I push into it.

Tags (3)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You need to increase/set TRUNCATE and MAX_EVENTS to be large enough to fit your entire file.

Also, a regex that will be guaranteed to never match is (?!) that is open-paren, question-mark, exclamation-mark, close-paren. An equivalent is (*FAIL)

jstillwell
Explorer

plain TCP input port. no forwarders anywhere. I just have a [tcp://:port] stanza in inputs.conf, and homebrew scripts that connect and send the data into that port.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

And if using forwarders, are these light or heavy forwarders?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Can you please clarify, are you using Splunk forwarders to monitor files, or are you sending data in to a plain TCP input port? Or really, what do you mean by "using a tcp monitor"?

0 Karma

jstillwell
Explorer

This doesn't work. I suspect splunk is ignoring my sourcetype stanza in props.conf. I don't know how to verify this.

0 Karma
Get Updates on the Splunk Community!

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...

.conf24 | Learning Tracks for Security, Observability, Platform, and Developers!

.conf24 is taking place at The Venetian in Las Vegas from June 11 - 14. Continue reading to learn about the ...

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...