Getting Data In

LineBreakingProcessor - Truncating line because limit of 1000000 bytes has been exceeded with a line length >= 1003520 - data_source="lsof", data_host="gbrdcr10328n02", data_sourcetype="lsof"

john_howley
Path Finder

I am getting this error in the splunkd.log.
i've seen a previous post which talks about the Line Breaking settings within Props.conf, but I don't have that section in any of my props,conf either system or nmon (which is the element being complained about)
in the Props.conf I have for NMON in [/apps/splunk-6.2.2-255606/splunk/etc/apps/nmon/default] directory I have the nmon config as

nmon config stanza

[nmon_config]

BREAK_ONLY_BEFORE=CONFIG,
MAX_EVENTS=100000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%d-%b-%Y:%H:%M
TIME_PREFIX=CONFIG,
TRUNCATE=0

The Truncate=0 would lead me to beleive, from what I've seen on a previous post, don't truncate, but clearly it is.

Can anyone suggest which setting might be influencing this please?

1 Solution

rphillips_splk
Splunk Employee
Splunk Employee

@john_howley : The splunkd error pertains to the sourcetype=lsof as reported in data_sourcetype=lsof. You will need a [lsof] stanza defined in props.conf to apply to these events:

example:
set in $SPLUNK_HOME/etc/system/local/props.conf on all of your indexers:
[lsof]
TRUNCATE=0

restart splunk
$SPLUNK_HOME/bin
./splunk restart

Use the following attributes to define the length of a line.

TRUNCATE =
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
garbage data).
* Defaults to 10000 bytes.

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

@john_howley : The splunkd error pertains to the sourcetype=lsof as reported in data_sourcetype=lsof. You will need a [lsof] stanza defined in props.conf to apply to these events:

example:
set in $SPLUNK_HOME/etc/system/local/props.conf on all of your indexers:
[lsof]
TRUNCATE=0

restart splunk
$SPLUNK_HOME/bin
./splunk restart

Use the following attributes to define the length of a line.

TRUNCATE =
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
garbage data).
* Defaults to 10000 bytes.

john_howley
Path Finder

Thanks rphillips - that worked..

0 Karma

Tejkumar451
Explorer

so , should we do this change on the indexer side or splunk forwarder side?

0 Karma

john_howley
Path Finder

as an additional note there are three .conf files that do contain a =1000000 they are

indexes.conf:maxMetaEntries = 1000000
limits.conf:max_chunk_queue_size = 1000000
props.conf:TRUNCATE = 1000000

The TRUNCATE one looks hopeful, but comes from the [kvstore] stanza which I initially thought was referring to certificate, but now I see it is key values - I will try creating a local version to allow > 1000000 and see what occurs.
[kvstore]
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = datetime
TIME_FORMAT = %m-%d-%Y %H:%M:%S.%l %z
INDEXED_EXTRACTIONS = json
KV_MODE = none
TRUNCATE = 1000000

0 Karma

john_howley
Path Finder

Adjusting that setting in ..local/props.conf and restarting had no affect - stil lget the same error.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...