Hello Splunkers.
I'm helping a client to find out why some of his events are not being broken correctly.
They are currently running a Search Head Cluster with 3 SHs, 2 Indexers, 1 Master Cluster and 1 License/Deployer.
Here is a example of log:
--
tstamp="20160105 23:59:39.893"
IdCmd=01
Port=01
tstampResp="20160105 23:59:40.390"
Cmd="XXXXXX"
tipoAcao=ABC
Pri=I
Rsgmt=H
mainkey=12345678
acao="A";vAcao="000100000000A";resp="O"
acao="A";vAcao="000200000000A";resp="O"
acao="A";vAcao="000300000000A";resp="O"
acao="A";vAcao="000400000000A";resp="O"
acao="A";vAcao="000500000000A";resp="O"
acao="A";vAcao="000600000000A";resp="O"
acao="A";vAcao="000700000000A";resp="O"
acao="A";vAcao="000800000000A";resp="O"
acao="A";vAcao="000900000000A";resp="O"
acao="A";vAcao="001000000000A";resp="O"
acao="A";vAcao="001100000000A";resp="O"
acao="A";vAcao="001200000000A";resp="O"
acao="A";vAcao="001300000000A";resp="O"
acao="A";vAcao="001400000000A";resp="O"
acao="A";vAcao="001500000000A";resp="O"
acao="A";vAcao="001600000000A";resp="O"
acao="A";vAcao="006700000000A";resp="O"
acao="A";vAcao="006A00000000A";resp="O"
acao="A";vAcao="006B00000000A";resp="O"
acao="A";vAcao="006C00000000A";resp="O"
acao="A";vAcao="006E00000000A";resp="O"
acao="A";vAcao="006F00000000A";resp="O"
acao="A";vAcao="007000000000A";resp="O"
acao="A";vAcao="007200000000A";resp="O"
acao="A";vAcao="007400000000A";resp="O"
acao="A";vAcao="007500000000A";resp="O"
acao="A";vAcao="007600000000A";resp="O"
acao="A";vAcao="007700000000A";resp="O"
acao="A";vAcao="007800000000A";resp="O"
acao="A";vAcao="007900000000A";resp="O"
acao="A";vAcao="007B00000000A";resp="O"
acao="A";vAcao="007E00000000A";resp="O"
acao="A";vAcao="008000000000A";resp="O"
acao="A";vAcao="008200000000A";resp="O"
acao="A";vAcao="008900000000A";resp="O"
acao="A";vAcao="008A00000000A";resp="O"
acao="A";vAcao="008E00000000A";resp="O"
acao="A";vAcao="008F00000000A";resp="O"
acao="A";vAcao="009800000000A";resp="O"
acao="A";vAcao="009B00000000A";resp="O"
acao="A";vAcao="009D00000000A";resp="O"
acao="A";vAcao="009F00000000A";resp="O"
acao="A";vAcao="00A000000000A";resp="O"
acao="A";vAcao="00AA00000000A";resp="O"
acao="A";vAcao="00AB00000000A";resp="O"
acao="A";vAcao="00AC00000000A";resp="O"
acao="A";vAcao="00B500000000A";resp="O"
acao="A";vAcao="00C000000000A";resp="O"
acao="A";vAcao="00C100000000A";resp="O"
acao="A";vAcao="00C200000000A";resp="O"
acao="A";vAcao="01AA00000000A";resp="O"
acao="A";vAcao="021100000000A";resp="O"
acao="A";vAcao="021200000000A";resp="O"
acao="A";vAcao="039100000000A";resp="O"
acao="A";vAcao="039C00000000A";resp="O"
acao="A";vAcao="01C100000000A";resp="O"
acao="A";vAcao="000500000000A";resp="O"
acao="A";vAcao="001400000000A";resp="O"
acao="A";vAcao="005300000000A";resp="O"
acao="A";vAcao="005C00000000A";resp="O"
acao="A";vAcao="008400000000A";resp="O"
acao="A";vAcao="001600000000A";resp="O"
acao="A";vAcao="00F300000000A";resp="O"
acao="A";vAcao="00F000000000A";resp="O"
acao="A";vAcao="01C200000000A";resp="O"
acao="A";vAcao="00EF00000000A";resp="O"
acao="A";vAcao="01C300000000A";resp="O"
acao="A";vAcao="01C400000000A";resp="O"
acao="A";vAcao="00EE00000000A";resp="O"
acao="A";vAcao="01C600000000A";resp="O"
acao="A";vAcao="01C500000000A";resp="O"
acao="A";vAcao="01D700000000A";resp="O"
acao="A";vAcao="00EC00000000A";resp="O"
acao="A";vAcao="01C700000000A";resp="O"
acao="A";vAcao="00ED00000000A";resp="O"
acao="A";vAcao="01DE00000000A";resp="O"
acao="A";vAcao="01DD00000000A";resp="O"
acao="A";vAcao="01E200000000A";resp="O"
acao="A";vAcao="01C800000000A";resp="O"
acao="A";vAcao="01E100000000A";resp="O"
acao="A";vAcao="01E000000000A";resp="O"
acao="A";vAcao="01C900000000A";resp="O"
acao="A";vAcao="01CA00000000A";resp="O"
acao="A";vAcao="00F400000000A";resp="O"
The log above always starts with --
and can have hundreads of acao="A";vAcao="XXXX00000000A";resp="O"
lines.
Here is the props.conf that I'm using:
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=--
disabled=false
TIME_PREFIX=\d+\=\"
TIME_FORMAT=%Y%m%d %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=30
And here is the transforms.conf that I'm using:
REGEX = acao=\"(.*?)\";vAcao=\"(.*?)\";resp=\"(.*?)\"
FORMAT = acao::$1 vAcao::$2 resp::$3
MV_ADD = true
Sometimes, the event is not broken correctly. It breaks on acao="A";vAcao="XXXX00000000A";resp="O"
lines.
I thought that I should include the TRUNCATE option in my props.conf.
However, when I try to Distribute Bundle Configuration using the UI at the Master, I receive an error message saying that I can't use TRUNCATE at my stanza.
So now I'm a bit lost.
Do you guys have any ideias that can help me?
Od course I can go directly to the Indexers and put the TRUNCATE option "by hand", but since they will soon add more indexers, this will not be feasible without the Master's Distribute Bundle Configuration.
Thanks in advance!
Regards,
GMA
Your props.conf does not specify MAX_EVENTS
- which defaults to 256. So, when you have an event with many, many lines, Splunk breaks even without a --
Add this to the props.conf
MAX_EVENTS = 1024
Or some other number that is larger than the maximum possible number of events. Setting TRUNCATE
would not have solved this problem anyway.
Your props.conf does not specify MAX_EVENTS
- which defaults to 256. So, when you have an event with many, many lines, Splunk breaks even without a --
Add this to the props.conf
MAX_EVENTS = 1024
Or some other number that is larger than the maximum possible number of events. Setting TRUNCATE
would not have solved this problem anyway.
From where you're ingesting this log? Is it a data monitoring setup on Search Head OR on a Forwarder? Which location you're putting the props.conf for the sourcetype (with TRUNCATE)?
Hi somesoni2. I'm getting this data on a Forwarder. /opt/splunk/etc/master-apps/app/local/props.conf is the path on the Master. When I use Truncate, I can't use the Distribute Bundle Configuration on Master's UI.
Are you able to push the bundle from the command line/shell on the Indexer Master?? using http://docs.splunk.com/Documentation/Splunk/6.2.0/Indexer/Updatepeerconfigurations#3._Apply_the_bund...