We have a system that sends logs with extra white space, causing everything to be off by 1. When the day of month changes from 1 digit to 2 digits, the proxy device sends the log with:
The first example has 2 spaces: Nov 3
and the second has 1 space: Oct 31
. When there are 2 spaces between the month and day, all of the fields are off by one.
Nov 3 16:27:47 10.1.126.214.100 2015-11-03 16:27:47 15 10.2.172.212 404 ...
Oct 31 23:59:59 10.1.126.99.100 2015-10-31 23:59:59 227 10.2.189.178 407 ...
My transforms.conf is as follows:
DELIMS = " "
FIELDS = month,date,time,dvc,date1,time-taken,duration,src,status,...
I was thinking about changing to a REGEX, but this seems to be a little more trouble. Is there an easy way to do this? I would like to do: DELIMS = "\s+"
or possibly do something to say "skip empty values". I do not know if this is possible, or is there a simple REGEX that will work?
Hi DeronJensen,
I tested DELIMS with \s+
and confirmed it will not work. You'd need to use REGEX instead to use \s+
notation but then FIELDS will not work. A simpler way would be to normalise the incoming data so that multi-whitespaces become one. You can easily do this using the SEDCMD in props.conf. In this case the following should work,
[my_sourectype]
...
SEDCMD-single-whitepace = s/\s\s*/ /g
This will only work on new data. Hope this helps.