I have a forwarder (4.2, build 96430) set up on one server to forward logs to two indexers (4.3, build 115073). When I wasn't doing anything but forwarding the logs, everything was working normally. However, when I started assigning a sourcetype to specific source through the following lines in props.conf, I started seeing issues:
[source::.../*.request_log]
sourcetype = access_webservices
When assigning a sourcetype at the forwarder, I see a variable number of null characters inserted into some events, usually at the end. Sometimes it's only 100 or so, sometimes it's thousands of them. Here are a couple of examples:
10.0.0.100 www.example.com - [01/Mar/2012:09:34:53 +0100] "GET /path/a.gif HTTP/1.1" 200 562 "https://www.example.com/path/index.html" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB7.3; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)" 5926 "https://127.0.0.1:8300" "cookie1=a2" \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...(followed by several more, 799 in total)...\x00\x00\x00\x00\x00\x00\x00test.html" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTbMP3R7/5.12.2.16749)" 14600 "http://127.0.0.1:7300" "cookie1=a2"
10.0.0.100 www.example.com - [01/Mar/2012:09:34:53 +0100] "GET /path2/b.gif HTTP/1.1" 200 1977 "http://www.example2.com/path/test.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)" 21767 "ajp://server3.internal.net:9201" \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...(followed by several more, 297 in total)...\x00\x00\x00\x00\x00\x00Windows NT 6.1; Trident/4.0; GTB7.3; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)" 7045 "ajp://server3.internal.net:9201" \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...(followed by several more, 7500 in total)...\x00\x00\x00\x00\x00\x00
10.0.0.100 www.example.com - [01/Mar/2012:09:34:53 +0100] "GET /path3/c.gif HTTP/1.1" 200 1947 "https://www.example.com/test/index.html" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB7.3; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)" 4170 "https://127.0.0.1:8300" "cookie1=a2" \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...(followed by several more, 2500 in total)...\x00\x00\x00\x00\x00\x0
It's only happening on a small percentage of events in a small percentage of files. I'm not doing anything with that sourcetype at the indexer or search head (also 4.3, build 115073) and I verified that the null characters are not occurring in the log file but are in the raw data in Splunk by piping the search to "table _raw".
My outputs.conf on the forwarder looks like this:
[tcpout]
defaultGroup = indexer_A
indexAndForward = false
disabled = false
[tcpout:indexer_A]
autoLB = true
server = server01:9997,server02:9997
The inputs.conf on both indexers looks like this:
[splunktcp://9997]
I looked at this thread but I don't think it's relevant since neither Windows servers nor other character sets are at play. I also found this thread but I don't think it applies since I am using splunktcp in inputs.conf and I'm not seeing any other metacharacters or keywords.
Has anyone run into this issue before?
... View more