Problem with Splunk outputting log entries as hex

[DevinG]

I am running into a problem I cannot seem to figure out. One log file I have splunk reading from suddenly starts reading log entries in what seems to be hexadecimal formats. A sample is as followed:

[\x00R\x00e\x00q\x00u\x00e\x00s\x00t\x00]\x00 \x00h\x00t\x00t\x00p\x00:\x00/\x00/\x00c\x00a\x00t\x00a\x00l\x00o\x00g\x00.\x00d\x00e\x00v\x00.\x00d\x003\x00n\x00w\x00.\x00n\x00e\x00t\x00/\x00c\x00a\x00t\x00a\x00l\x00o\x00g\x00s\x00v\x00c\x00/\x00v\x001\x00/\x00c\x00a\x00t\x00a\x00l\x00o\x00g\x00s\x00/\x00m\x00o\x00v\x00i\x00e\x00p\x00l\x00e\x00x\x00 \x00 \x00[\x00R\x00e\x00s\x00p\x00o\x00n\x00s\x00e\x00]\x00 \x00A\x00c\x00c\x00e\x00p\x00t\x00e\x00d\x00 \x00 \x00 \x00 \x00

First off I have no clue while the file is being read this way as the log file itself seems to have appropriately formatted information in there. What's even stranger though is this isn't a consistent issue. When the file first gets read when setup it reads the data correctly in splunk and has entries displayed as expected. Is there a setting that I may be overlooking that is causing this behavior.

One last thing to note is I have reviewed the log file for any unexpected characters that may be throwing off the encoding rules when splunk is reading the data. However all that is in the file is your typical characters.

Any help is incredibly appreciated.

[477450]

can i please get some answer for this question, i am blocked here.

[sideview]

If you notice, the letters actually spell out a little blob of text. I'm not sure what kind of crazy UTF-8/UTF-16 foo is causing the weird escape sequences, but if you remove all the \x00's, you get:

[Request] http://catalog.dev.d3nw.net/catalogsvc/v1/catalogs/movieplex
[Response] Accepted

My guess is that this data is being sent in over TCP, and something periodically gets pointed at this port, resulting in the HTTP traffic getting indexed by Splunk?

UPDATE:

Then again, you mentioned that you opened the files themselves and they looked fine, So does that mean that the [Request]... [Response]... text is in the raw file itself, and is quite readable there?

Is it possible that there is more than one data input active for the same sourcetype, one that's a TCP port and one that's reading from your file?

[DevinG]

Hey thanks for your input. I've researched this quite a bit further and it seems like the potential problem may be NULL padding on the file while it's being written to and the reader pulling it too quick.

That is accurate though the REQUEST REPSONSE is what's written to the file.