How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

[lukasz92]

Hi,

I have a serious problem with logs.. some events (below 0.01%) have strange characters.
- such strange characters seems to not repeat (all are unique)
- there are no regional characters in events - OS has english language set
- it affects mainly Splunk processes, but not only
- for one selected host - there are "good" and "bad" process names. For example host ending with "PAL01" has hundreds of events with "splunk-powershell.exe" process name, and one with chinese artifacts.

My example search to find such hosts:

index="wineventlog" host=* (EventID=4688 OR EventID=861) NewProcessName=* SubjectUserName=* *splunk-pow* NOT *splunk-power*

What is going on? It is a serious problem, because we don't know how many artifacts are there and how to find them all.

[sjohnson_splunk]

I suggest that you examine the actual event logs on one of the servers with the event viewer and see if it originates there. If the logs are OK I would re-install the UF on those servers.

[buysse]

lukasz92 - did reinstalling work to eliminate the artifacts?

We're seeing the same thing. What versions were you running (our forwarders are on 6.3.2, indexers on 6.4.3)?

[jkat54]

What is creating these event logs?

[lukasz92]

I don't understand your question.

They are forwarded to indexers with input:

[WinEventLog://Security]
disabled = 0
current_only = 1
index = wineventlog
renderXml=true

[jkat54]

Valid answers might include McAfee antivirus, custom GPOs, UAC, standard windows auditing when access is granted, custom software, etc.

[lukasz92]

This is only WinEventLog. I catch the events with EventCode 4688.

[jkat54]

Yeah doesnt make much sense unless there is some data corruption along the way... packet loss, pinched network cable, disk corruption, etc.

[MuS]

Hi lukasz92,

check the CHARSET = <string> in props.conf for this sourcetype on the universal forwarder. Maybe you need to adjust this in your case http://docs.splunk.com/Documentation/Splunk/6.4.0/Admin/Propsconf#GLOBAL_SETTINGS

cheers, MuS

[lukasz92]

Changed also on forwarders. It didn't help.

[lukasz92]

I changed it to CP1252, it didn't help - strange characters appear.

• I observed that number of strange characters usually equals number of normal characters that should be in this place
  • Strange characters usually(but not always) end with something similar to 'r' - after reading URL it seems to be http://www.fileformat.info/info/unicode/char/ffff/index.htm - this character.
  • Only values are broken, not tag names.