Beating my head off this one guys. I'm simply trying to forward several logs from my SEPM (SYmantec EndPoint Manager). All except the risk log is staying up to date. I've restarted the manager service to have it create it's local type backup of .txt files and changed my input to have it read the agt_risk.txt and it still doesn't make it. The forwarder read the file yesterday! What am I doing wrong?
Excerpt from the splunkd.log:
05-28-2015 10:31:05.516 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_risk.txt.
05-28-2015 10:31:05.516 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_scan.tmp.
05-28-2015 10:31:05.517 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_security.tmp.
05-28-2015 10:31:05.517 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_system.tmp.
05-28-2015 10:31:05.517 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_traffic.tmp.
05-28-2015 10:31:05.518 -0400 INFO TailingProcessor - Adding watch on path: D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_behavior.tmp.
05-28-2015 10:31:05.518 -0400 INFO TailingProcessor - Adding watch on path: D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_proactive.tmp.
05-28-2015 10:31:05.518 -0400 INFO TailingProcessor - Adding watch on path: D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_risk.txt.
My inputs.conf
[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_behavior.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_behavior
[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_risk.txt]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_risk
[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_scan.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_scan
[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_security.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_security
[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_system.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_system
[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_traffic.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_traffic
[monitor://D:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\data\dump\agt_proactive.tmp]
disabled = false
crcSalt = <SOURCE>
sourcetype = sep_proactive
Yes I know I'm trying to read the agt_risk.txt and the rest of the inputs are .tmp files. This is me trouble-shooting.
I'm stumped. The version of the forwarder I'm running is: 6.2.2
The size of the file I'm struggling with is: 97kb
Perhaps you have a TZ issue which is sending events "into the future" so when you know "the file was just written" (and it was) and then you look with "last 5 minutes" or something, you don't see any entries. Try searching for "All time"; do you see the recent events (in the future)? Try this search:
index=* | eval lagSecs=_indextime - _time | stats avg(lagSecs) by index,sourcetype,host
The avg should NEVER be neagive and it should be very small.
Thanks woodcock, so I assume this is bad:
main sep_risk SERVERNAME -282.000000
Any ideas or threads on how to fix this? /back to splunk answers......
Yes, that sourcetype is mis-timestamped for sure: it is impossible for events to "occur" after they have been indexed; furthermore, Splunk's default configuration is to ignore events that "will occur" too far in the future (I believe there is an error log in _index
for this) so this may be why some of your events are completely gone. You either have a NTP (clock-sync, wrong time) issue or an incorrect TZ. Only you can debug from here because it is "all in the data".
I rather like the troubleshooting steps taken in the Answers post on debugging a UF that's not reading a log file.
Are you sure the file is being written? Are you expecting the entire file to be re-forwarded for some reason?
No sir, and yes there's data in the file, it's not a chatty one, but last update was at 1:31 EST today, small file though only 10kb, could that be something???