I have noticed that at random times my indexer is indexing old data logs from days, and sometimes even months in the past. I have no clue as to why this is happening. The logs are formatted like this:
1452006410 January 5, 2016 9:06:50 AM CST NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED Proview A1827-2100 ATM - A1827-2100 - SERVICEMODE ENTERED 11 DXA CLEAR Server: INCHARGE-OI
1452006410 January 5, 2016 9:06:50 AM CST NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED Proview A1827-2100 ATM - A1827-2100 - SERVICEMODE ENTERED 12 SYSTEM ESCALATION MATCHED: Proview2/ArchiveInActiveTraps
1452006410 January 5, 2016 9:06:50 AM CST NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED Proview A1827-2100 ATM - A1827-2100 - SERVICEMODE ENTERED 13 SYSTEM ESCALATION MATCHED: Notification Clear/Archive - InActive/Archive Inactive Resolved Notifications
1452006410 January 5, 2016 9:06:50 AM CST NOTIFICATION-Proview_A1827-2100_ATM_20-_20A1827-2100_20-_20SERVICEMODE_20ENTERED Proview A1827-2100 ATM - A1827-2100 - SERVICEMODE ENTERED 14 SYSTEM ESCALATION REACHED: Proview2/ArchiveInActiveTraps, Level-0
At times, I see in the searched logs the date from the indexer will say, this:
1/5/16
9:06:50.000 AM
1448550410 November 26, 2015 9:06:50 AM CST NOTIFICATION-CPU_PerformanceCiscoSystem_I-CPUPerformanceCiscoSystem-PSR-ALBMDSP301/0_HighUtilization CPU_Performance_CiscoSystem I-CPU_Performance_CiscoSystem-PSR-ALBMDSP301/0 HighUtilization 8 SYSTEM ESCALATION SCHEDULED: Resources/ResoursesClearEvent for Level-1 due at November 26, 2015 9:11:51 AM CST
1448550416 November 26, 2015 9:06:56 AM CST NOTIFICATION-MemoryPerformanceHostResources_I-MemoryPerformanceHostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 22 SYSTEM ESCALATION REACHED: Resources/ResoursesClearEvent, Level-1
1448550416 November 26, 2015 9:06:56 AM CST NOTIFICATION-MemoryPerformanceHostResources_I-MemoryPerformanceHostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 23 SYSTEM Action invoked... ClearEvent
1448550417 November 26, 2015 9:06:57 AM CST NOTIFICATION-MemoryPerformanceHostResources_I-MemoryPerformanceHostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 24 SYSTEM Action completed successfully... ClearEvent
1448550417 November 26, 2015 9:06:57 AM CST NOTIFICATION-MemoryPerformanceHostResources_I-MemoryPerformance_HostResources-MEM-ALVPHASE3UI/6_InsufficientFreeMemory Memory_Performance_HostResources I-Memory_Performance_HostResources-MEM-ALVPHASE3UI/6 InsufficientFreeMemory 25 SYSTEM Action invoked... zArchiveEvent
Show all 257 lines
ClassName = CPU_Performance_CiscoSystem Escalations = SCHEDULED: Resources/ResoursesClearEvent for Level-1 due at November 26 EventName = HighUtilization InstanceName = I-CPU_Performance_CiscoSystem-PSR-ALBMDSP301/0 SourceEsc = Server: INCHARGE-AM-PM-GA-FL eventtype = ActionSuccess eventtype = Escalations Scheduled eventtype = Notification Clear eventtype = Notification Notify host = ALVIONIX01 source = \ALVIONIX01\d\InCharge\SAM\smarts\local\logs\INCHARGE-SA.audit sourcetype = SAM_Audit
So as you can see, the indexer is picking up older log entries and indexing them as a group as one date.
What can be done?
Any help would be appreciated.
I don't know that there is "an answer" for this, but I think the following is a pretty good process for figuring it out.
And just a thought: do you zip your old log files? Because if you do, that creates a new file. And if that file is in the directory that you are monitoring, Splunk will say "Look! A new file!" and then decompress and index it.
BEST PRACTICE
When you roll old log files, keep the current log file and one prior in the monitored directory.
Zip the older files if you want, but always move them (zipped or not) to a different directory.
Finally the "different directory" should not be a subdirectory of any monitored directory.
Bottom line: I think that old log files may have reappeared in a new location or different format. But they showed up in a directory or subdirectory that Splunk is monitoring...
lguinn,
Thanks for the response, but I don't think that is the problem. The system rolls the logs on an almost daily basis. The rolled logs are excluded from indexing so I think I am good there.
The logs its picking up are OLD logs, like months old. Is always only 297 lines every time this happens. The old data is indexed and given todays date with the old 297 lines as a record. The current data is always only one line record from the log file.
So, I still have the issue and don't know how to fix it.