Getting Data In

Forwarder not forwarding for 6 hours every day.

ssearwar
Explorer

I'm running 6.01 server and forwarders. I've got a couple of linux servers that are not forwarding between a 6 hour period of time everyday for a particular log. Can someone point me in the right direction to correct this issue?

Tags (2)
1 Solution

linu1988
Champion

Hello,
i was testing with your sample data and suspected the same when i saw your year field is not available. Splunk understands it's own way. In my case i was able to make it happen for 5 hours instead of 6 hours. The data exists in your splunk instance but it would hard to find the time period for that if you don't dig deep. If you see below.

alt text

You could see where your logs are going! So recommendation would be not to have log timestamp for the source rather have option as current i.e. file modification time or setup proper timestamp in the log itself. Thanks

View solution in original post

linu1988
Champion

Hello,
i was testing with your sample data and suspected the same when i saw your year field is not available. Splunk understands it's own way. In my case i was able to make it happen for 5 hours instead of 6 hours. The data exists in your splunk instance but it would hard to find the time period for that if you don't dig deep. If you see below.

alt text

You could see where your logs are going! So recommendation would be not to have log timestamp for the source rather have option as current i.e. file modification time or setup proper timestamp in the log itself. Thanks

ssearwar
Explorer

You're right. It's every 5 hours. And yes, a proper timestamp in the log would be the best option. Thanks.

0 Karma

linu1988
Champion

make the search

source=xx

in search time criteria : custom time earliest choose 2/18/11 11:00:00.000 to latest "2/18/11 23:00:00.000"

Do you get any event?

0 Karma

ssearwar
Explorer

Had to edit the entries due to sensitive data.

INFO Feb 18 11:25 com.web.struts.ViewSummary - randomcharacters.stuff : Retrieving information.
INFO Feb 18 11:25 com.web.struts.ViewDetail - randomcharacters.stuff : Getting details for document randomnumbers
INFO Feb 18 11:25 com.web.struts.ViewSummary - randomcharacters.stuff : User Annuities Robot_test owns document: MONITOR

0 Karma

linu1988
Champion

Could you paste some sample log updated between 9 AM to 2 PM your time when the file is not being read? That would be helpful.

0 Karma

ssearwar
Explorer

Nothing there between 08:59:24 AM and 14:01:01 PM.

0 Karma

linu1988
Champion

i doubt this could actually happen.

just type the in search and see the result.

source="whatever filename"|eval Time=strftime(_indextime,"%d/%m/%y %H:%M:%S %p")|table Time

Run it for last 7 days, and see you find the time series for the missing interval

0 Karma

ssearwar
Explorer

The forwarder appears to be functioning properly. It is forwarding all other logs on the server execpt for a small handfull.

0 Karma

kristian_kolb
Ultra Champion

Did you check what the forwarder is really doing at the time of the 'outage'?

https://your_forwarder:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Read more here;
http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

/K

ssearwar
Explorer

It seems to be just the log data. Although there is another log in the directory that is being forwarded just fine. It's just 4 logs that stop being forwarded.

0 Karma

MuS
SplunkTrust
SplunkTrust

does 'stops at 9am' mean that also no more _internal logs are forwarded by the universal forwarder or just the log data?

0 Karma

ssearwar
Explorer

I know that splunk can grab date from the source file's modification time, so I don't understand why it would work most of the time and then not work for a specific duration.

0 Karma

ssearwar
Explorer

Hmmm, I think their time format in their logs is bad. I don't see a year.

0 Karma

ssearwar
Explorer

Nothing fancy. The raw data is "Feb 11 09:47" format. Like I said, data gets indexed correctly then stops at 9am. I've checked "all time" and nothing shows.

0 Karma

MuS
SplunkTrust
SplunkTrust

Splunk tries to recognize the time stamp of events in multiple ways. One is to look in the raw data and search for date and time in it. Do you have some fancy date and time setting in the raw data that could confuse Splunk and makes it think your time is a date? Have you check 'all time' to see if the events come in with a wrong time stamp?

0 Karma

ssearwar
Explorer

No one is killing the process. REL 6 x86_64. Logrotation is on, but doesn't occur until midnight. It's just an application file that is perfectly fine until 9am and starts working again at 2pm.

0 Karma

kristian_kolb
Ultra Champion

No, well, I meant that it could be that someone/something kills your forwarder process.

What type of file is it, what OS? Log rotation scheme?

0 Karma

ssearwar
Explorer

Ok. The forwarder was restarted to see if forwarding would kickstart. Found a few errors.

ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=....). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

ERROR TailingProcessor - Ignoring path="/apps/splunk/splunkforwarder/var/log/splunk/.metrics.log.swp" due to: Bug: tried to check/configure STData processing but have no pending metadata.

kristian_kolb
Ultra Champion

that is a message that shows up when the splunk instance starts up - the forwarder in your case. Is the machine being rebooted? or the forwarder service shutdown/restarted?

0 Karma

ssearwar
Explorer

No errors for the log that I need:

02-10-2014 11:02:53.021 -0500 INFO WatchedFile - Will begin reading at offset=1236998 for file='....

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...