Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About sujithkumarkb12
sujithkumarkb12
Explorer
Member since:
02-14-2020
04-01-2021
Community Statistics
| Posts | 7 |
| Solutions | 1 |
| Karma Given | 0 |
| Karma Received | 0 |
| Member Since | 02-14-2020 |
Activity Feed
- Posted Re: splunk shows no logs at a particular time frame in day when searched on the specific index and sourcetype on Getting Data In. 03-10-2020 01:31 AM
- Posted Re: splunk shows no logs at a particular time frame in day when searched on the specific index and sourcetype on Getting Data In. 02-18-2020 05:03 AM
- Posted Re: splunk shows no logs at a particular time frame in day when searched on the specific index and sourcetype on Getting Data In. 02-18-2020 04:57 AM
- Posted Re: splunk shows no logs at a particular time frame in day when searched on the specific index and sourcetype on Getting Data In. 02-18-2020 12:55 AM
- Posted Re: splunk shows no logs at a particular time frame in day when searched on the specific index and sourcetype on Getting Data In. 02-17-2020 11:44 PM
- Posted Re: splunk shows no logs at a particular time frame in day when searched on the specific index and sourcetype on Getting Data In. 02-14-2020 10:51 PM
- Posted Why does splunk show no logs at a particular time frame in day when searched on the specific index and sourcetype? on Getting Data In. 02-14-2020 12:56 AM
- Tagged Why does splunk show no logs at a particular time frame in day when searched on the specific index and sourcetype? on Getting Data In. 02-14-2020 12:56 AM
- Tagged Why does splunk show no logs at a particular time frame in day when searched on the specific index and sourcetype? on Getting Data In. 02-14-2020 12:56 AM
- Tagged Why does splunk show no logs at a particular time frame in day when searched on the specific index and sourcetype? on Getting Data In. 02-14-2020 12:56 AM
- Tagged Why does splunk show no logs at a particular time frame in day when searched on the specific index and sourcetype? on Getting Data In. 02-14-2020 12:56 AM
- Tagged Why does splunk show no logs at a particular time frame in day when searched on the specific index and sourcetype? on Getting Data In. 02-14-2020 12:56 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 |
03-10-2020
01:31 AM
@jacobevans and @nickhillscpl
I was able to resolve this issue with the concept "Rebalance the indexer cluster"
https://docs.splunk.com/Documentation/Splunk/8.0.2/Indexer/Rebalancethecluster
When I had checked all the aspects of Splunk instance to cause this issue , I was able to see discrepancies in buckets replication and transition from hot to warm buckets , which explains data available when the data is ingested (HOT bucket) and unavailable later (transition failed to warm bucket and no replication happened) .
Thanks anyways to have helped me narrow down the issue 🙂
Cheers
Take care!!
... View more
02-18-2020
05:03 AM
you are right , i no more see the data that was indexed few hours back .
How do you suggest I check the search peers are fine and Job inspector has any reported jobs
... View more
02-18-2020
04:57 AM
@nickhillscpl , I applied the configs you mentioned but the issue prevails .
The data was logged successfully until an hour ago but now when i check its gone from 4am to 5am est.Whereas it was there before when i checked . Rest of the logs are intact.
... View more
02-18-2020
12:55 AM
@jacobpevans , let me break it down some more as in for 16th Feb 2020
The sourcetype I am facing issue is say ABC , latency value for that is 59.846 .
The linechart has gaps in it for a particular time frame that is 12 am to 7 am
_time indextime latency_mins _raw
2020-02-16 10:56:46.856 2020-02-16 10:56:47 0 2020-02-16T10:56:46.856-0500 INFO
2020-02-16 10:56:46.792 2020-02-16 10:56:47 0 2020-02-16T10:56:46.792-0500 INFO
2020-02-16 10:56:46.617 2020-02-16 10:56:47 0 2020-02-16T10:56:46.617-0500 INFO
2020-02-16 10:56:46.562 2020-02-16 10:56:47 0 2020-02-16T10:56:46.562-0500 INFO
2020-02-16 10:56:46.395 2020-02-16 10:56:47 1 2020-02-16T10:56:46.395-0500 INFO
2020-02-16 10:56:46.381 2020-02-16 10:56:47 1 2020-02-16T10:56:46.381-0500 INFO
2020-02-16 10:56:46.129 2020-02-16 10:56:47 1 2020-02-16T10:56:46.129-0500 INFO
2020-02-16 10:56:46.126 2020-02-16 10:56:47 1 2020-02-16T10:56:46.126-0500 INFO
2020-02-16 10:56:46.085 2020-02-16 10:56:47 1 2020-02-16T10:56:46.085-0500 INFO
Logs :
2020-02-16T14:51:07.112-0500 INFO 209 ABC.splunk.log: -4: | UNIQ_ID=2d-228523f | TRANS_ID=0000285235 | CLIENT_IP=123.34.45.7.67.56 | API_KEY=5f1c42727678a| ROUTING_CODE=Routed_200 | RESPONSE_ERROR_CODE=0 | RESPONSE_HTTP_CODE=200 | STATUS_MESSAGE=Message processed successfully_ | HTTP_BACKEND_CODE=200 | HTTP_EPAT_CODE= | HTTP_GUA_CODE= | HTTP_TCU_CODE= | INTERNAL_ERROR_MSG= | UTC_ENTRY=2020-02-16T19:51:06.985Z | UTC_EXIT=2020-02-16T19:51:07.103Z | REQUEST_PAYLOAD={"Data":{"AccountID":"46035002"}} | RESPONSE_PAYLOAD={"Data","Response":{"ResponseCode":"2000","ResponseStatus":"Success","ResponseDescription":"Successful Execution"}}
All the logs have the first time stamp (_raw) which ends with INFO and UTC_ENTRY and exit .
Which has almost 5 hours of difference which is the same time the data is missing .
And these format of logs are generated at only this time . early morning to be specific
... View more
02-17-2020
11:44 PM
@nickhills , I have this configuration in the props already .
[mysourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)\d{4}-\d\d
MAX_EVENTS=1000
TRUNCATE=10000
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z
What's surprising for me is that I can see logs getting ingested at 2 to 8 AM timeframe .
but later in the day I do not see them at all .
Is the Line break affecting here ?
... View more
02-14-2020
10:51 PM
@jacobevans , thanks for the response .
I could not share the entire response , few i have pasted below
sourcetype max_latency
abc 4492800.0000
def 3486215.000
ghi 3196800.3570
jkl 3196798.3560
mno 1814345.000
pqr 777648.3730
Also , i see the same gaps in the line chart from the below search you gave .
index=
| eval _time = _indextime
| timechart count
... View more
02-14-2020
12:56 AM
I have a clustered splunk environment and monitoring in place for quite a few application logs. Lately , I have been encountering an issue with data collection in Splunk .
For some frame of time everyday(2 to 5 hours) , I do not see any data even though the application server has logs generated. But for the rest of the day it works just fine .
Universal Forwarders and indexers are working just fine. This is affecting the dashboards and alerts , as the data is been missed out .
Example log: 2020-02-13T05:01:45.249-0500 INFO 801 | UNIQ_ID=20200213050500000170151fda6c-171dcee | TRANS_ID=000001da6c-171dce8 | VERSION=1.09 | TYPE=AUDIT | INTERNAL_ERROR_MSG= | UTC_ENTRY=2020-02-13T10:05.178Z | UTC_EXIT=2020-02-13T10:01:45.230Z,"Timestamp":"2020-02-13T10:01:45.062Z","Organization":"abc","Region":"RStS","ApplicationName":"Anoid"},"Data":{"rsCommand":"Clization","rsStatus":"executed","statusTimeStamp":"2020-02-13T09:57:13.000Z","rsStatusReason":"executed","lastRemoTimeStamp":"2020-02-13T09:57:29.000Z","rsMinutesRemaining":"6","remoDuration":"10","interTemperature":"12","interTimeStamp":"2020-02-13T10:00:20.000Z","Successful Execution"}}
Can anyone give some insight ,If you have faced or come across this kind of issue. I suspect Splunk is getting confused with the time format of the actual event and the time and year value format inside the event like status time stamp , last remo timestamp in the example log above.. But doesn't help me how to go about and solve this issue.
... View more
Labels
- Labels:
-
other
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
04-01-2021
11:00 AM
|
