I am trying to list specific events, but I am not able to view them. Splunk shows that events exist, but it comes up with no events found. Screenshot attached
Okay found a solution to a similar problem.
ou limits.conf had the following conf :
[search]
max_count=0
This prevent splunk to store events as search results.
Removing the faulty settings resolve our search issue
We encounter the same error last Week (see my comment)
The issue was on our limits.conf where we'd set a
[search]
max_count=0
which prevents Splunk to store any events.
Removing that settings resolved our issue
Hello,
Did you succeed in solving your issue ?
We've encounter the same problem:
We are using a search index=_internal Error
We got the same result as the main screenshot .
If we use the search index=_internal Error | table *
We got a table in statistics with every fields. But still no event.
We are using Splunk 6.6.x on a SHCluster/IdxCluster.
After some tests, the error appears only on SH, not if we launch a search from the indexer.
Are those sources quite large in size? I've recently ran into this with a sourcetype that has very large log files. My resolution was that I needed to specify the index I wanted to search. I don't know exactly why this needed to occur but I have a hunch it's due to some sort of max being reached when scanning very large events without an index supplied.
Could it be you have the No Event Sampling activated?
Take a look at this: http://docs.splunk.com/Documentation/Splunk/6.4.0/Search/Retrieveasamplesetofevents#Specify_a_sampli...
UPDATE
Take a look at the following two values from your inspect job trace:
resultCount 0
scanCount 287
Now take a look at their correspondent description:
resultCount - The total number of results returned by the search. In other words, this is the subset of scanned events (represented by the scanCount) that actually matches the search terms.
scanCount - The number of events that are scanned or read off disk
Specifically this bit:
This is the subset of scanned events
(represented by the scanCount) that
actually matches the search terms.
In summary, your search filter does not match any events you are reading off disk.
Hope that makes sense.
Below is the job inspection report I hope its format your looking for.
Execution costs
Duration (seconds) Component Invocations Input count Output count
0.00 command.fields 5 287 287
0.23 command.search 5 - 287
0.13 command.search.index 6 - -
0.00 command.search.filter 1 - -
0.00 command.search.calcfields 1 287 287
0.00 command.search.fieldalias 1 287 287
0.00 command.search.index.usec_1_8 5,325 - -
0.00 command.search.index.usec_512_4096 5 - -
0.00 command.search.index.usec_64_512 114 - -
0.00 command.search.index.usec_8_64 21 - -
0.06 command.search.kv 1 - -
0.03 command.search.rawdata 1 - -
0.01 command.search.typer 1 287 287
0.00 command.search.lookups 1 287 287
0.00 command.search.summary 5 - -
0.00 command.search.tags 1 287 287
0.00 dispatch.check_disk_usage 1 - -
0.00 dispatch.createdSearchResultInfrastructure 1 - -
0.08 dispatch.evaluate 1 - -
0.08 dispatch.evaluate.search 1 - -
0.28 dispatch.fetch 6 - -
0.23 dispatch.localSearch 1 - -
0.00 dispatch.readEventsInResults 1 - -
0.23 dispatch.stream.local 5 - -
0.01 dispatch.timeline 6 - -
0.01 dispatch.writeStatus 6 - -
0.03 startup.configuration 1 - -
0.07 startup.handoff 1 - -
Search job properties
canSummarize 0
createTime 2016-04-20T12:48:20.000+05:30
cursorTime 2015-09-28T16:00:00.000+05:30
custom
{
"search": "(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*"
}
defaultSaveTTL 604800
defaultTTL 600
delegate None
diskUsage 122880
dispatchState DONE
doneProgress 1.0
dropCount 0
eai:acl
{
"app": "search",
"can_write": "1",
"modifiable": "1",
"owner": "admin",
"perms": {
"read": [
"admin"
],
"write": [
"admin"
]
},
"sharing": "global",
"ttl": "600"
}
earliestTime 2012-07-08T01:30:52.000+05:30
eventAvailableCount 0
eventCount 287
eventFieldCount 0
eventIsStreaming True
eventIsTruncated False
eventSearch search (source="nafx.g1303v00'" OR source="nafx.g1304v00'") opc* AND acf*
eventSorting desc
indexEarliestTime 1460630991
indexLatestTime 1460631052
isBatchModeSearch False
isDone True
isFailed False
isFinalized False
isPaused False
isPreviewEnabled True
isRealTimeSearch False
isRemoteTimeline False
isSaved False
isSavedSearch False
isTimeCursored 1
isZombie False
keywords acf* opc* source::nafx.g1303v00' source::nafx.g1304v00'
label None
modifiedTime 2016-04-20T12:48:27.086+05:30
normalizedSearch litsearch ( source="nafx.g1303v00'" OR source="nafx.g1304v00'" ) opc* AND acf* | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
numPreviews 0
pid 21431
priority 5
remoteSearch litsearch ( source="nafx.g1303v00'" OR source="nafx.g1304v00'" ) opc* AND acf* | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
reportSearch None
request
{
"adhoc_search_level": "verbose",
"auto_cancel": "30",
"check_risky_command": "false",
"custom.search": "(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*",
"earliest_time": null,
"indexedRealtime": null,
"latest_time": null,
"preview": "1",
"rf": "*",
"sample_ratio": "1",
"search": "search (source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\") opc* AND acf*",
"status_buckets": "300",
"ui_dispatch_app": "search"
}
resultCount 0
resultIsStreaming True
resultPreviewCount 0
runDuration 0.388
runtime
{
"auto_cancel": "30",
"auto_pause": "0"
}
sampleRatio 1
sampleSeed 0
scanCount 287
search search (source="nafx.g1303v00'" OR source="nafx.g1304v00'") opc* AND acf*
searchCanBeEventType 1
searchProviders
[
"oc8001872801.ibm.com"
]
searchTotalBucketsCount 13
searchTotalEliminatedBucketsCount 0
sid 1461136700.12
statusBuckets 300
ttl 600
Additional info timeline search.log
I deleted the source log file and added it again with a new index instead of the default index and now when I run this command I am able to see the records And I have created separate indexes for each of the log files.
May be I used default index for all the log files which caused this issue.
Ok, I think I might have the answer.
Take a look at this two values from your inspect job trace:
resultCount 0
scanCount 287
Now take a look at their correspondent description:
resultCount - The total number of results returned by the search. In other words, this is the subset of scanned events (represented by the scanCount) that actually matches the search terms.
scanCount - The number of events that are scanned or read off disk
Specifically this bit:
This is the subset of scanned events
(represented by the scanCount) that
actually matches the search terms.
In summary, your search filter does not match any events you are reading off disk.
Hope that makes sense.
I am not able to understand the logic. There are something like 300,000+ events for this log file. So it scanned only 287 events and did not find any matches on these 287 events.
Yeah, that seems to be case.
One way to verify this is to run all the following searches and compare the number of events you get:
(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\")
--
(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\")
| search opc*
--
(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\")
| search acf*
--
(source=\"nafx.g1303v00'\" OR source=\"nafx.g1304v00'\")
| search opc* AND acf*
Event sampling returns me very few records. and not the actual set of records that match. This is a normal problem that I have even with another search that I was performing.
There was another similair query I tried and when I filter based on date month it returns value for one particular month and not the other.
I have splunk installed on my linux laptop. Could that have any implication in indexing.
Running on a Linux laptop shouldn't make any difference.
Could you try running your query in Smart Mode instead of Verbose?
Do you get any results when running any other query?
For instance, if you run the following:
index=_internal | head 10000
Do you get 10,000 results?
I ran the index command and it returned me 10000 results. The case is sometime when I am adding more filter let me give another example. Where I tried to look up all log entries that had recall in them and it returned me 292 results. When I added recall AND fail*. It showed me 36 entries but no results displayed. When I did an inspect job I found that event count is 36 but available event count is 0.
I guess there might be something wrong with the way I am writing the search query or may be indexing.
Can you run one of those searches that do not return any results but then click on Job > Inspect Job, copy the report and paste it here as Code Sample (use the button with 1s and 0s above)?
Maybe there's something in the Job Inspector telling us what's going on
Looks like I dont have enough Karma points to add more files.
Scene 1 - This does not work
1st query
source=syslog.txt recall* --- Returns 292 records
2nd query
source=syslog.txt recall fail* -- Returns 36 records in event count but does not display results.
But when I try
source=syslog.txt migrat* fail* --- This query returns me results.
And when I concatenate both the query it does not return any results.
source=syslog.txt (migrat* fail*) AND (recall fail*)