Splunk Search

Why Splunk results are not showing properly?

iqbalintouch
Path Finder

Hi,

Can anyone please help me to understand why I am seeing the results in a linear format and I can not see the result properly. Here is the screenshot:

alt text

Tags (2)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

It would be very useful to have the search you are running, but perhaps this will help anyway:

You are looking at the timeline running over the past hour. The timeline isn't a "fancy view" but is instead a very plain "count" of the events that are being returned by your search, whatever it is.

So for instance, if you type into your search bar index=_internal you should see a chart like the one you took a screenshot of. It'll have parts where it goes down, parts where it goes up, or maybe it'll be more or less constant. You can click on parts of that to temporarily "filter" the list of events below to just show those.

Once you have that showing fields, though, you can click on one (well, hopefully one that would make sense to do this with) and select an aggregation that may look prettier.

For instance, run index=_internal. When you get results, on the left near the top of the interesting fields should be sourcetype (If it's not at the top, look for it alphabetically a bit farther down.). Click sourcetype and you'll see a little flyout giving you basic information about this field - the top items, how many there were in that time period and so on. Click the link at the top of that flyout that says "Top values by time" and that should take you to a prettier and more interesting view of that.

There's a great virtual .conf session on how to use the search bar and other basics of Splunk, I'd recommend going through that. There are also some free things at Splunk Education (shorter and not as thorough) that you may want to look at too.

View solution in original post

Richfez
SplunkTrust
SplunkTrust

It would be very useful to have the search you are running, but perhaps this will help anyway:

You are looking at the timeline running over the past hour. The timeline isn't a "fancy view" but is instead a very plain "count" of the events that are being returned by your search, whatever it is.

So for instance, if you type into your search bar index=_internal you should see a chart like the one you took a screenshot of. It'll have parts where it goes down, parts where it goes up, or maybe it'll be more or less constant. You can click on parts of that to temporarily "filter" the list of events below to just show those.

Once you have that showing fields, though, you can click on one (well, hopefully one that would make sense to do this with) and select an aggregation that may look prettier.

For instance, run index=_internal. When you get results, on the left near the top of the interesting fields should be sourcetype (If it's not at the top, look for it alphabetically a bit farther down.). Click sourcetype and you'll see a little flyout giving you basic information about this field - the top items, how many there were in that time period and so on. Click the link at the top of that flyout that says "Top values by time" and that should take you to a prettier and more interesting view of that.

There's a great virtual .conf session on how to use the search bar and other basics of Splunk, I'd recommend going through that. There are also some free things at Splunk Education (shorter and not as thorough) that you may want to look at too.

iqbalintouch
Path Finder

Here is the query string: index=app sourcetype=tc_jsp_ejb host="ejb" "main query string here"

Still I am seeing the results in same way if I extend the time duration to 24 hrs

0 Karma

Richfez
SplunkTrust
SplunkTrust

So that's a custom sourcetype, I think. Which is totally fine, just means I don't have any sample data.

You are still using the timeline. The timeline is a raw count of events. If it's flat, that just means you have about the same number of events each time period. What you want to do is instead pick a metric - a value inside your data - and timeline it.

Let me give you an example using the _internal index because it's one everyone has.

Take a look at this screenshot of my _internal index over the past 15 minutes. You see from my overlaid red line that I have that same nearly flat green timeline. ALL data is more or less like this, don't get caught up that I'm using _internal instead of your data, but do try do run that same search as I have and follow along.

So along the left are interesting fields. If you click one you get more information about that field. In this case I clicked the Field "component". When I do, there's more information like which are the top 10 values of this field ("Metrics" being most, with 74.573% of all the results having that field set to that item). Meanwhile, about 8 percent of the time the value of component is "SpecFiles".

After looking at that and realizing what it is you are seeing, trying clicking the "Top values by time". When I do, I get a view of my data that looks like this screenshot.

The green circle - notice the change in my search. It's added to the origal index=_internal this: | timechart count by component limit=10. That's mostly self explanatory. Limit 10 means it'll show 10 values separately then lump everything else into "other".

The red circle is the legend - usually on the left but I used the format button to move it - to tell you what's in the blue circle.

I hope that the graph on the right in the blue in that screenshot is more like what you are trying to achieve?

So, the ways to get that are back to your interesting fields. Look for fields that have a number of items between, oh, say 3 and 30. (Sometimes more or less is fine too, but this is probably best places to start). Click through fields to display what makes up the fields and if they look interesting try a timechart, using that method above on them, to see what it looks like over time.

Also, please do check out the things mentioned in the last paragraph, I think they'll really help clear up the confusion. I've copied the links below to make them easier to find.

Virtual .conf session on the basics of splunk
And look for the free short "using search" at Splunk Education

iqbalintouch
Path Finder

Hi @rich77,

I understand what you are explaining here and I appreciate the same, but I still having the same issue. The issue I am trying to raise here is, whatever I search for any time range...the result is showing a flat output under events tab. Lets take an example here with search string:

host=myhost source="/var/log/xyz/health_etcd.log" sourcetype=var_log_xyz status=healthy | top limit=5 host
so in this case if you look fr host it should be showing up and down bar as a result under event tab but in my case this is not happening and this is happening everytime. here is the url:![screenshot][1]

http://imgur.com/4Mnnw8X

0 Karma

Richfez
SplunkTrust
SplunkTrust

Great, that does look suspiciously flat, as opposed to the screenshot in the question which is entirely normally flat.

So let's scope this problem: When you click on the tab "Statistics" what numbers do you see down the "count" field? What do they average (eyeballing is fine) and what's the range of them? What's the biggest and the smallest numbers in there?

Also check the Visualizations tab - what does it look like? Lots of variation or just a tiny bit (or none?)

For your time drop down - what specifically do you have selected? Is it "past 24 hours" or is it Relative, last 24 hours with a snap to "Beginning of Hour" with Latest snapping to "Beginning of hour" as well? Can you change to "Presets" and "last 24 hours" (or whatever is closest to the time range you are looking at)? If it already is, flip it to previous 24 hours as I described. There IS a difference in how these behave at the beginning and the ending of the timeline - check the results when you change that and see if the last or first green column changes to be less flat.

Since we have both examples of normally flat (the screenshot in the question itself, which looks entirely and perfectly fine to me) and a very-flat one (the last screenshot), can you open manually confirm the number of entries in "/var/log/xyz/health_etcd.log" in any particular time-slot matches what splunk has (e.g. open the file in an editor, count the lines between 8 am and 9 am or whatever, see if that's what splunk says for that same time period, then repeat for at least one other time slot).

Lastly, does this affect all browsers? Can you try with one other web browser and see what it does?

0 Karma

Richfez
SplunkTrust
SplunkTrust

Hi, iqbalintouch, Just checking in to see if you've been able to narrow down further precisely when the potentially anomalous timeline happens? Is this still a problem?

iqbalintouch
Path Finder

Hi @rich7177 ,

Thanks a lot for your help, I have tried it in a different browser and I was able to see the results are coming fine. So, it seems to be a browser issue as you have already said...this issue is happening in Google Chrome browser only.

Thank you once again ! 🙂

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...