Splunk Search

How do I edit my timechart search to prevent all of my data being put into one bucket?

johnraftery
Communicator

Hi,

I have this search:

eventtype=mlc sourcetype=murex_log4j source=launchermxmlc.mxres.log | stats earliest(_time) as start | appendcols [ search eventtype=mlc sourcetype=sun_jvm service_name=mxmlc.gc.log | table relative_time after_gc max_gc ] | eventstats max(start) as start_time | eval _time = start_time + relative_time | fields - start - start_time - relative_time | timechart fixedrange=false bins=2000 max(after_gc), max(max_gc)

The idea is that the GC log doesn't have a time stamp, only the relative time since the start of the log, so I need to get the start time from a different log before I can make the timechart. This works perfectly when I set span=x, but not with bins=x. When I set bins=2000, all the data goes into a single bucket, which doesn't make any sense to me. I need it to work with bins because there may be days worth of data, so span=1s would give too many points to display.

Can anyone help please?

sideview
SplunkTrust
SplunkTrust

Interesting. Can you replace the timechart command with just table _time after_gc max_gc , rerun, export those rows and either paste them here or put them on pastebin? Sounds like some interesting behavior in timechart, quite possibly a bug, and I'd like to reproduce.

0 Karma

johnraftery
Communicator

Sure, this is the result

_time                     after_gc    max_gc
2016-05-09 10:56:11.265   242241      478464
2016-05-09 10:54:59.594   241847      467712
2016-05-09 10:53:48.860   245289      456448
2016-05-09 10:53:08.290   212081      452608
2016-05-09 10:52:42.023   187632      454656
...

For 33 rows

0 Karma

sideview
SplunkTrust
SplunkTrust

I think this search may have once made sense, but that it has lost it's way recently. I'm going to walk through it pipe by pipe.

In particular, I think the problem is that all the search language before the timechart command can only ever produce a single row and a single _time value for timechart to chart.

Here are the first two commands.

eventtype=mlc sourcetype=murex_log4j source=launchermxmlc.mxres.log 
| stats earliest(_time) as start 

As always when a stats has no group-by field, this will produce only a single output row. that row has one field named "start".

the next command will add two fields to that single row, but it will not add any rows.

| appendcols [ search eventtype=mlc sourcetype=sun_jvm service_name=mxmlc.gc.log | table relative_time after_gc max_gc ] 

The next command is a strange-looking way to create a new field called "start_time" and set it to the value of "start". If there were ever more than one row here, it would make more sense, so I suspect at one point the earlier clauses were quite different.

| eventstats max(start) as start_time 

Then there follow an eval and a fields clause, neither of which can ever create more than our one row. So when the timechart clause receives its input rows, there will only be one value coming in, and only one value of "_time".

0 Karma

johnraftery
Communicator

Hi, thanks very much for giving a reply, but one point I would make is that the query does work when I set span=1s. And if I take off the timechart command altogether, leaving me with this query:

eventtype=mlc sourcetype=murex_log4j source=launchermxmlc.mxres.log | stats earliest(_time) as start | appendcols [ search eventtype=mlc sourcetype=sun_jvm service_name=mxmlc.gc.log | table relative_time after_gc max_gc ] | eventstats max(start) as start_time | eval _time = start_time + relative_time | fields - start - start_time - relative_time

I end up with 33 rows of results; the appendcols has provided the extra rows.

0 Karma

sideview
SplunkTrust
SplunkTrust

Oh right. Sorry I forgot that appendcols will happily append extra rows if the base search has fewer rows. It's a strange way to do that. By way of apology, here is your search refactored as a disjunction without the appendcols:

eventtype=mlc (sourcetype=murex_log4j source=launchermxmlc.mxres.log ) OR ( sourcetype=sun_jvm service_name=mxmlc.gc.log ) 
| eval log4j_start=if(sourcetype="murex_log4j",_time,null())
| fields log4j_start relative_time after_gc max_gc
| eventstats min(log4j_start) as start_time
| eval _time = start_time + relative_time 
| fields - start - start_time - relative_time 
| timechart fixedrange=false bins=2000 max(after_gc), max(max_gc)
0 Karma

johnraftery
Communicator

Cool, I've never seen it done like that - thanks!

0 Karma

ppablo
Retired

Hi @johnraftery

If the answer by @sideview solved your question, please don't forget to resolve the post by clicking "Accept" directly below his answer. Also, be sure to upvote his answer and/or comment that helped you out.

0 Karma

sideview
SplunkTrust
SplunkTrust

Actually this particular answer+comment thread didn't resolve his issue - That seems to be a bug or at least a strange convention in timechart. I just refactored his search a bit as a side matter.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...