Alerting

How to compare a json value field in a 1h window, and detect a 50% increase in the value

matiasburni
Engager

Hi all,

I have a search showing the memory usage of an specific process of the server. Why am I doing this with Splunk and not with Nagios? Because in the JSON log we have detailed data of the tool running in the server which allows us to identify the UCase which triggered the event.

I would like to detect a 50% INCREASE of the memory usage in a period of 1 hour, and configure an ALERT to send an email to the admins.

Base search:

host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | timechart span=1m avg(extra.memory) AS Memory

I tried with diff, but I can't specify the 1h span, just saying the position:

host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | rename extra.memory as Memory|  diff pos1=1 pos2=500 attribute=Memory

JSON EXAMPLE - BEFORE

{ [-] 
    extra: { [-] 
      build_number:  3082 
      builder_name:  master-preintegration 
      duration:  2996 
      memory:  10876836
      message:  build finished 
      parent_build: 
      reason:  SUCCESS 
      slave_name:  tlsisbld108lbuild 
      url:  https://buildbot.tl.intel.com/absp/builders/master-preintegration/builds/3082 
   } 
    funcName:  buildFinished 
    host:  tlsbldmaster001 
    levelname:  INFO 
    levelno:  3 
    lineno:  644 
    logger:  events_log.master 
    master:  master 
    message: 
    path:  /data/prod/src/buildbot/master/buildbot/process/build.py 
    process:  54608 
    tags: [ 
   ] 
    timestamp:  2015-10-28 10:59:50,210 
    type:  Splunk 
}

JSON EXAMPLE - AFTER

{ [-] 
    extra: { [-] 
      build_number:  6670 
      builder_name:  master-engineering 
      duration:  2185 
      memory:  78950620 
      message:  build finished 
      parent_build: 
      reason:  SUCCESS 
      slave_name:  shsisbld009lbuild 
      url:  https://buildbot.tl.intel.com/absp/builders/master-engineering/builds/6670 
   } 
    funcName:  buildFinished 
    host:  tlsbldmaster001 
    levelname:  INFO 
    levelno:  3 
    lineno:  644 
    logger:  events_log.master 
    master:  master 
    message: 
    path:  /data/prod/src/buildbot/master/buildbot/process/build.py 
    process:  24041 
    tags: [ 
   ] 
    timestamp:  2015-10-28 11:17:11,264 
    type:  Splunk 
}
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Assuming you plan to run this search every hour, try something like this

 host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | stats latest('extra.memory') AS Current earliest('extra.memory') as LastHour | where Current>=LastHour*1.5

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Assuming you plan to run this search every hour, try something like this

 host="10.102.165.212" sourcetype=_json | spath| WHERE LIKE(master, "master") | stats latest('extra.memory') AS Current earliest('extra.memory') as LastHour | where Current>=LastHour*1.5
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...