Splunk Search

need to filter out those error which occur due to a particular deployment for that sourcetype.

Nisha18789
Builder

I have a site and errors on that site are being recorded in splunk.
I basically need to filter out those error which occur due to a particular deployment for that sourcetype.
suppose before 10 days of deployment Errors(Exceptions) were: A,B, C and D
after 2 days of deployment errors were: A, B,C, D and E

so need to prepare a query which gets the result as E i.e, the new errors that are occurring after the recent deployment.

we need to compare the Message's first 100 characters of today's error with that of previous days. And if the message has occured before also, then we will ignore that message in result. The result should include only fresh messages after a particular deployment.
also the only characteristic feature of the error is 'Message' which has value like: 'Advisory Code: 1315 was not found in the CMS....' we need to compare first 100 letters approx.

please help me out....

I am posting the event logs for reference:

exception on 23 Oct and 25 Oct:

20131025091329.000000
Category=0
CategoryString=NULL
EventCode=0
EventIdentifier=0
EventType=1
Logfile=BookingEngine
RecordNumber=34135049
SourceName=BookingEngine
TimeGenerated=20131025131329.000000-000
TimeWritten=20131025131329.000000-000
Type=Error
User=NULL
ComputerName=cclprdecoweb25
OriginalComputerName=cclprdecoweb25.cclinternet.com
wmi_type=WinEventLog:BookingEngine
Message=Error in SubmitPrimaryGuestInfoException on 25 Oct:

20131025091138.000000
Category=0
CategoryString=NULL
EventCode=0
EventIdentifier=0
EventType=1
Logfile=BookingEngine
RecordNumber=34408305
SourceName=BookingEngine
TimeGenerated=20131025131138.000000-000
TimeWritten=20131025131138.000000-000
Type=Error
User=NULL
ComputerName=cclprdecoweb26
OriginalComputerName=cclprdecoweb26.cclinternet.com
wmi_type=WinEventLog:BookingEngine
Message=An exception of type System.Data.EntityCommandExecutionException was encountered while executing storedprocedure: SP_ADD_TO_BOOKING_CART, with sql parameters as :89097031,,0,1/1/0001 12:00:00 AM,

Here what is needed it the exception that occured on 25th oct only..
Please help me out...

Tags (1)
0 Karma

kristian_kolb
Ultra Champion

You will want to do any comparison etc on extracted fields. So the ~100 characters stuff is not really relevant (probably). So the question is how to identify the relevant parts of the events and extract that information.

Could you post a complete event?

There are multiple ways for filtering events out based on previously seen field values.


UPDATE:

Given your other question ( http://answers.splunk.com/answers/107577/filtering-errors-based-on-two-timestamps ); something like this could perhaps be a start;

in props.conf

[your sourcetype here]
EXTRACT-first_hundred = (?<short_msg>.{100}) in Message

This will create a truncated field called short_msg from the Message field for all events of the specified sourcetype. Then search like so;

sourcetype=blah earliest=-2d@d NOT [search sourcetype=blah earliest=-32d@d latest=-2d@d | dedup short_msg | fields + short_msg] | top 100 short_msg

Essentially what this does is to run the subsearch first (within square brackets), and this returns the list of previously seen short_msg's. This list is then added to the outer search with a NOT condition, effectively removing them from the search results.

The outer search then works like;

sourcetype=blah earliest=-2d@d NOT (short_msg=some_message OR short_msg=some_other_message OR short_msg=yet_another_message)

After that, the top command is used to show the 'new' errors that have occurred in the last two days.

Please note that you'll have to adjust the earliest/latest conditions (of both searches) to reflect on how soon after the deploy you run the query.

/K

0 Karma

kristian_kolb
Ultra Champion

Well, IFX may not be able to create the regex for you straight away, but with access to IFX you have the vehicle with which you can store a field extraction in the environment.

IFX will generate a regex for you based on the text in the events you copy/paste, right. This regex will most likely not be generated properly, but you can edit the regex that IFX suggests.

In your case it might be something like;

(?m)Message=(?P<short_msg>.{100})

I suggest you play around a little bit with the tool to get the hang of it.

/K

0 Karma

Nisha18789
Builder

Yes I have access to IFX. I tried to use it according to the link mentioned but was not able to get what I want. Could u please help me out again. Thanks.

0 Karma

kristian_kolb
Ultra Champion
0 Karma

Nisha18789
Builder

Hey I don't have admin rights on splunk so cannot add in props.config. Is there any other way out?

0 Karma

Nisha18789
Builder

Hey could you please help with what do you mean by "in props.conf". Where do I need to write below query:
[your sourcetype here]
EXTRACT-first_hundred = (?.{100}) in Message

0 Karma

Nisha18789
Builder

Thanks Kristian,I will try that out tommorow and will post my findings. Thanks for help.

0 Karma

kristian_kolb
Ultra Champion

see update

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...