Splunk Search

Why is Splunk combining several log lines into one event, and how do I search within these multiline events to get accurate count results?

joemadeus
New Member

We're trying to count the number of times a particular call is made to a service. To do that, we're logging a log line for every call, one that contains a well-known string, to a gunicorn file (one file per instance), which is then being picked up by Splunk for analysis.

When I search for these log lines, I find that what's returned are events, not lines -- and events have many lines in them. In some cases, I see four or six of the calls we're interested in logged, but count (when I use timechart) is equal to one. Emitting sum(linecount) doesn't produce the correct result either, since it emits the total number of lines in the event, not just the lines we're interested in. I can't seem to get it to return the correct number (four, six).

What's the way through? It seems like this is something Splunk should be able to do. I'm told it's possible that the parser we're using is wrong, but I don't see where I change that. Any hints would be very, very welcome.

Thanks for your time.

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee

The problem is that Splunk does not understand how to break up the lines in your data. To fix that, you need to set the correct LINE_BREAKER: http://docs.splunk.com/Documentation/Splunk/6.2.8/Data/Indexmulti-lineevents

Do you have a sourcetype for your data or did you use an existing one?

0 Karma

joemadeus
New Member

@mreynov, thanks. I did see this and ([\r\n]+) (the default) should work. I think SHOULD_LINEMERGE=false is even more useful -- but the docs only tell us how to set this for the Enterprise version. Is there a way to do this for the Cloud offering?

And, we used the existing sourcetype -- I didn't change it and I don't think anyone else did.

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee

You should be able to play around with sourcetype settings after clicking "Add Data" and using a file.
To get the file, you should be able to export some of your search results in raw format.
Then save the sourcetype and you will be able to use that when defining your actual input.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...