Getting Data In

Consume Free-Form text

kmattern
Builder

I have a help desk database in SQL Server that I want to export to log type files and have Splunk consume. I'm not having any trouble getting the data into Splunk but I can't seem to get Splunk to understand where the boundaries for each record/event is. I have defined my output as follows:


TicketNum=000001
CustName=Bob Smith
CallDate=2011-11-01
Status=Closed
CallDesc=Mr. Smith had trouble accessing his hydro accelerator while in mimsy mode.
CallResolution=Told Mr. Smith that he had to be sure his vorbis was in gear

Of course the CallDesc and CallResolution fields can be quite long. They contain copies of emails, comments and more. I have been careful to separate them with only line feeds. The only carriage return/linefeed is at the end of each record/event. There are 14 fields in each record/event.

When I run a search on the raw data many of the records/events run together and they do not necessarily break at the end of a record/event. Yet others do. I have set the DELIMS="\n" in transforms.conf but it doesn't seem to help.

Does anyone know how I can break these records/events out properly?

Thanks

Tags (3)
0 Karma
1 Solution

tgow
Splunk Employee
Splunk Employee

You will need to instruct Splunk that this is a mulit-line event and also tell it where the line breakers are. Assuming that the TicketNum field is where a new event starts try this in your $SPLUNK_HOME/etc/system/local/props.conf:

[yoursourcetype]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^TicketNum

View solution in original post

tgow
Splunk Employee
Splunk Employee

You will need to instruct Splunk that this is a mulit-line event and also tell it where the line breakers are. Assuming that the TicketNum field is where a new event starts try this in your $SPLUNK_HOME/etc/system/local/props.conf:

[yoursourcetype]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^TicketNum

_d_
Splunk Employee
Splunk Employee

Try using DELIMS="([\r\n])+" as there may be carriage returns and/or new lines.

Hope this helps

> please upvote and accept answer if you find it useful - thanks!

0 Karma

kmattern
Builder

That didn't seem to do anything different. Maybe part of the problem is that in the free-form text theare are usually a number of dates. Emails are copied comppletely into these records and that includes the date and time of the email. I moved all of my date fields to the top of the event but that didn't seem to help either.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...