Splunk Search

Teaching Splunk the fields in a custom log format

dpadams
Communicator

I'm new to Splunk and may have a question that's a bit out of my depth. I've got Splunk configured now to aggregate a collection of standard Web logs from a group of servers. The next step I'd like to take is to have Splunk integrate a custom log format from our server app. The log format is entirely up to us but I figure it should include:

-- Timestamp for Splunk -- Host name/address for Splunk

Beyond that, we'll want several custom fields like an error name, diagnostic text and the request URL for a Web-provoked error.

What I'm trying to sort out is how to teach Splunk what my fields mean. I've seen mention of transform.conf files but don't follow how this would work. I've seen a few other questions but didn't quite nail the answer.

It seems like this out to be a standard, if not universal, thing to do with Splunk. Can anyone point me in the right direction?

Thanks very much for any help.

0 Karma
1 Solution

lguinn2
Legend

There are several ways to accomplish this, but the easiest way is to create your log with a format similar to

timestamp host=hostname errorName=xxxx errorText="your message here" requestURL=http://page.site.com/path/page.html (and so on)

Splunk can automatically process most common timestamp formats, so just pick something sensible. Your timestamp should definitely include a timezone designation.

For the remainder of the log format, use the form name=value and Splunk will automatically identify your fields for you. BTW, it is not necessary for every log entry to contain exactly the same fields. Put quotes around values that contain whitespace (like the error text in the example).

In my example, I used camel case for the field names, but you can name the fields whatever you like -- using letters, numbers and underscores. The field name must begin with a letter. So use error_text instead of errorText if you prefer. BTW, field names are case-sensitive: errortext and errorText are not the same thing.

If you do it this way, your log will contain more characters. If you want to go with a more compact log format, you will need to define field extractions.

View solution in original post

lguinn2
Legend

There are several ways to accomplish this, but the easiest way is to create your log with a format similar to

timestamp host=hostname errorName=xxxx errorText="your message here" requestURL=http://page.site.com/path/page.html (and so on)

Splunk can automatically process most common timestamp formats, so just pick something sensible. Your timestamp should definitely include a timezone designation.

For the remainder of the log format, use the form name=value and Splunk will automatically identify your fields for you. BTW, it is not necessary for every log entry to contain exactly the same fields. Put quotes around values that contain whitespace (like the error text in the example).

In my example, I used camel case for the field names, but you can name the fields whatever you like -- using letters, numbers and underscores. The field name must begin with a letter. So use error_text instead of errorText if you prefer. BTW, field names are case-sensitive: errortext and errorText are not the same thing.

If you do it this way, your log will contain more characters. If you want to go with a more compact log format, you will need to define field extractions.

dpadams
Communicator

Thanks very much for the help, it's much appreciated. The details and key terms you provided give me a good path forward. I'll be digging into the field extractions features and docs next. -- Thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...