Getting Data In

Defining fields in transforms.conf

kmattern
Builder

Splunk Version 4.0.11

I’m working on getting Splunk to consume “log” files that I have dumped from a SQL Server helpdesk database. There are a lot of date and time fields in the events as well as dates and times that are included in the free-form text fields. I have managed to extract fields for my journal logs. My assignment log looks like this


2011-10-03^09:15:37^ ^ ^00021361^Brandon^ ^ ^
2011-10-03^08:10:35^2011-10-03^14:51:43^00021584^Bonnie^Completed^15^
2011-10-03^08:25:28^2011-10-04^07:17:13^00021585^Bonnie^Completed^20^
2011-10-03^08:40:19^2011-10-05^13:40:38^00021587^Bonnie^Completed^30^
2011-10-03^08:59:21^2011-10-04^07:15:03^00021588^Bonnie^Completed^20^
2011-10-03^09:14:46^2011-10-03^14:05:20^00021589^Patrick^Completed^15^
2011-10-03^09:24:33^2011-10-04^07:02:54^00021591^Stephanie^Completed^10^
2011-10-03^10:21:26^2011-10-03^14:25:04^00021592^Jeff^Completed^15^
2011-10-03^10:42:09^2011-10-03^10:42:25^00021593^Robert^Completed^10^

transforms.conf looks like this


[AsgLog]
DELIMS="^"
FIELDS="DateAssign",TimeAssign","DateResolv","TimeResolv","CallID","Assignee","Resolution","TotalAsgnmntTime"

Splunk consumes the data and I can search it but the fields defined in transforms.conf are not available. I’ve tried extracting the fields but with two date fields and two time fields Splunk doesn’t seem able to discriminate between them.
What am I doing wrong?

Tags (1)
0 Karma

Jon_Webster
Splunk Employee
Splunk Employee

Here's a quick tip. You don't have to restart Splunk for changes to the .conf files, you can use the extract command with the reload=true option as documented in the Search command cheat sheet: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/SearchCheatsheet

Extract field/value pairs and reload field extraction settings from disk.
... | extract reload=true

0 Karma

Drainy
Champion

This isn't generally recommended. Although documented it doesn't actually always work very effectively. I have wasted many an hour thinking my configs were at fault when in fact they were fine but Splunk wasn't reloading them! 😉 Best practice would be to restart the system to be sure

0 Karma

tgow
Splunk Employee
Splunk Employee

You will have to backslash out the "^" because it is an anchor in REGEX.

I took your log snippet and used the following:

props.conf

[source::...timefile...]
sourcetype = timefile
REPORT-time = time

transforms.conf

[time]
DELIMS = "\^"
FIELDS = DateAssign,TimeAssign,DateResolv,TimeResolv,CallID,Assignee,Resolution,TotalAsgnmntTime

and now it is working fine creating the fields.

If this helps don't forget to give me some points.

Drainy
Champion

Good shout!

0 Karma

Drainy
Champion

Have you tried it with the FIELDS not surrounded by " "'s. E.g.

FIELDS=DateAssign,TimeAssign,DateResolv,TimeResolv,CallID,Assignee,Resolution,TotalAsgnmntTime 

Also, have you restarted Splunk? this is required to ensure that the configs are reloaded successfully.

0 Karma

kmattern
Builder

Yep! Been there and done that. I always remember to stop and restart Splunk.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...