Getting Data In

Index Time Sourcetyping

tprzelom
Path Finder

props.conf:
[pan_event]
TRANSFORMS-traffic = traffic_source

transforms.conf:
[traffic_source]
REGEX = (,TRAFFIC,)
FORMAT = sourcetype::pan_traffic
DEST_KEY = MetaData:Sourcetype

I've got data being indexed as sourcetype=pan_event. I have distributed the above props and transforms to the indexers and restarted them, but I am unable to create a pan_traffic sourcetype. Where am I going wrong?

1 Solution

tprzelom
Path Finder

The issue was the light forwarder was being managed by cfengine3 and was transitioned to chef and something was lost in the transition which resulted in partial functionality.

My solution was rip all that out and use a universal forwarder that has a deploymentclient.conf sent to it by chef. Then use the deployment server from there.

View solution in original post

tprzelom
Path Finder

The issue was the light forwarder was being managed by cfengine3 and was transitioned to chef and something was lost in the transition which resulted in partial functionality.

My solution was rip all that out and use a universal forwarder that has a deploymentclient.conf sent to it by chef. Then use the deployment server from there.

_d_
Splunk Employee
Splunk Employee

You don't need parens, nor do you need to escape the commas: REGEX= TRAFFIC would be fine.

Is the data coming to the indexers with sourcetype=pan_event or is it transformed to be become that? If the latter, you need to scope the transform on source instead. Is it perhaps coming from a heavy forwarder?

ShaneNewman
Motivator
REGEX= \,TRAFFIC\,
0 Karma

tprzelom
Path Finder

I'm using
REGEX = ,TRAFFIC,
with no success

There is no space after the comma. This feels like something more sinister that just a regex. I had a much more complex system in place that all of a sudden stopped working, so I dialed it back to basics to try and uncover the problem. These logs are coming through an rsyslog tier, could there be some kind of metadata affecting the logs?

0 Karma

sdaniels
Splunk Employee
Splunk Employee

Yep, forget those and it seems like you might have a space with the comma before traffic? Not sure though. You could remove those or go with REGEX= ,\sTRAFFIC,

0 Karma

kristian_kolb
Ultra Champion

I think you can skip the parenthesis in the REGEX.

/k

0 Karma

tprzelom
Path Finder

Nov 5 15:32:38 hostname.net 1,2013/11/05 15:32:38,0002C100698,TRAFFIC,end,1,2013/11/05 15:32:37,IP_ADDR,IP_ADDR,IP_ADDR,IP_ADDR,firewall_rule,,,incomplete,vsys1,trust,untrust,ethernet1/2,ethernet1/1,anycast_vip_IP_ADDR,2013/11/05 15:32:37,159998,1,47031,3978,47031,3978,0x400000,tcp,allow,74,74,0,1,2013/11/05 15:32:33,0,any,0,12676624,0x0,IP_RANGE,IP_RANGE,0,1,0

hot off the press

0 Karma

sdaniels
Splunk Employee
Splunk Employee

Can you post a couple of your events. Typically this is regex related because the rest of what you have looks good.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...