Splunk Search

Why are some default fields not being extracted for data coming in via TCP syslog with my current props and transforms.conf?

dannestor
Explorer

I have data incoming via TCP syslog. I have created the following transforms to process them:

  • etc/system/local/props.conf:

    [source::tcp:1514]
    TRANSFORMS-windows = set_sourcetype_snare, set_source_wineventlog

  • etc/system/local/transforms.conf:

    [set_source_wineventlog]
    REGEX = AgentDevice=WindowsLog.AgentLogFile=(.?)\s
    FORMAT = source::WinEventLog:$1
    DEST_KEY = MetaData:Source

    [set_sourcetype_snare]
    REGEX = AgentDevice=WindowsLog
    FORMAT = sourcetype::windows_snare_syslog
    DEST_KEY = MetaData:Sourcetype
    These work as expected, and the source and sourcetype are set accordingly. However, I expected that setting these two fields would also trigger some other Splunk built-in transforms. For example:

    [splunk@l1807s local]$ ~/bin/splunk btool props list windows_snare_syslog
    [windows_snare_syslog]
    ...
    TRANSFORMS = syslog-host
    and

    [splunk@l1807s local]$ cat ~/etc/apps/Splunk_TA_windows/default/props.conf
    ...

    Apply the following properties to all Windows events

    [source::(MonitorWare|NTSyslog|Snare|WinEventLog|WMI:WinEventLog)...]
    ...
    FIELDALIAS-event_id_for_windows = RecordNumber as event_id
    ...
    As far as I can tell, the default processing is not happening. I see the source and sourcetype fields as set by my transforms, however for example I don't find the field event_id, and host is incorrectly set. What am I doing wrong here, and how can I achieve the intended behaviour?

0 Karma

woodcock
Esteemed Legend

Although you can change the host and sourcetype fields, all configurations inside of props.conf may reference the original/pre-overridden values if the overriding happens afterwards.

0 Karma

dannestor
Explorer

Are you sure about this? This documentation page states the contrary: http://docs.splunk.com/Documentation/Splunk/6.3.1511/Admin/Wheretofindtheconfigurationfiles

0 Karma

woodcock
Esteemed Legend

Yes, I am sure: I am battle-tested on this. Where in that link do you see anything to the contrary? Please post the text in a comment/followup.

0 Karma

dannestor
Explorer

Another thought: in my example, isn't event-id a field extracted at search time? Shouldn't it pick up changes to source and sourcetype, which are made during indexing?

0 Karma

dannestor
Explorer

Under the heading "precedence in a global context" (which is the context where indexing happens), the system local directories have the highest priority. I just realized however that this applies only to the situation where the same attribute is defined in multiple files, and says nothing about the order of evaluation of different attributes, which is what you were referring to. Do you have a documentation link where the latter is documented?

0 Karma

Runals
Motivator

Is the data coming in and the props/transforms on the same Splunk instance or is there some separation?

0 Karma

dannestor
Explorer

No, there isn't any separation. It's a single-machine test installation.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I recommend you take the transforms and other actions you desire from other sources/sourcetypes and copy them to your own props.conf and transforms.conf files. That will ensure they work and protect you from future changes made to the other apps.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...