Knowledge Management

Transform on extracted field

pixelseventy2
Explorer

Hi. I'm using OSSEC to send logs from a variety of sources by syslog to Splunk. I've created transforms to extract the fields, the main one of which gets extracted as "message". One of the log sources is ISA Server, and I'm trying to do some extractions on those fields. I've got a transform to change the "host" to the original host, not the OSSEC server. Everything has the same sourcetype, which I tried to change using a "[host::" entry in props.conf, but this failed because, I think, it's looking at the original host, not the transformed one.

I can get an extraction to work against the "_raw" field, by not against the "message" field. Is it possible to get an extract to work against a field extracted an search time? I've tried referencing the field as "message" and "field:message", but the extract fails. I can get a more accurate regex if I can reference this field.

Alternatively, and this might be better, is to use a delimiter transform rather than regex if I can get that to work on the "message" field, but I need to be able selectively target just the ISA logs.

Hope this makes some sense.

Thanks

Tags (2)
0 Karma

pixelseventy2
Explorer

In props.conf I had:

[ossec_alerts]
REPORT-ossec-isa = ossec-splunk-isa

in transforms.conf I tried a number of permutations, but what I want is:

[ossec-splunk-isa]
SOURCE_KEY = field:message
REGEX = ^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\t[^\t]+\t(?P.*?)\t\d{4}-\d{2}-\d{2}\t\d{2}:\d{2}:\d{2}\t[^\t]+\t[^\t]+\t(?P[^\t]+)\t\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\t

thanks

0 Karma

pixelseventy2
Explorer

Hi. OSSEC is a log server, which collects logs and forwards alerts using syslog. ISA logs look like this as they are forwarded, the "message" field is tab-delimited:

Feb 3 08:40:19 log-01 ossec: crit=3
id=108001 description="ISA Internet
access permitted" component="(isa01)
172.19.1.1->\ISALogs\ISALOG_20140203_WEB_000.w3c",
classification=" proxy,allowed,",
src_ip="172.19.2.2", acct="anonymous",
message="172.19.2.2 anonymous ... 0x800
Allowed"

Sorry, ISA Logs are too long to post here in full ...

0 Karma

bandit
Motivator

I would first recommend checking the regex from the search bar to validate the field extraction works. Then, once working, you could move it to props and transforms.

i.e.

your search | rex field=message (?<my_field>my capture pattern here)
0 Karma

lguinn2
Legend

A-ha! What you are asking is to create a new field based on an existing field - when both fields are extracted at search time!

AFAIK, this is not possible in props.conf or transforms.conf - but you can do it with the rex command, as Rob illustrates above.

0 Karma

pixelseventy2
Explorer

Hi. my regex works fine, using rex and as a transform, if I run it on the "_raw" field. It runs fine using rex against the extracted "message" field. I just can't get it to run as a transform against the "message" field.

0 Karma

lguinn2
Legend

Your question makes sense - but a lot of us are not familiar with OSSEC or the ISA Server logs. A few lines of the log would be very helpful! Otherwise, only a subset of the folks on this forum will be able to answer your question.

Also, the relevant props.conf and transforms.conf stanzas would be helpful as well.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...