All Apps and Add-ons

Regex not working in props.conf as per when searched with Rex command

Dark_Ichigo
Builder

I have created a regex that works fine during search time, but when added to props.conf and/or transforms.conf to extract the field during index time, the field doesnt get extracted?

I dont understand how this could work during search time in the Splunk Search bar search page, but not when added to props.conf?

Here it is:

rex field=_raw "set=(?<phoneid>.+)\snotTime"
0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

^(?:.+?,){4}(?.+?),(?.+?),.+?,(?.+?),(?.*?),(?.+?),

piebob
Splunk Employee
Splunk Employee

please stop posting comments as new answers. thanks.

d646800
Explorer

i tried with the props only and i still cannot see the fields. has this anything to do with splunk 6.0.3. the other colleague of mine created field extraction and does not see them as well. it was ok two weeks ago before upgrade to splunk 6.0.3

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

props.conf
[sdf_bpel_metric]
REPORT-sdf_policy_metric = SDFCorepolicymetrics

transforms.conf
[SDFCorepolicymetrics]
REGEX = ^(?:.+?,){4}(?.+?),(?.+?),.+?,(?.+?),(?.*?),(?.+?),

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

props.conf only
EXTRACT-SDFCorepolicymetrics = ^(?:.+?,){4}(?.+?),(?.+?),.+?,(?.+?),(?.*?),(?.+?),

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee
  • | rex field=_raw "^(?:.+?,){4}(?.+?),(?.+?),.+?,(?.+?),(?.*?),(?.+?),"
0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee
0 Karma

d646800
Explorer

requestApplicationLabel [40-52] MetricLogger

requestTransactionID [53-67] TDI_CLOUDCSX_1

callingApplication [82-115] hymlxsdfbpe11_1401889362113_11537

callType [116-116] ``

function [117-140] RetrieveIdentityDetails

second log

requestApplicationLabel [63-75] MetricLogger

requestTransactionID [76-111] TELSTRA_PREPAIDACTIVATION_STRATEGIC

callingApplication [148-180] chslxsdfbpe05_1401889356427_2871

callType [181-181] ``

function [182-212] CCandB.CreateNewBillingAccount

i got this from regex101.com. and tested in search field in splunk. it was ok. unless it works differently?

0 Karma

d646800
Explorer

yep. here are two sample logs

2014-06-04 23:42:42,115,,,1401889361349,MetricLogger,TDI_CLOUDCSX_1,1401889361349,hymlxsdfbpe11_1401889362113_11537,,RetrieveIdentityDetails,148

2014-06-04
23:42:36,427,,,0dedf85a-fbdb-43cb-b9f1-d4a0f636ab97,MetricLogger,TELSTRA_PREPAIDACTIVATION_STRATEGIC,0dedf85a-fbdb-43cb-b9f1-d4a0f636ab97,chslxsdfbpe05_1401889356427_2871,,CCandB.CreateNewBillingAccount,2983

i tried two methods,
FIRST method, just in props as below. does not quite work. worked when i use rex field=_raw "regex" though in search field though. tested in on one of those regex online as well

[sdf_bpel_metric]
EXTRACT-SDFCorepolicymetrics = (?:[^,\n],){5}(?P[a-zA-Z]+),(?P[^,]),(?:[^,\n],)(?P[^,]),(?P[^,]),(?P[^,])

SECOND method

in props

[sdf_bpel_metric]
REPORT-sdf_policy_metric = SDFCorepolicymetrics

in transforms

[SDFCorepolicymetrics]
FORMAT = requestApplicationLabel::$1 requestTransactionID::$2 callingApplication::$4 callType::$5 function::$6
REGEX = ([a-zA-Z]+),([^,]),([^,]),([^,]),([^,]),([^,]*),

0 Karma

kristian_kolb
Ultra Champion

And perhaps of the (relevant portions of) props.conf, and perhaps inputs.conf as well (only the portion where you configure the input of this file).

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Can you provide a sample of the raw log?

0 Karma

kristian_kolb
Ultra Champion

This may indicate that the EXTRACT is not applied at all. Under what stanza header have you put the EXTRACT? Does this match the sourcetype/source/host?

Ayn
Legend

Just want to point out that you don't need to reingest the log, or restart Splunk. Field extractions happen (mostly) at search-time, regardless of if they happen in props.conf/transforms.conf or inline in your search.

0 Karma

Dark_Ichigo
Builder

Yes, thats exactly what I have set and it doesnt work, no matter how much I perform a restart or Log Re-ingestion.

0 Karma

linu1988
Champion

what's your props.conf settings? Is it the below or not?

EXTRACT-PHID= set=(?<phoneid>.+)\snotTime

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...