Splunk Search

Why is the regex in inputs.conf not working for monitoring my log files?

robertlynch2020
Motivator

Hi

I have the following file in multiple sub directories. I am trying to pick them up but the below is not working and i can't crack it.
The regex is good, but it just won't take them it... any help would be super... I am thinking something very small is wrong here.

-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:24 mxtiming_730010_dell427srv_121.log
-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:26 mxtiming_730018_dell427srv_504.log
-rw-rw-r-- 1 autoengine murex 4772 Apr 6 17:27 mxtiming_730022_dell427srv_531.log

[monitor:///net/dell427srv//data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../*.log]
disabled = false
host = RSAT_Campaign
index = mlc_live
whitelist = mxtiming_\d+_\w+_\d+.*\.log$
sourcetype = MX_TIMING
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi,
Can you share sn example of not wanted files?
At a first sight you could insert part of filename in monitor
[monitor://net/dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../mxtiming_*.log]
Beware that there is a double slash in tour path.
Every way you could change your regex in this way
mxtiming_[^]+[^]+[^.]+.log
Bye.
Giuseppe

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Can you please give the full path to a few of these? Feel free to alter directory names as needed for confidentiality, but we need to see the way the full path looks in order to check some things.

0 Karma

robertlynch2020
Motivator

Thanks for your help on this , I think we are close.
To explain the issues more.

Wanted Files
Any sub directory of the main

[dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART]

 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_730010_dell427srv_121.log

 Or 
 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/X/mxtiming_730018_dell427srv_504.log    
 Or.
 Etc..

 mxtiming_730010_dell427srv_121.log
 mxtiming_730018_dell427srv_504.log
 mxtiming_730022_dell427srv_531.log

Unwanted Files
The files I don’t want are below and again they can also come into any subdirectory

 mxtiming_commoditynearbyonindexhistoricaldata_20170306-093752167_1294331273_6902.log
 mxtiming_commoditynearbyonindextofutures_20170306-093752167_1718781102_6902.log
 mxtiming_datamartdatasetlabelstransfer_20170306-093752167_1714912538_6902.log
 mxtiming_fillcommoditytimeunits_20170306-093752167_1066971732_6902.log
 mxtiming_obsolete_typology_cleanup_in_stp_rights_20170306-093752167_1206801397_6902.log
 mxtiming_new_stp_rights_migration_20170306-093752167_252516786_6902.log
 mxtiming_adapt_warehouse_rebuild_20170306-093752167_1385637444_6902.log
 mxtiming_updatepricingbookingpretraderouters_20170306-093752167_904493553_6902.log
 mxtiming_collateralinterestopsobjectupgrade_20170306-093752167_1527129704_6902.log
 mxtiming_refreshaccountssidata_20170306-093752167_421251909_6902.log

 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
 Or 
 dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/Z/A/mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log
 OR .
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi,
Can you share sn example of not wanted files?
At a first sight you could insert part of filename in monitor
[monitor://net/dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../mxtiming_*.log]
Beware that there is a double slash in tour path.
Every way you could change your regex in this way
mxtiming_[^]+[^]+[^.]+.log
Bye.
Giuseppe

0 Karma

robertlynch2020
Motivator

Hi

I have files like below that i don't want to take in.
mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log

Cheers for you help

0 Karma

robertlynch2020
Motivator

Hi No Luck, i tried this but nothing come out

[monitor://net/dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/.../*.log]
disabled = false
host = RSAT_Campaign
index = mlc_live
whitelist = mxtiming_[^]+[^]+[^.]+.log$  
sourcetype = MX_TIMING
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi robertlynch2020,
sorry but I answered using my smartphone that has limited function keyboards!
regex isn't correct, try:

mxtiming_[^_]+_[^_]+_[^\.]+\.log$

Bye.
Giuseppe

robertlynch2020
Motivator

Hi - This worked thanks 🙂

0 Karma

robertlynch2020
Motivator

Thanks for your help on this , I think we are close.
To explain the issues more.

Wanted Files
Any sub directory of the main

[dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART]

dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_730010_dell427srv_121.log

Or 
dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/X/mxtiming_730018_dell427srv_504.log  
Or.
Etc..

mxtiming_730010_dell427srv_121.log
mxtiming_730018_dell427srv_504.log
mxtiming_730022_dell427srv_531.log

Unwanted Files
The files I don’t want are below and again they can also come into any subdirectory

mxtiming_commoditynearbyonindexhistoricaldata_20170306-093752167_1294331273_6902.log
mxtiming_commoditynearbyonindextofutures_20170306-093752167_1718781102_6902.log
mxtiming_datamartdatasetlabelstransfer_20170306-093752167_1714912538_6902.log
mxtiming_fillcommoditytimeunits_20170306-093752167_1066971732_6902.log
mxtiming_obsolete_typology_cleanup_in_stp_rights_20170306-093752167_1206801397_6902.log
mxtiming_new_stp_rights_migration_20170306-093752167_252516786_6902.log
mxtiming_adapt_warehouse_rebuild_20170306-093752167_1385637444_6902.log
mxtiming_updatepricingbookingpretraderouters_20170306-093752167_904493553_6902.log
mxtiming_collateralinterestopsobjectupgrade_20170306-093752167_1527129704_6902.log
mxtiming_refreshaccountssidata_20170306-093752167_421251909_6902.log

dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/X/Y/A/mxtiming_adaptposnbstoredpltables_20170306-093752167_44364646_6902.log
Or 
dell427srv/data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/A/Z/A/mxtiming_removecommodityfuturesindexplinstruments_20170306-093752167_222279393_6902.log
OR .

…etc..

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi robertlynch2020,,
try this one, it should be correct (see https://regex101.com/r/8Mzm3g/1)

mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log

Bye.
Giuseppe

robertlynch2020
Motivator

Hi

So it looks like i get this to work, with the regex - thanks.
whitelist = mxtiming_(?\d*[^]_\d).log

0 Karma

woodcock
Esteemed Legend

Try this change:

 [monitor:///net/dell427srv//data1/apps/QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART/*/]

This assumes only a single directory layer between QCST_DBS_RSAT_v3.1.38_MASTER_DONOTRESTART and your files. Leave everything else the same. Restart your forwarder's splunk instance.

0 Karma

robertlynch2020
Motivator

Hi, Thanks, but these files could be multiple levels down from the main directory

0 Karma

robertlynch2020
Motivator

Hi - I tried this, but it only give me the subdirectories and not the main directory.

So it looks like i need to lines in my inputs.conf to get the main directory and all its subdirectories

[monitor:///net/dell425srv/dell425srv/apps/SPLUNK_BACK_UP_LIVE/MXTIMING_MEDIUM5/*.log]
disabled = false
recursive = true
host = MXTIMING_LIVE_TEST5
index = mlc_live
whitelist = mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log
crcSalt = <SOURCE>
sourcetype = MX_TIMING

[monitor:///net/dell425srv/dell425srv/apps/SPLUNK_BACK_UP_LIVE/MXTIMING_MEDIUM5/.../*.log]
disabled = false
recursive = true
host = MXTIMING_LIVE_TEST5
index = mlc_live
whitelist = mxtiming_(?<ppp>\d*_[^_]*_\d*)\.log
crcSalt = <SOURCE>
sourcetype = MX_TIMING
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...