How can I index all events without a specific word from a monitor stanza?

[davidepala]

Hi all,
I'm trying to index all events without a specific word from a monitor stanza. This is my input.conf:

[default]
host = srvname
[monitor://C:\inetpub\logs\LogFiles\W3SVC1]
source = ExchangeIISLog
index=exchange_iis
ignoreOlderThan = 1d`

i've configured a stanza in props.conf and transform.confhere are the stanzas

props.conf

[source::ExchangeIISLog]
TRANSFORMS-MBdrop = ExchangeIISLogFilter

transform.conf

[ExchangeIISLogFilter]
REGEX = (?m)^(.*HealthMailbox.*)$
DEST_KEY = queue
FORMAT = nullQueue

i want index only event without the word "healthmailbox" .... below an example of the txt file

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2018-02-07 13:00:04
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken


2018-02-07 13:00:04 192.168.1.71 GET /Microsoft-Server-ActiveSync/healthcheck.htm
- 80 - 192.168.1.111 - - 200 0 0 15 2018-02-07 13:00:04 192.168.1.71 GET /ews/healthcheck.htm - 80 -
192.168.1.111 - - 200 0 0 15 2018-02-07 13:00:04 192.168.1.71 GET /rpc/healthcheck.htm - 80 -
192.168.1.103 - - 200 0 0 15 2018-02-07 13:00:04 192.168.1.71 GET /owa/healthcheck.htm &encoding=; 80 -
192.168.1.103 - - 200 0 0 15 2018-02-07 13:00:05 192.168.1.71 GET /autodiscover/healthcheck.htm - 80 -
192.168.1.111 - - 200 0 0 46 2018-02-07 13:00:14 192.168.1.71 GET /Microsoft-Server-ActiveSync/healthcheck.htm
- 80 - 192.168.1.111 - - 200 0 0 15 2018-02-07 13:00:14 192.168.1.71 GET /rpc/healthcheck.htm - 80 -
192.168.1.111 - - 200 0 0 31 2018-02-07 13:00:14 192.168.1.71 GET /owa/healthcheck.htm &encoding=; 80 -
192.168.1.103 - - 200 0 0 31 2018-02-07 13:00:14 192.168.1.71 GET /ews/healthcheck.htm - 80 -
192.168.1.103 - - 200 0 0 31 2018-02-07 13:00:15 192.168.1.71 GET /autodiscover/healthcheck.htm - 80 -
192.168.1.103 - - 200 0 0 15 2018-02-07 13:00:18 ::1 GET /Microsoft-Server-ActiveSync/default.eas &CorrelationID=<empty>;&cafeReqId=be1ce778-6b21-4a8b-a776-045738407c8d; 443 HealthMailbox0d8a2628ac704cf8ae7afa1c86846f0f@gesca.it ::1 AMProbe/Local/ClientAccess - 200 0 0 31 2018-02-07 13:00:18 ::1 GET /ecp/ &CorrelationID=<empty>;&cafeReqId=1631e6ea-8235-42ab-9931-a95d19a0a656;&LogoffReason=NoCookiesGetOrE14AuthPost 443 - ::1 AMProbe/Local/ClientAccess - 302 0 0 31 2018-02-07 13:00:18
127.0.0.1 GET /AutoDiscover/ &CorrelationID=<empty>;&cafeReqId=e7dd627f-62e2-4a17-b7e8-2c1418c7835b; 443 GESCA\HealthMailbox0d8a262
127.0.0.1 AMProbe/Local/ClientAccess - 200 0 0 62 2018-02-07 13:00:23
127.0.0.1 GET /ews/ &CorrelationID=<empty>;&cafeReqId=d583378e-ee9a-496c-b51c-a84508702876; 443 GESCA\HealthMailbox0d8a262
127.0.0.1 AMProbe/Local/ClientAccess - 200 0 0 31 2018-02-07 13:00:23
192.168.1.71 GET /owa/healthcheck.htm &encoding=; 80 - 192.168.1.103 - - 200 0 0 46 2018-02-07 13:00:23
192.168.1.71 GET /rpc/healthcheck.htm - 80 - 192.168.1.111 - - 200 0 0 46 2018-02-07 13:00:23 192.168.1.71 GET /ews/healthcheck.htm - 80 -
192.168.1.111 - - 200 0 0 46 2018-02-07 13:00:23 192.168.1.71 GET /Microsoft-Server-ActiveSync/healthcheck.htm
- 80 - 192.168.1.103 - - 200 0 0 46 2018-02-07 13:00:26 192.168.1.71 GET /autodiscover/healthcheck.htm - 80 -
192.168.1.111 - - 200 0 0 0

[davidepala]

I don't know why but using props.conf and transform.conf on the UF it works ... can someone explain why? It's the working scenario:

UF (IIS on windows. custom props.conf and transform.conf) => UF (linux) => INDEXER (linux, custom props.conf and transform.conf) 
with  ./splunk btool props list | grep -A 30 ExchangeIISLog

I can see the correct transform rule:

[source::ExchangeIISLog]
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
HEADER_MODE =
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MATCH_LIMIT = 100000
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRANSFORMS-MBdrop = ExFilter

NOTE i've changed the stanza name from the original post, here is transform.conf, the stanza is the same of UF on windows machines

[ExFilter]
REGEX = (?msi)(.*HealthMailbox.*)
DEST_KEY = queue
FORMAT = nullQueue

It makes me crazy!!!!

[Yunagi]

Can you verify whether you are running a universal forwarder or a heavy forwarder? Check the installation directory. All I can think of is that your universal forwarder is in reality a heavy forwarder.
For the universal forwarder, the installation directory should be /opt/splunkforwarder or c:\Program Files\SplunkUniversalForwarder.
For the heavy forwarder, it should be /opt/splunk or c:\Program Files\Splunk.

[davidepala]

it's universal forwarder ....

[Yunagi]

I noticed that you wrote transform.conf instead of transforms.conf. Make sure the file is named transforms.conf.

The regex should be:

REGEX = HealthMailbox

Restart Splunk after making those changes.

[davidepala]

is a typo, i've already other working stanza inside transforms.conf

[davidepala]

Sory i forgot the reason of my question! And yes: my conf not work. As i write above it's the regex REGEX = (?m)^(.HealthMailbox.)$ there are " * " after " . ". Now, i've tried with your suggestion and it not work, this is transform .conf stanza

[ExchangeIISLogFilter]
REGEX = "HealthMailbox"
DEST_KEY = queue
FORMAT = nullQueue
~

and this is the result

As you can see i've some events HealthMailbox..... that i want remove from the indexing process.

Tnx

[Yunagi]

Try REGEX = HealthMailbox instead of REGEX = "HealthMailbox".

Do you have a heavy forwarder in place? Perhaps there is a heavy forwarder between the indexer and a universal forwarder?
If so, you need to put this configuration on the heavy forwarder.

[davidepala]

Tnx Yunagi, i've tried with and without double quote, and no, i dont have HF only a universal forwarder between the UF on the monitored machine and the indexer

[Yunagi]

So you have a universal forwarder on a Windows system which forwards to another universal forwarder which in turn forwards to the indexer? Are you sure the intermediate Splunk instance is a universal forwarder?

You can use btool to troubleshoot your configuration. Run "splunk btool props list" and "splunk btool props list" to see the actual configuration which Splunk uses. Check that your stanzas appear as expected.

[harsmarvania57]

Hi @davidepala,

On which splunk server have you configured props.conf and transforms.conf ? This props.conf and transforms.conf should be on Indexer OR Heavyforwarder which ever comes first from UF -> Indexer path.

[davidepala]

it's on indexer ... i've only an indexer behind a forwarder, the forwarder installed on windows send data to the universal forwarder and then on the indexer with those file

[FrankVl]

So you have:
FW on Windows -> UF -> Indexer?

What type of Forwarder is on the Windows box? Heavy or Universa?

[somesoni2]

Try with this for your transforms.conf entry (keep all others)

[ExchangeIISLogFilter]
REGEX = HealthMailbox
DEST_KEY = queue
FORMAT = nullQueue

[FrankVl]

And the question is why the config you have isn't working, I assume?

Don't think that regex is correct:
^ matches start of string
. matches a single character (any character)
HealthMailbox matches the word you want to detect
. matches again a single character
$ matches end of string

So your regex only matches lines that exist of only your keyword, with just 1 character in front of it and 1 character behind. You would need to replace the . with .*, to make the regex match the whole lines.

For the filtering to work, you don't need to write a regex that matches the whole line though, just write a regex that matches the relevant part will already cause it to trigger.

So just put your keyword in the regex:

REGEX="HealthMailbox"

That should do the trick, I think.