How to modify format of MS DNS server debug log events?

[tomasmoser]

Hello,

I would like to modify format of MS DNS debug logs in order to get rid of some unimportant strings within domain names. I was playing with SEDCMD stanza in props.conf but not with success.

Log format as extracted by Splunk add-on for Microsoft DNS:
1. 2. 2017 20:19:22 0D80 PACKET 0000002548D040A0 UDP Rcv 10.17.81.32 7be7 Q [0001 D NOERROR] A (5)h42-m(3)sec(3)lab(0)

The problem is with (5)h42-m(3)sec(3)lab(0)"

I need to get events to look as follows:

2017 20:19:22 0D80 PACKET 0000002548D040A0 UDP Rcv 10.17.81.32 7be7 Q [0001 D NOERROR] A h42-m.sec.lab

When I implemented this ...
SEDCMD-remove_parens_num = s/((\d))/./g
SEDCMD-remove_first_period = s/^(.)//g
SEDCMD-remove_last_period = s/(.)$//g

... I stopped seeing my DNS logs in GUI permanently after the restart of Splunk. I do not understand. Any idea?

Tomas

[gdavid]

each solution had its own trouble for me.

if you use SEDCMD-remove-count = s/((\d+))/./g
you turn a PTR record into this: (.).(.).(.).(.).(.)in-addr(.)arpa(.)

also if read it correct SEDCMD happens at index time so your logs are lost if there are errors here. to have this modify the entire string like this is less than ideal.

[gdavid]

each solution had its own trouble for me.

if you use SEDCMD-remove-count = s/((\d+))/./g
you turn a PTR record into this: (.).(.).(.).(.).(.)in-addr(.)arpa(.)

also if read it correct SEDCMD happens at index time so your logs are lost if there are errors here. to have this modify the entire string like this is less than ideal.

[gdavid]

thanks for the fix here. this should be added standard to the app. i don't know why its logged this way and not sure why splunk app wouldn't normalize the data field.

[gdavid]

there are a lot of issues using the SEDCMD command to try to fix this. it applies to the whole string messing up other data in the DNS log.

i found issues with all the solutions suggested here, messing up PTR records or leaving a trailing . or leading (##) in the logs.

still searching for the fix here.

[bengoerz]

Did you ever find a fix you liked?

I'm curious what problems you saw in PTR records? To me, looks like they have the same weird parenthetical-count formatting problem as A records, so this fix would be common to both.

[mikaelbje]

Instead of recreating this by yourself, I believe the following add-on already does what you're trying to achieve: https://splunkbase.splunk.com/app/3377/

It's even CIM compliant, meaning the fields are normalized.

[tomasmoser]

Well, it did help but I am not really happy. I ran into three problems:

1. Why "SEDCMD-remove1 = s/((\d))/./g" and not "SEDCMD-remove1 = s/((\d))/./g"? I am not getting logic. Seems it works the same.

2. Once I modified props.conf with SEDCMD above, all of a sudden I am not extracting any other fields during search time (as defined in default/props.conf) - ALL OTHER FIELDS VANISHED. I am getting just host, source, sourcetype.

3. adding "SEDCMD-remove-head-dot = s/\s(.)//g" into props.conf does not do anything (was working with rex in search bar)

4. adding "SEDCMD-remove-tail-dot = s/(.)$//g" into props.conf does not do anything (was warking with rex in search bar)

I do not understand Splunk's logic. Simply not.

[tomasmoser]

I can confirm now that skoelpin's solution works!

My props.conf

[MSAD:NT6:DNS]

Replace (3)www(6)google(3)com with www.google.com etc.

SEDCMD-remove-count = s/((\d+))/./g
SEDCMD-remove-head-dot = s/\s(.)//g
SEDCMD-remove-tail-dot = s/(.)$//g

[skoelpin · Accepted solution]

Try testing this at search time before modifying props.conf..

Try this

index=whatever | rex mode=sed s/(\(\d)\)/./g

[ssubhani]

I saw this solution today after almost 4 years and it works for me too . However I do get a trailing dot OR a number before each .

Original

7/26/2021 4:14:02 PM 1498 PACKET 0000020027AD4070 UDP Rcv 10.38.2.63 e92f Q [0001 D NOERROR] A (5)ctldl(13)windowsupdate(3)com(0)

7/26/2021 4:14:02 PM 1498 PACKET 000002002D51DCC0 UDP Rcv 10.34.23.50 7494 Q [0001 D NOERROR] A (5)ctldl(13)windowsupdate(3)com(0)

7/26/2021 4:14:02 PM 1498 PACKET 0000020024F70D10 UDP Rcv 10.38.5.167 30e6 Q [0001 D NOERROR] A (4)pypi(3)org(0)

After applying Sedcmd ,notice the dot

7/26/2021 4:14:02 PM 1498 PACKET 0000020027AD4070 UDP Rcv 10.38.2.63 e92f Q [0001 D NOERROR] A .ctldl(13)windowsupdate.com.

7/26/2021 4:14:02 PM 1498 PACKET 000002002D51DCC0 UDP Rcv 10.34.23.50 7494 Q [0001 D NOERROR] A .ctldl(13)windowsupdate.com.

7/26/2021 4:14:02 PM 1498 PACKET 0000020024F70D10 UDP Rcv 10.38.5.167 30e6 Q [0001 D NOERROR] A .pypi.org.

[tomasmoser]

Yes, it does work correctly in this first stage.

1. 2. 2017 23:07:08 0D7C PACKET 0000002549C0C0A0 UDP Snd 10.18.1.51 b1aa Q [0000 NOERROR] SRV ._ldap._tcp(23)Default-First-Site-Name._sites.dc._msdcs.develop3.develop2.develop.local.

[skoelpin]

So you verified its working correctly at search time.. Do you want me to give you the SEDCMD so you can add it to your props.conf for index time now?

[tomasmoser]

Yes. Please.

[skoelpin]

Place this in your props.conf under $SPLUNK_HOME\etc\apps\#APP_NAME\local

[YourSourcetype]
SEDCMD-remove_parens = s/(\(\d)\)/./g

Don't forget to restart the Splunk service after making this change.

Lastly, if this works for you then please accept the answer