I'm receiving duplicate events from IIS logs being sent through the universal forwarder.
The forwardeds 'splunkd.log' is showing:
10-24-2013 14:45:02.882 +1100 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\path\to\iis\logs\u_ex131024.log'.
10-24-2013 14:45:02.882 +1100 INFO WatchedFile - Will begin reading at offset=0 for file='C:\path\to\iis\logs\u_ex131024.log'.
10-24-2013 14:45:02.882 +1100 INFO WatchedFile - Resetting fd to re-extract header.
Splunk versions are:
inputs.conf
[monitor://C:\path\to\iis\logs\*.log]
disabled = false
sourcetype = iis
props.conf (as per universal forwarder defaults)
[iis]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
INDEXED_EXTRACTIONS = w3c
detect_trailing_nulls = auto
Any ideas where I am going wrong?
This is a known issue with 6.0, SPL-77048. It is tentatively scheduled to be fixed in the forthcoming maintenance release, which will be post 6.0.1.
This is a known issue with 6.0, SPL-77048. It is tentatively scheduled to be fixed in the forthcoming maintenance release, which will be post 6.0.1.
On Indexer,.
Create or edit " $SPLUNK_HOME\etc\system\local\props.conf"
[iis]
TZ = GMT
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True
REPORT - iis2 = iis2
Add more stanzas if nessesary (sample)
[u_ex-too_small]
rename = iis
TZ = GMT
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True
REPORT - iis2 = iis2
[u_ex-2]
rename = iis
TZ = GMT
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = True
REPORT - iis2 = iis2
Create or edit " $SPLUNK_HOME\etc\system\local\transforms.conf"
[iis2]
DELIMS = " "
FIELDS = date, time(GMT), s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs(User-Agent), sc-status, sc-substatus, sc-win32-status, time-taken
I think this is default fields from IIS, add or remove if more or less fields are chosen.
Restart splunkd service
Just one note - I added these to the two files you mentioned above, so that the IIS log comments get removed from the results:
To each stanza in the props.conf:
TRANSFORMS-removecomments = removecomments
To the transforms.conf:
[removecomments]
REGEX = ^#.*
DEST_KEY = queue
FORMAT = nullQueue
Thanks again!
Excellent, thank you! It works perfectly. Hopefully Splunk fixes this in the next release...
Sorry for the delayed comment - the automated SplunkBase email went to my Junk folder and I just saw it...
We had the same problem with our IIS logs.
Think I have tried anything with UF version 6.0-82037 & 6.0-82611, upgrades and fresh install with different configurations (input.conf).
Uninstalled UF version 6 and reinstalled version 5.0.5-179365.
So far it has been stable, and no checksum error.
Splunk 6.0.182037 (indexer and heavy forwarder) &
Splunk Universal Forwarder 5.0.5-179365(again)
Hi mParticle. You will find my answer below. Couldn’t comment it here, too many characters…..
Thanks arvidn! I tried this and so far the UF doesn't seem to get thrown in a loop, however the indexer doesn't parse the logs properly/automatically as it did with the 6.0 UF, so I am guessing some transforms are in order. Would you mind sharing what other conf file changes you have made on the UF/Indexer side to get this to work?
+1... Splunk indexer and UF both on 6.0.182037
inputs.conf
[monitor://C:\inetpub\logs\LogFiles\W3SVC1]
sourcetype=iis
index=iis_logs
props config
[iis]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
INDEXED_EXTRACTIONS = w3c
detect_trailing_nulls = auto
I also tried adding
initCrcLength = 1024
crcSalt = <SOURCE>
(crcSalt first by itself, then together with initCrcLength), neither is helping.
Splunk guys, any suggestions? Anyone?
Glad to know someone else is facing the same issue