Problem forwarding Advanced IIS Logs

[DaClyde]

I am experiencing an issue where my universal forwarder (v5.0.4) is not forwarding my IIS Advanced Logs to the indexer. Here is the stanza from my inputs.conf

[monitor://F:\inetpub\logs\LogFiles\W3SVC1\]

disabled = false

whitelist = iis_D(\d+)-(\d+).log

sourcetype = adviis

index = adviis

I can tell it sees the logs because I get these entries in my metrics.log:

11-05-2013 14:28:01.726 -0600 INFO Metrics - group=per_index_thruput, series="adviis", kbps=0.012349, eps=0.161290, kb=0.382813, ev=5, avg_age=0.600000, max_age=3

11-05-2013 14:28:01.726 -0600 INFO Metrics - group=per_source_thruput, series="f:\inetpub\logs\logfiles\w3svc1\iis_d20131105-202617637.log", kbps=0.012349, eps=0.161290, kb=0.382813, ev=5, avg_age=0.600000, max_age=3

11-05-2013 14:28:01.726 -0600 INFO Metrics - group=per_sourcetype_thruput, series="adviis", kbps=0.012349, eps=0.161290, kb=0.382813, ev=5, avg_age=0.600000, max_age=3

And my splunkd.log shows it watching the folder:

11-05-2013 13:38:04.363 -0600 INFO TailingProcessor - Adding watch on path: F:\inetpub\logs\LogFiles\W3SVC1.

So why is nothing showing up in my index? I can forward WinEventLogs and standard IIS logs with no issue between these two machines. I even manually imported one of these logs into the indexer just to make sure the "adviis" index and sourcetypes existed (I know that shouldn't be necessary).

I've cleared the fishbucket multiple times, but these files just won't budge.

UPDATE

Shane's answer accounted for everything but my ineptitude and ignorance about how the search head relates to an indexer. From the Manager in the web interface, if you create an index, it is going to create the index on the machine that web UI represents. In my case, I was just creating the adviis index on the search head. No amount of forwarding to the indexer is going to find that index.

To get everything lined up properly, I had to delete the adviis index from the search head and delete the corresponding adviis index folder from /Splunk/var/lib/splunk/. Then I deleted the assorted attempted copies of the same folder and indexes.conf files that had been generated. Then (after finally remembering the login for the indexer), I logged into the web UI on the indexer and created the adviis index through the Manager that way. Now all the bits and pieces were in place, I flushed the fishbucket on my forwarder and data started moving.

So all of Shane's advice was correct, if I had created the index properly in the first place. Thank you Shane!

[ShaneNewman]

Anytime! Glad I could help.

[yong_ly]

I agree with Shane, make sure you don't have conflicting stanzas in inputs/props/transforms..

Another thing you can check the timestamp recognition, I've had something similiar in the past where I thought the logs weren't being indexed only to discover that it had been put into the wrong index or under the wrong sourcetype or the timestamp had read it wrong so it was sitting there but marked as a year ago..

You can check by just searching for the source=inetpub\logs\LogFiles\W3SVC1 over ALL TIME.. that should pick up any instances of the files that have been indexed..

[DaClyde]

The timestamps look good on the file I manually imported. Searching all time, and even adding a latest=+10d doesn't find any stray data outside the one manual import.

[kristian_kolb]

also, you can verify that the timestamps are parsed correctly.

| dbinspect index=adviis

Check that the earliestTime and latestTime timestamps match your data.

[lukejadamec]

You can also check the adviis index in Manager>Indexes to see if it is getting data.

[ShaneNewman]

If you are monitoring that exact path twice, one entry will be ignored. You will have to bring them both in with a single monitor stanza, then use props.conf and transforms.conf to distinguish sourcetypes at index time.

props.conf

[iis_log]
NO_BINARY_CHECK = 1
#TRANSFORMS-0_define_sourcetype = iis_sourcetype_transform

transforms.conf

[iis_sourcetype_transform]
SOURCE_KEY = MetaData:Source
REGEX = iis_D(d+)-(d+).log
DEST_KEY = MetaData:Sourcetype
FORMAT =  sourcetype::adviis

Does that help?

[ShaneNewman]

send me an email, you can get my contact info by clicking on my username.

[DaClyde]

Sure, that would be great.

[ShaneNewman]

Do you have time for a quick webex?

[DaClyde]

That is correct.

[ShaneNewman]

Just to clarify, SPLUNK402 is your indexer, correct?

[DaClyde]

It is enabled. I even deleted the index, re-created it, restarted the indexer, flushed the fishbucket on the forwarder and restarted that, but I'm still getting the same error. If I generate traffic on the forwarder, I can see new entries pop up in the metrics.log on the forwarder, so it's trying to work.

[ShaneNewman]

This means that the index has been disabled. When you go through the Manager console and select indexes, does it say the index is disabled? If so, enable it.

[DaClyde]

To append to that, the adviis index does exist because I can see yesteday's log (that I manually imported through the Manager>Data Inputs page) if I do a "search index=adviis sourcetype=adviis" for the last 24 hours.

[DaClyde]

For now, my inputs.conf on the forwarder only has the monitor one stanza.

I've added your recommended stanzas to my props and transforms on the indexer, but so far, no luck. Now I'm getting this message on the search head:

Search peer SPLUNK402 has the following message: received event for unconfigured/disabled/deleted index='adviis' with source='source::F:\inetpub\logs\LogFiles\W3SVC1\iis_D20131105-001238183.log' host='host::Weeble' sourcetype='sourcetype::adviis' (1 missing total)