Forwarder Data not showing in indexes

[auragrp]

Recently I upgraded our Splunk installation from the 5 version to the new 6.0 version.

The installation is pretty vanilla and has a single Splunk sever with one a secondary server using the Universal Forwarder to send IIS logs to the Splunk server.

I noticed to my dismay that after the upgrade the new 6.0 version did not migrate my receiver port setting for the Forwarder to sent to. ie: port 9997

Looking at the data listed in the indexes the last update listed is 12/3/13 on the Splunk server for the data source that is associated with the Forwarder.

I have checked the Time, TimeZone and Date on both the Splunk and Forwarder server and they are the same. I also checked the IIS logs that are being sent to verify time and date and what is listed is prior to the current time.

I verified that the Forwarder configuration is reading the IIS logs and is configured to send to the Splunk server on the correct port. I also made sure that the port is open on the firewall as it was on the 5.0 install.

If I check the Deployment Monitor app > All Forwarders, I see the Forwarder server listed and the last Data Received as of 30 seconds ago.

But when searching All Time - Real-time I get: No results in current time range.

So it looks to me like the Forwarder is sending and the Splunk server via the Deployment monitor says that it is receiving. But instead of adding the data to the indexes, the data looks like it is getting dumped.

I will have to manually import the logs that have not made into the indexes eventually when I get this working. But in the mean time anyone have any thoughts or ideas of how to fix this?

Thanks in advance

[lguinn2]

I don't see anything about the IIS logs in the inputs.conf so perhaps you have more than one inputs.conf under $SPLUNK_HOME\etc (which would be pretty common). If you don't have any other inputs.conf on your forwarder, then it is clear that the input you want is simply not specified...

Also the input for the script is actually enabled, so perhaps you want to turn it off:

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 1

That may solve your problems with the Windows System log information.

Finally, did you install any apps or add-ons on the forwarder?

[lguinn2]

Sorry if this is a dumb question, but are you sure that you are searching all the indexes? One of the things that might have changed when you installed Splunk 6 (depending on how you did it) -> the default indexes searched by a role. I would check the role. Also, try this search

index=* host=theHostName

[auragrp]

Inputs.conf
[default]
host = MC-TRACKING

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

Outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = domainname.com:9997

[tcpout-server://domainname.com:9997]

The only thing that I see that is being sent from the server with the Forwarder is Windows System log information. Which when I setup the forwarder I told it not to send. The only thing it should be sending is the log files I told it to send was the IIS logs specific for one site.

[auragrp]

Sorry for the mulitiple posts but this site wont let me post anything but a comment.

Yes I upgraded the Forwarder software to be the same version as the Splunk install. Current version of the both the Indexer and the Forwarder are: 6.0.1

At first I did an upgrade and most recently I uninstalled the Forwarder and made sure the config files were deleted. Then re-installed and reconfigured the Forwarder to send to the Splunk install.
Below is what is contained in the inputs.conf and outputs.conf files on the Forwarder. (C:\Program Files\SplunkUniversalForwarder\etc\system\local)

[rsennett_splunk]

What it sounds like is that you might have had your configurations in .../default/ directories... and that they were overwritten. Did you upgrade the forwarder as well as the indexer? you haven't mentioned what the actual settings are in the config files. Of interest would be inputs.conf and outputs.conf on the forwarder and inputs.conf on the indexer. you also mentioned that the upgrade didn't preserve the receiving port on the indexer. Do you mean that it seemed to have wiped it out on both the forwarder and indexer or just on one end? but it does sound like lost configurations.

[auragrp]

yes you are correct that the only data is internal. so the log files that should be sent are not being sent.
I just tried uninstalling the forwarder, delete the config, reboot and install from scratch.
When I install I specifically only select the IIS log directory as what will be sent. When I search with: host="MC-TRACKING" now the data that I am seeing is WinEventLog:System events that I did not tell the Forward to send.
Still no IIS logs being sent.

[lguinn2]

It is possible that the data being sent is just the forwarder's Splunk internal logs.

Try this search and see what you get:

index=_internal MC-TRACKING

or replace MC-TRACKING with the name of one of the sources you want to monitor.

Also, take a look at this article on the wiki -

http://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

[auragrp]

below what is listed in the Data Summary popup inside the Search app:
Host: MC-TRACKING
Count: 2,134,522
Last Update: 12/3/13 6:17:13:00 PM

Below is information from Deployment monitor:
Forwarder: MC-TRACKING
Splunk Version: 6.0.1
Forwarder Type: universal forwarder
Platform: Windows
Last Connected: 12/31/13 17:30:00 PM
Last Data Received: 12/31/13 17:33:51 PM
Current Status: active
Total KB: 9.6190
Average Events Per Second: 0.3232

So I am at a loss of why the data is being seen and received by the Deployment monitor but is not being added to the indexes so that it can be viewed.

[auragrp]

thanks for the response, but unfortunately I have checked that and the only data that is showing is the historic data from before the upgrade.

I have tried it both ways ie:
1. index=* host=theHostName
2. index=*
(the post is removing the asterisk but it is there.)

Both only show latest data from just before the upgrade.