Missing Source data from Forwarder

[thomas_bengtsen]

Hi Group,

I am new to the Splunk thing so bear with me.
I have installed an indexer, configured it to look at some local log files and that seems to work ok. I have also installed a forwarder on another machine and configure it to monitor a file and connect to the Indexer. As far as I can tell the file is being monitored and the data is sent to the Indexer and being indexed – at least I can see the index having count and size_bytes if I look under “Status –> Index activity -> Index activity overview”.

The problem is that if I look on the search page I can only see one source – namely the local file. My searches do not show any entries form the file on the indexer. Additionally – and this I find very strange – if under “Status –> Index activity -> Index activity overview” I drill down into the index for the remote server it shows me the entries from the local file on the indexer.
(Splunk 4.2 on Solaris 10 X86)

Thanks

[ehoward]

I am also have a problem with Indexing Volume stats. I am logged in as Admin with full permissions and role access to everything, yet when run Status, Indexing Volume I only see data for is for _thefishbucket and _internal. I have a lot of Windows WMI event data being forwarded to the Indexer the indexing stats are not showing when when I display the Indexing Volume by Source/Sourcetype.

[kristian_kolb · Accepted solution]

Well, there may be the simplest of possible explanations.

Have you enabled the correct indexes to be searched by default in Manager > Access Controls > Roles > .

If you're really new to this, you are probably logging in as 'admin', and you should check if your newly created indexes for app1 and app2 are enabled for default searching. (almost at the bottom of the page.)

Unless an index is selected for default searching, your searches will not return any events from it unless you include index=<your_index> as part of your search.

hope this helps,

/Kristian

And no, you do not need to define the index in indexes.conf on the forwarder

[thomas_bengtsen]

Yes - I am really that new to this 🙂

Thanks for your help

[alacercogitatus]

The first thing I would to is remove [splunktcp://9997] from the forwarder inputs.conf.
The next thing I would to is add the [app2] index stanza to the system/local/indexes.conf on the forwarder. Then the forwarder knows it exists as a valid index, and can forward it. I use something similar on one of my systems and I have the index declared on both systems, and it works fine. It may or may not use that index on the forwarder, it depends on the type of forwarder.

[alacercogitatus]

Is splunk indexer listening on the port? netstat -an | grep LISTEN should return *.9997

[thomas_bengtsen]

Thanks for the suggestion but still no change.

I think it it something simpler - remember I am new to this.

When I check the Summery screen the only Source that show up is the source I added via the GUI on the Indexer itself - app1.

There are other indexes apart from app1 and app2 from splunks own files and they don't show up either. It seems to me just a question of adding this data which already exists in indexes to the searchable view.

Thanks

[thomas_bengtsen]

Formatting seems screwed up – sorry about that.
Thanks

[thomas_bengtsen]

Indexer:

No outputs.conf

inputs.conf
[splunktcp://9997]

[monitor:///com/logs/grp/app1.log]
disabled = false
followTail = 0
index = app1

indexes.conf
[app2]
coldPath = $SPLUNK_DB/app2/colddb
homePath = $SPLUNK_DB/app2/db
thawedPath = $SPLUNK_DB/app2/thaweddb
disabled = 0

[app1]
coldPath = $SPLUNK_DB/app1/colddb
homePath = $SPLUNK_DB/app1/db
thawedPath = $SPLUNK_DB/app1/thaweddb

[thomas_bengtsen]

Forwarder:

outputs.conf
[tcpout]
defaultGroup = indexer1
heartbeatFrequency=10

Defect 396475

maxQueueSize=10000

[tcpout:indexer1]
server=192.168.53.6:9997

inputs.conf
host = TEST01A
[splunktcp://9997]
[monitor:///com/logs/grp/app2.log]
disabled = false
followTail = 0
index = app2

splunk list monitor -auth admin:*****

Monitored Directories:
$SPLUNK_HOME/var/log/splunk
...lines removed
Monitored Files:
$SPLUNK_HOME/etc/splunk.version
$SPLUNK_HOME/var/log/splunk/splunkd.log
$SPLUNK_HOME/var/spool/splunk
/com/logs/grp/app2.log

[kristian_kolb]

If you could share a little more informatiion, it would be easier to help you.

What do the inputs.conf and outputs.conf look like on the forwarder.

What do the inputs.conf and indexes.conf look like on the indexer.

/kristian