I have a simple setup: a Universal Forwarder monitoring a couple of files and sending the cooked data over TCP port 8080 to an indexer which is also used for search. The UF is running on Ubuntu; the indexer on Windows Server 2008. Both are version 4.3.1.
I had to make some changes to sourcetype definitions and also wanted things to go in a special index, and the data is all intact on the UF, so I decided to delete all the data on the indexer:
splunk.exe clean eventdata main
And from the Forwarder:
splunk clean all
I then switched off the forwarder and did my new setup. inputs.conf :
[monitor:///home/ubuntu/data_collector/queries1.log]
index=reporting
[monitor:///home/ubuntu/data_collector/queries2.log]
index=reporting
And outputs.conf :
[tcpout]
defaultGroup = 210.XXX.XXX.44_8080
[tcpout:210.XXX.XXX.44_8080]
server = 210.XXX.XXX.44:8080
[tcpout-server://210.XXX.XXX.44:8080]
I've created the reporting app on the indexer.
I even deleted and re-added the forward-server on the UF for good measure.
I turned the UF back on. The data from splunkd.log on the UF is showing up in the indexer, but I don't get any data at all in my reporting index.
I tried it without the index line in inputs.conf - in the hope that I could get the data showing up in main again - but no such luck.
Here's a sample of splunkd.log from the indexer:
04-17-2012 19:43:53.281 +1000 INFO databasePartitionPolicy - creating new bucket C:\Program Files\Splunk\var\lib\splunk\audit\db\hot_v1_9
04-17-2012 19:43:53.281 +1000 INFO databasePartitionPolicy - lazy loading database for: C:\Program Files\Splunk\var\lib\splunk\audit\db\hot_v1_9, id=9, ts=1334655831 dirMgr::nextId=9]
04-17-2012 19:43:53.281 +1000 INFO HotDBManager - index=_audit Creating new hot (id=9, time=1334655831)
04-17-2012 19:43:53.281 +1000 INFO loader - Server supporting SSL v2/v3
04-17-2012 19:43:53.281 +1000 INFO loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
04-17-2012 19:43:53.531 +1000 INFO ProcessTracker - (child_0__Fsck) Fsck - Rebuild --bloom-only bucket C:\Program Files\Splunk\var\lib\splunk\audit\db\db_1334655799_1334649015_8 took 203.1 milliseconds
04-17-2012 19:43:53.687 +1000 INFO TailingProcessor - TailWatcher initializing...
04-17-2012 19:43:53.687 +1000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk.
04-17-2012 19:43:53.687 +1000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk\...stash_new.
04-17-2012 19:43:53.687 +1000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\etc\splunk.version.
04-17-2012 19:43:53.687 +1000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk.
04-17-2012 19:43:53.687 +1000 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
04-17-2012 19:43:53.687 +1000 INFO WatchedFile - Will begin reading at offset=2366820 for file='C:\Program Files\Splunk\var\log\splunk\audit.log'.
04-17-2012 19:43:54.156 +1000 INFO WatchedFile - Will begin reading at offset=17887152 for file='C:\Program Files\Splunk\var\log\splunk\metrics.log'.
04-17-2012 19:43:54.297 +1000 INFO HotDBManager - index=_internal No hot found for event ts=1334655831, closest match=null [expanded span=0] hotbucketsize=0 numbucks=1 maxhot=3
04-17-2012 19:43:54.297 +1000 INFO databasePartitionPolicy - creating new bucket C:\Program Files\Splunk\var\lib\splunk\_internaldb\db\hot_v1_14
04-17-2012 19:43:54.297 +1000 INFO databasePartitionPolicy - lazy loading database for: C:\Program Files\Splunk\var\lib\splunk\_internaldb\db\hot_v1_14, id=14, ts=1334655831 dirMgr::nextId=14]
04-17-2012 19:43:54.297 +1000 INFO HotDBManager - index=_internal Creating new hot (id=14, time=1334655831)
04-17-2012 19:43:56.484 +1000 INFO ExecProcessor - Ran script: "C:\Program Files\Splunk\bin\splunk-admon.exe", took 2296.9 milliseconds to run, 0 bytes read
04-17-2012 19:43:56.640 +1000 INFO WatchedFile - Will begin reading at offset=665463 for file='C:\Program Files\Splunk\var\log\splunk\web_service.log'.
04-17-2012 19:43:57.593 +1000 INFO ExecProcessor - Ran script: "C:\Program Files\Splunk\bin\splunk-perfmon.exe", took 890.6 milliseconds to run, 0 bytes read, exited with code -1
04-17-2012 19:43:59.875 +1000 INFO ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-regmon.exe" --driver-path "C:\Program Files\Splunk\bin"" splunk-regmon - SysmonMigrator::read - 'sysmon.conf' was not found, no migration is required.
04-17-2012 19:43:59.875 +1000 INFO ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-regmon.exe" --driver-path "C:\Program Files\Splunk\bin"" splunk-regmon - No enabled entries have been found for regmon or procmon in the conf file.
04-17-2012 19:44:00.140 +1000 INFO ExecProcessor - Ran script: "C:\Program Files\Splunk\bin\splunk-regmon.exe" --driver-path "C:\Program Files\Splunk\bin", took 921.9 milliseconds to run, 0 bytes read
04-17-2012 19:44:02.625 +1000 INFO ExecProcessor - Ran script: "C:\Program Files\Splunk\bin\splunk-wmi.exe", took 890.6 milliseconds to run, 0 bytes read
04-17-2012 19:44:10.687 +1000 INFO ProcessTracker - (child_1__Fsck) Fsck - Rebuild --bloom-only bucket C:\Program Files\Splunk\var\lib\splunk\_internaldb\db\db_1334655799_1330647338_13 took 1093.8 milliseconds
04-17-2012 19:44:18.359 +1000 INFO databasePartitionPolicy - rebuildMetadata called: full=true path=C:\Program Files\Splunk\var\lib\splunk\audit\db reason= repaired_buckets
04-17-2012 19:44:19.406 +1000 INFO databasePartitionPolicy - rebuildMetadata called: full=true path=C:\Program Files\Splunk\var\lib\splunk\_internaldb\db reason= repaired_buckets
04-17-2012 19:48:32.594 +1000 WARN PipelineInputChannel - channel "source::/home/ubuntu/data_collector/queries1.log|host::ip-10-166-206-183|web|remoteport::44680" ended without a done-key
04-17-2012 19:55:30.063 +1000 WARN ProcessRunner - Process with pid 2604 did not exit within a given grace period after being signaled to exit. Will have to forcibly terminate.
04-17-2012 20:06:46.858 +1000 WARN PipelineInputChannel - channel "source::/home/ubuntu/data_collector/queries1.log|host::ip-10-166-206-183|web|remoteport::44996" ended without a done-key
04-17-2012 20:07:38.717 +1000 ERROR AuthenticationManagerSplunk - Login failed. Incorrect login for user: admin
04-17-2012 20:09:41.108 +1000 WARN PipelineInputChannel - channel "source::/home/ubuntu/data_collector/queries1.log|host::ip-10-166-206-183|web|remoteport::45229" ended without a done-key
04-17-2012 20:31:57.961 +1000 WARN PipelineInputChannel - channel "source::/home/ubuntu/data_collector/queries1.log|host::ip-10-166-206-183|web|remoteport::45511" ended without a done-key
04-17-2012 21:13:58.781 +1000 WARN PipelineInputChannel - channel "source::/home/ubuntu/data_collector/queries1.log|host::ip-10-166-206-183|web|remoteport::45809" ended without a done-key
Could the 'ended without a done key' errors be the culprit here?
... View more