Why am I getting error "Ran out of data while looking for end of header" configuring the Splunk Add-on for Bro IDS?

[adamblock2]

We are currently running version 2.4 of Bro, and I have been having difficulty properly configuring the Bro add-on.

According to the documentation, versions 2.1 and 2.2 are supported. I am curious if this could be part of our problem.

The following bro logs are currently being written to "/syslog_hot/splunk/bro" on our syslog server.

-rw-------. 1 root root   3621901 Mar  1 12:36 notice.log
-rw-------. 1 root root    140147 Mar  1 12:30 other.log
-rw-------. 1 root root 654588548 Mar  1 12:36 ssl.log

I added the following to inputs.conf file:

[monitor://syslog_hot/splunk/bro]
    index=bro
    sourcetype=bro
    blacklist = \.(gz*|\d+|txt)$

When I added the "sourcetype=bro" statement to the inputs.conf file as per the documentation, I started receiving the following error:

03-01-2016 12:19:07.616 -0500 ERROR TailingProcessor - Ran out of data while looking for end of header
03-01-2016 12:19:07.633 -0500 ERROR TailingProcessor - Ran out of data while looking for end of header
03-01-2016 12:19:42.778 -0500 ERROR TailingProcessor - Ran out of data while looking for end of header
03-01-2016 12:19:42.796 -0500 ERROR TailingProcessor - Ran out of data while looking for end of header
03-01-2016 12:20:03.077 -0500 ERROR TailingProcessor - Ran out of data while looking for end of header

As soon as I removed the "sourcetype=bro" statement, logs started being forwarded to the indexers. However, they appeared with the following sourcetypes:
ssl.log - "ssl-3"
notice.log - "notice-2"
other.log - "syslog"

Assistance with this would be appreciated.

Thank you.

[jorritf]

Perhaps it has something to do with other (hidden) files that are also present in the directory you monitor? When I look at my own installations there are various Bro related files like .state, .status .rotated. etc that may get scanned for lines starting with "#fields", so INDEXED_EXTRACTIONS = TSV returns proper header names.

You may try to change the monitor stanza to something like [monitor:///syslog_hot/splunk/bro/*.log], or also blacklist any files in that directory that don't start with "#fields"

[zabbasi_splunk]

Make sure your logs are written as type.log or something.type.log because the Splunk platform uses the second part of the name to more specifically source type the log. For example, conn.log produces the bro_conn sourcetype.

[adamblock2]

I included a listing of the file names in my original question (notice.log, ssl.log, other.log).

[ehaddad_splunk]

I don't think it has to do with the version of Bro - not yet at least.
I see you need to add an extra '/'. Not sure if this causing the issue but very possible. I would try to change things to:
[monitor:///syslog_hot/splunk/bro]

[adamblock2]

I added the additional '/', but I continue receiving the " ERROR TailingProcessor - Ran out of data while looking for end of header" messages.