Getting Data In

Logs Not Being Indexed

cuppma
Explorer

I'm fairly new to Splunk and I can't figure out how to get Splunk to index my logs. I have configured my WebSense device to send logs to Splunk on UDP 6667 and I have configured Splunk to listen for logs on UDP 6667. I did a packet capture to make sure the logs were getting sent to the Splunk server. I have confirmed that they are getting sent to the server, but I cannot search through the logs. I believe the logs are not being properly indexed. Any ideas?

Tags (3)
0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

What is your search that isn't working?

It might be a matter of time. Some logs from devices send in UTC, but if you are in EST, they won't show up! Try adding "latest=+10h@h" to your search and see if that makes your logs show up.

0 Karma

cuppma
Explorer

I'm getting this error when trying to use the Fire Brigade App "Unable to fetch REST endpoint uri="/services/data/indexes?count=0" from server=""." I also am getting a lot of N/A, No results found when trying to use this app. Do I need to do some initial setup with this app?

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

They shouldn't interfere with each other. Check the physical size of the index (The FireBrigade App can help with this). If the physical size isn't changing, you may have a listening problem.

0 Karma

cuppma
Explorer

This may be a stretch, but I know my instance of Splunk listens for traffic from forwarders on TCP 6667. Could this be interfering with the UDP traffic? I know you can use the same port concurrently with both UDP and TCP but would doing this be an issue in Splunk?

0 Karma

cuppma
Explorer

Still no results.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Looks good to me, restart Splunk for it to take effect.

0 Karma

cuppma
Explorer

_internal index returned no results, I tried to set an index via the GUI and got the following error: "Timed out while waiting for splunkd daemon to respond. Splunkd may be hung." So I went into the server and configured /etc/apps/websense/local/indexes.conf with the following:

[websense]
homePath = $SPLUNK_DB\websense\db
maxDataSize = auto_high_volume
thawedPath = $SPLUNK_DB\websense\thaweddb
coldPath = $SPLUNK_DB\websense\colddb

Does that look correct? For some reason the backslashes are getting removed when I click comment, but they are in the config.

0 Karma

mgonter
Engager

You're forgetting directory slashes here.

homePath = $SPLUNK_DB/websensedb.

Unless you have a variable set for $SPLUNK_DBwebsensedb defined. Have you looked in $SPLUNK_HOME/var/lib/splunk to see if the index is there?

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Has the index been defined in indexes.conf or via the GUI? What does the _internal index say?

0 Karma

cuppma
Explorer

Yea, I tried it with and without that colon. Still wasn't working. I should be setting this in /etc/apps/websense/local correct?

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

There's your typo - you have an extra :

Should be: [udp://6667]

0 Karma

cuppma
Explorer

index=main returns a lot of logs from Perfmon:LocalNetwork that look like this:

02/27/2014 10:55:26.775
collection=LocalNetwork
object="Network Interface"
counter="Current Bandwidth"
instance="Broadcom BCM5709C NetXtreme II GigE [NDIS VBD Client] _6"
Value=1000000000

Is it possible that those are the logs I want but the correct information isn't being extracted?

From inputs.conf
[udp://:6667]
sourcetype = websense
index = websense
disabled = false

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Try looking in index=main or index=default. or just do "index=*" and see if they show. Make sure that index=websense is part of your inputs configuration.

0 Karma

cuppma
Explorer

I am doing the most basic search something like "index=websense" just to see if the logs are even being indexed into Splunk and I'm getting no results. I tried adding "latest=+10h@h" that didn't work.

0 Karma

cuppma
Explorer

I set the sourcetype and index in the inputs.conf file. What do I need to edit in the props.conf and transform.conf files?

0 Karma

emaccaferri
Communicator

Did you have set the sourcetype and the index when you configured Splunk?
Did you edited your local copy of props.conf and trasform.conf?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...