Splunk Search

Missing data ?

BWHarris
Explorer

I am using the free license trying out Splunk. I've noticed that some data(events) are missing when I do a search. I have installed universal forwarders on 3 hosts and added monitors accordingly. All 3 hosts have the letters "PDM" in the host name. I do a search for host=PDM and I get some events from the logs, but not all. I'm using the default UDP port 9997 for the receiver. Any ideas ?

Tags (2)

BWHarris
Explorer

I'm going to try to explain this problem with a video. Watch as I list the monitors on a windows host and you'll see the file I want to search is listed. After that I open the log file and look at the time of the last event. Then I go over to the splunk web portal and do a search. I even click on the source directly from the summary page and the event is still not listed. You can see the last event in the search results is not what I saw from the real Windows event log. After that, I restart the universal forwarder then go back to the web portal, click the search a couple more times, and it is there. So it only seems to pick up the data after I do a restart of the forwarder...

http://screencast.com/t/ILPGo40B

0 Karma

BWHarris
Explorer

Does anyone have any ideas ? It seems when I restart the forwarder, then the events get updated in the search, but only when I do a restart. Obviously I am not going to restart the forwarder every time I want to see new events. I'm going to have to fail this application as something our team can use, unless this fundamental ability to search on hostname will work. I don't have the time to research errors on something that should work out of the box.

0 Karma

BWHarris
Explorer

Sorry, I meant to say I am using host=PDM. (Edit: This text box won't let you use asterisks so what I typed the first time was right). I just deleted the UDP data input and cleaned all event and global data. I configured the receiver to listen on tcp 9998. It's still not reporting all events for the file I am monitoring. I added the file using the CLI...splunk add monitor ""

0 Karma

BobM
Builder

For a start you should be using splunktcp and not udp for the forwarders to send to. If you configured it via the UI it is dificult to get wrong so assume that was a mistake.

The problem looks to me that you need to use a wildcard in your search. Splunk is not case sensitive (in most places) but it expects complete words or values so add a star before, after or both and it should work. Try the following

host=*pdm*
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...