Getting Data In

Why is host=myhost giving no results?

iiooiiooiioo
Explorer

Sorry for the complete noob question. But I have had this splunk project dropped on me and I need to spin up fast.

I have added a monitor on "myhost" like so:

[root@myhost bin]# pwd
/apps/splunkforwarder/bin
[root@myhost bin]# ./splunk add monitor /var/log/foo/
Your session is invalid.  Please login.
Splunk username: admin
Password:
Added monitor of '/var/log/foo'.

That was yesterday.

I executed a script that writes data to a log file that is in the /var/log/foo directory on myhost.
But when I execute this search host=myhost I get zero results.

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Splunk configures index = default when you add new monitor. And default index is not created in indexer servers. So you need to specify index and sourcetype for your monitor. Edit /apps/splunkforwarder/etc/apps/search/local/inputs.conf and add index and sourcetype like below. Restart splunk forwarder and check data in index=main sourcetype=foo.

[monitor:///var/log/foo/]
index = main
sourcetype = foo

As @woodcock suggested. Instead of updating splunk internal search app it is better to put inputs.conf in your own add-on and deploy it. Move /apps/splunkforwarder/etc/apps/search/local/inputs.conf file to /apps/splunkforwarder/etc/apps/fwd-2-dev-indexers/default/ and restart splunk forwarder.

iiooiiooiioo
Explorer

Here is an update to my original post. Here are the locations of the inputs.conf and outputs.conf file I have on "myhost":

[root@myhost splunkforwarder]# pwd
/apps/splunkforwarder
[root@myhost splunkforwarder]# find . -name inputs.conf
./etc/system/default/inputs.conf
./etc/system/local/inputs.conf
./etc/apps/search/local/inputs.conf
./etc/apps/SplunkUniversalForwarder/default/inputs.conf
./etc/apps/introspection_generator_addon/default/inputs.conf
./etc/apps/splunk_httpinput/default/inputs.conf
[root@myhost splunkforwarder]# find . -name outputs.conf
./etc/system/default/outputs.conf
./etc/apps/SplunkUniversalForwarder/default/outputs.conf
./etc/apps/fwd-2-dev-indexers/default/outputs.conf
0 Karma

woodcock
Esteemed Legend

Never use the CLI for this. Create a $SPLUNK_HOME/etc/varlog_inputs/default/inputs.conf file like this:
https://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories

iiooiiooiioo
Explorer

Thanks for the reply. But I do not seem to have the varlog_input directory on my server:

[root@myhost etc]# pwd
/apps/splunkforwarder/etc
[root@myhost etc]# ls -l | grep varlog
[root@myhost etc]#
[root@myhost etc]# env | grep -i SPLUNK_HOME
[root@myhost etc]#
0 Karma

woodcock
Esteemed Legend

Yes, you will be creating a custom app called anything that you like. I made up varlog_inputs.

0 Karma

mguhad
Communicator

Hi,
Since you didnt specify an index, splunk will, by default, place your data in the 'main' index. The server would check in under its hostname/ip address so you could this in your host= parameter.

So you could try index=main host=<myhost> or <ipdress>

If you want to find out the hsotname of the forwarder: ./splunk show default-hostname
then pass this hostname in your search: index=main host=<output_of_above_command>

Best practice is to simply create an inputs.conf file either under /system/local or /etc/apps//local and monitor files that way assuming you have configured the outputs.conf to send data to the indexers (unless standalone-all-in-one box).

iiooiiooiioo
Explorer

Thanks for the reply!

Here's what I tried:

[root@myhost bin]# ./splunk show default-hostname
Default hostname for data inputs: myhost.

Then I tried this search:

index=main host=myhost

But I still got no results.

0 Karma

mguhad
Communicator

@iiooiiooiioo cheeck if your forwarder (myhost) are actually sending data at all to the _internal index.

index=_internal host=myhost

ALternatively, check to see if the main index has ANY data :
| eventcount summarize=false index=* OR index=_*

0 Karma

holowolf3500
Loves-to-Learn

Im having the same issue as ownerpost I tried your index=_internal host=___ I typed in my agent1 , agent2 and agent3 along with controller each tiime and data popped up for each 4 of them. but when I type in the command Index=”main” host=* | table host | dedup host it does not show anything at all?

Can you help me troubleshoot this 😞

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...