add new data source via UDP to indexer

[glenngermiathen]

I have a Splunk indexer running on Ubuntu that forwards to the Splunk web on a Windows box and I want to add a new data source to the Ubuntu system. How do I go about doing this from the command line? I am fairly new to both Splunk and Ubuntu, so the more detailed answer the better.

[jtrucks]

As shown in the manual section on getting data into Splunk via CLI, the most basic method is simply to run:

$SPLUNK_HOME/bin/splunk add monitor /path/to/log/directory

OR

$SPLUNK_HOME/bin/splunk add monitor /path/to/log/filename

Run this as the user Splunk runs as on the system.

If you want to specify the sourcetype and/or index, then it would look like:

$SPLUNK_HOME/bin/splunk add monitor /path/to/log/filename -sourcetype mysourcetypehere -index myotherindexhere

If you use a different index, it should already exist on the indexer. You can omit either or both of the -sourcetype or -index flags above.

Run $SPLUNK_HOME/bin/splunk help monitor to see more complete details.

[jtrucks]

Also, check this out to understand forwarders: http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Aboutforwardingandreceivingdata

And to understand ingesting data: http://docs.splunk.com/Documentation/Splunk/latest/Data/WhatSplunkcanmonitor

[jtrucks]

Further reading for you: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Howtousethismanual

[jtrucks]

I think the issue you posted on here is answered, which is to get your firewall data into Splunk.

Perhaps go read the above, maybe experiment, and then create a NEW answers post with the extract question to get better visibility into it. Meanwhile, this issue of getting the data should likely be marked answered...

[jtrucks]

I think you need to read http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Addfieldsatsearchtime and check out the http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Addfieldsatsearchtime#Use_interactive_fiel... section specifically. Try to extract some of the fields with your new source and then it will suggest what extracts it sorta looks like. Also, you need to have a much stronger understanding of how Splunk field extractions, sources, and sourcetypes work to get much further managing it.

[glenngermiathen]

That doesnt seem to be it. The firewall doesnt have the source:: before it.

[jtrucks]

I suspect the new firewall might be showing as a different source and the extracts are per source, not sourcetype. Go to Manager » Fields » Field extractions to see if the firewall extracts are listed with "source::" in the front. If so, you may need to create a new one using sourcetype instead, or copy the existing one and create it using your new firewall source.

[glenngermiathen]

The firewall information is getting to the Splunk web. I can see the raw data, but it isnt being parsed like the old firewall we were using. I cant create searches by src or dest IP port etc.

[glenngermiathen]

Yes, UDP into the ubuntu forwarder

[jtrucks]

To clarify, it is a UDP feed into the Ubuntu machine?

[glenngermiathen]

It appears to be a straight UDP feed into the forwarder.

[jtrucks]

That means that your ubuntu system is a Splunk Forwarder client and the Windows system is the Splunk Indexer server. Go look at the "source" field for any event in your indexer listed as "sourcetype=syslog" like:

sourcetype=syslog | head 1 | table source

That should get you a list of the source, which is most likely a file path to the files on the syslog machine. It could be a TCP connection, which means the data goes directly to the indexer, which makes it a rather different situation.

[glenngermiathen]

Im not sure if this is right or not, but I dont think my indexer is running a splunk web UI. From my understanding the indexer is just a command line ubuntu build which feeds into the splunk web head on a Windows build. I checked the data inputs from the web interface on the Windows box, but I couldnt see anything that helped me there. Any other ideas?

[jtrucks]

Ahh! In your indexer's splunkweb UI, go to Manager » Data inputs and look in those sections to see if you can find something obviously labeled that corresponds to your syslog based sources. This will start showing you where these things come from to get to your indexer.

[glenngermiathen]

I guess that is where Im confused, so Im taking over an existing build for the former admin that left. Mutiple sources are feeding into the indexer and I want to add a new firewall into it too. Im trying to determine where each data stream is going, but when I look at both /etc/syslog-ng.conf and /etc/rsyslog.conf I dont see anything indicating this information.

[jtrucks]

If the indexer is getting the data via syslog, look at /etc/syslog.conf, /etc/syslog-ng.conf, or /etc/rsyslog.conf depending on what syslog daemon you use.

[glenngermiathen]

All great information. I know our firewall is feeding data into the indexer, but how do I determine where the /path/to/log/directory or /path/to/log/logfile is so that I can add the data source?

[jtrucks]

$SPLUNK_HOME is the installation location of Splunk on that system. Usually on *nix based machines, this is /opt/splunk/ and I think on Windows is C:\Program Files\Splunk\

[glenngermiathen]

Thanks, how do I determine where $SPLUNK_HOME is?