The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices. After testing functionality and performance - one good option is REST::Client The example file below is useful for the use case of, testing performance of injection from perl. On my test VM, the code below inserted 20,000 events per minute into Splunk 7.3.1. Of course, this example's real purpose is to show you how to establish the connection, disable SSL verification if needed, and how to inject using POST routines. In a real script, you'd set the JSON content in the $client->POST commands. (Fancy coders might even use additional perl libraries to build up the JSON.) #!/usr/bin/perl -w use strict; use v5.16.3; use REST::Client; my $token = '7a49c676-22b8-40e3-8e37-9dad19a9bdb2'; my $hec_uri = 'https://127.0.0.1:8088'; my $n=3600; # 3600 is one insert every 1 second for 1 hour my $client = REST::Client->new(); $client->getUseragent()->ssl_opts(verify_hostname => 0); $client->getUseragent()->ssl_opts(SSL_verify_mode => 'SSL_VERIFY_NONE'); $client->setHost($hec_uri); $client->addHeader("Authorization", "Splunk $token"); $client->POST('/services/collector/event', '{"sourcetype": "mysourcetype", "event":"my payload here"}'); ## initial test print $client->responseContent(); print "\n=========================="; print "\nTiming $n runs using direct perl libs: "; my $start=time(); my $i=0; while ($i++ <=$n) { $client->POST('/services/collector', "{\"sourcetype\": \"mysourcetype\", \"event\":\"testrun=$$ speed TEST #$i\"}"); } my $end=time(); my $elapsed=$end-$start; print "took $elapsed seconds\n"; ... View more

@michaeljorgensen is indeed correct, things have changed since 2017. ( @cmerriman's answer was correct at that time.) As of this writing, there are two different ITSI courses available - and there's an ITSI certification path as well. Details on the requirements are here: https://www.splunk.com/en_us/training/certification-track/splunk-itsi-certified-admin.html ... View more

Whether one is using the Splunk App for VMware, or the VMware integration into Splunk IT Service intelligence, one typically looks at three different sources of data. The storage paths, latency, and performance data is in the third data source listed below. You also mentioned the ESXi and vCenter logs, so I'm listing what is - and is not - in those log files. ESXi logs ESXi logs contain useful information when debugging issues with individual hypervisors, and to audit certain activity such as direct SSH access. Note that these logs do NOT include inventory data or performance information. Nothing is installed onto the ESXi servers to collect the log data. Collection of ESXi logs requires a simple change to the Hypervisors' syslog configuration. Typically, one configures the ESXi servers to forward their data using syslog, TCP port 1514. Because the hypervisors are already sending this data to the syslog facility, the additional step of forwarding the log data over TCP requires very little overhead. For evaluation purposes or for smaller environments, this data can be sent directly to a server running a full version of Splunk and with Splunk_TA_esxilogs installed and configured to listen on this port. A more scalable and robust solution involves sending the data to a syslog-ng or rsyslog server, where it would be written to disk and then picked up vCenter logs vCenter logs contain information about access to the vCenter environment, audit information (who assigned permissions, added/edited/removed VMs), and health information about vCenter's processes. Note that these logs do NOT include inventory data or performance information. Collection of vCenter logs will depend on which version of vCenter you are using. Older environments (5.x on Windows) require a Splunk forwarder and Splunk_TA_vcenter. Newer environments (6.x on Windows, or linux-based vCSA) each support relay of these logs via syslog. In either case, the collection of the logs does not significantly impact performance of the vCenter server vSphere API data The most interesting data from a VMware environment is not available in the log files, and instead must be collected from the vCenter server using vSphere API calls. This information includes: Inventory data for VMs and hypervisors and datastores (including configuration specifics), performance data for all of the above, vSphere environment alerts. Collection of this data to send to Splunk requires the use of one or more Splunk Data Collection Nodes (DCNs), which are managed by a Splunk Scheduler. DCNs are typically deployed from a Splunk-provided OVA, as a linux-based virtual appliance. The DCNs connect to the vCenter environment using an API call, over TCP port 443 to the vCenter server, using credentials you provide. Read-only access is sufficient for all data collection via the API. The CPU load to the vCenter environment depends on many factors: the number of CPU cores on vCenter, the number of ESXi servers and VMs in the environment, and whether vCenter is Windows-based or vCSA. vCenter CPU load is typically increased by about 5% for mid-sized environments. ... View more

First, there are a number of Splunk Apps that will help you bring in and visualized information from the devices you mentioned. I'd start there, using the appropriate F5, Juniper, and Cisco apps. All are on http://splunkbase.splunk.com Those apps won't help with the end-to-end monitoring you're asking about (which I'll get to as well) but what they will do is provide deeper insight into the operation of those different products. Most (or all) of those devices also support sending information via syslog. Typically this will include hardware level events, logs of configuration changes, and so on. This is generally low volume data but worth looking at. I'd certainly want Splunk to open a ServiceNow ticket whenever a device reports a fan failure or loss of a redundant power supply! With all of that being said, it still doesn't address your need: I'm looking to identify top talkers and want to be alerted when a network link is approaching full capacity. When a switch port is experiencing degraded service, I want to know which servers and applications are affected. What you'll want is the network meta-information that's available from each of your devices. Commonly called "flow" data, this is the data set that will help you answer your question. This comes in many flavors, including: - NetFlow v5 or v9 data from older Cisco gear - J-Flow data from your Juniper equipment - sFlow (for sampled NetFlow) from your switches - IPFIX data from your VMware virtual switches and pretty much every other intelligent networking device released since 2015 Note that all of these network flow types are in binary format; Splunk cannot ingest them directly. Wikipedia has a great write-up on each of these for those who are interested. The TLDR version: NetFlow was invented by Cisco, other vendors had their own versions. IPFIX replaces them as a common, universal standard. In Splunkbase, you'll find a few different TAs from Splunk, one for IPFIX data and one for NetFlow v5/v9 data. They'd help you bring in some of the data, but would not address your Juniper devices or the sampled flow data from your switches. I believe that to accomplish what you're after, you'll want to use NetFlow Logic's "NetFlow Integrator". (See their app at https://splunkbase.splunk.com/app/489/) How it works: First, NetFlow Integrator acts a sort of middleware. It takes in all of the different flow data types, converts them from binary format. When Integrator sees data coming in, it reaches back to the sending network device to do some SNMP-based data collection. This allows Integrator to determine data such as port speed and duplex, and other device information. What Integrator does next is up to you. It can send each flow record to Splunk (converted to syslog format) or send aggregated information periodically, or both. Sending the aggregated data is the best fit for most Splunk environments, Flow data can be VERY high volume and this allows you to keep your Splunk license usage low. Once the data gets to Splunk, you'll finally have your answer. NetFlow Logic has apps on Splunkbase that use the Splunk platform to tie all of the data together. This includes reports top talkers, network utilization/health/saturation, and traffic flows affected by networking issues. It does this even with your VMware switches, top of rack devices, and (as I saw at VMworld this week) VMware NSX. I hope that helps point you in the right direction! While we're talking about the network - I'll mention "Splunk App for Stream" as well. It wouldn't help with the use case you asked about, but it's worth knowing about. Stream allows you to look at the application protocol level to analyze communication between servers via TCP or UDP. When the log files don't give the information you want, Stream allows you to bring in data for both IT Ops and Security use cases. ... View more

Splunk cannot work against ‘nisNetGroupTriple’. That type of group is specifically intended to control who can log into a set of hosts, and as you note the format includes hostname and domain name. The way authentication works, the user's login is validated first - then a selected value from the user's own entry is extracted from that entry. This value is used to identify which other groups contain that user. You can read more about this approach here: http://ldapman.org/authentication Unfortunately, there is no value in the user's entry that would match the format used in the nisNetGroupTriple. We do support the other formats from the NIS / LDAP RFCs: objectClass=rfc822MailGroup objectClass=posixGroup ... Or any other group whose members are stored either as a login name, email address, or full DN. ... View more

In Splunk 6.2.1 (and older), the "enable boot-start" on the Mac relies on the StartupItems feature of OSX. The StartupItems feature was deprecated awhile back, OS X Yosemite removed support completely. With Yosemite, Apple relies on "launchd/launchctl" for process startup. SERIOUS AND MAJOR KUDOS to the folks who created http://launchd.info for a great site on how launchd works! The good news is that you can fix this with the workaround below. This is provided as a workaround, and not officially supported. Test on non-critical systems, etc. If you don't feel comfortable in a terminal window - you should probably not proceed. Use the attached file and save it to: /Library/LaunchDaemons/com.splunk.bootstart.plist Edit the file, note the file path, and change it as necessary. (Unless you're running Splunk in /opt/splunkforwarder like I do!) The sample file is functional but doesn't show off any fancy launchd options, Splunk will run as user "root" unless you modify the .plist file by adding UserName key/value pairs. StdOut / StdErr are set to help troubleshoot, but may be removed if you don't want them. Once the file is edited and saved, issue the following commands to set permissions and load the profile: sudo chown root /Library/LaunchDaemons/com.splunk.bootstart.plist sudo chmod644 /Library/LaunchDaemons/com.splunk.bootstart.plist sudo launchctl load -w /Library/LaunchDaemons/com.splunk.bootstart.plist If everything worked, splunk will start immediately when you run the command above. And at every reboot thereafter. Happy Splunking! <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>com.splunk.bootstart</string> <key>ProgramArguments</key> <array> <string>/opt/splunkforwarder/bin/splunk</string> <string>start</string> </array> <key>RunAtLoad</key> <true /> <key>StandardErrorPath</key> <string>/tmp/splunkboot.stderr</string> <key>StandardOutPath</key> <string>/tmp/splunkboot.stdout</string> </dict> </plist> ... View more