That's really odd! Well, I don't think it matters much, but it does sound like a bad install, maybe something didn't get properly overwritten. If you can request that the Ops Team reinstall 7.0.3 on that machine is might not be a bad idea. I don't think you're in any danger of Bad Things. ... View more

This looks like it may be a bug in the AIX build. I just tried the linux tgz and Mac dmg, and they both look right. I suggest opening a support case, it may have just been a bad build of the AIX UF. ... View more

I reached out to Splunk Engineering (via Support) to confirm this on RHEL7 and Splunk 6.5, and received the following: Support confirms that "...it is still REQUIRED practice, not simply best practice that THP be disabled." Here are Engineering's comments regarding the comment from "oschnar": "His chart is inconclusive. THP doesn't affect/improve indexing rate as much as it severely affects search performance negatively. At low volumes, it doesn't matter, but at higher rates, you'll notice the effect, especially under high search loads/pressure. AFAIK, the severe performance bug was fixed in later Linux kernels (circa kernel 4.x+, but not back-ported to rhel/centos 7.2) so that madvise is the default, and is reasonably safe to use. However, if the box is performing double duty, cycles will be spent to manage and defrag any use of THP outside of the Splunk process. Since we require dedicated ownership/use of the system, enabling THP for "alien" processes is not recommended. I had recently confirmed with engineering that we do NOT use THP, but rather we directly allocate/manage hugepages (not the same feature), so disabling THP will be appropriate for any systems running a heavy Splunk process (such as search heads, cluster masters, indexers)." ... View more

This worked great for me, thanks for posting it! ... View more

You don't need to create the users in Splunk, just configure Splunk to use your LDAP authentication and map groups to roles. If a user exists and has a valid group to Splunk role map, they will be able to authenticate. I suspect you haven't mapped your LDAP Groups to Splunk Roles. See the docs at: http://docs.splunk.com/Documentation/Splunk/6.3.2/Security/SetupuserauthenticationwithLDAP ... View more

Ok, at least we have confirmation of an issue and a bugID. ... View more

Make sure that the lookup of hostdetails.csv is available inside the DBXv1 app context. ... View more

Try doing a "chown -R splunkd:splunkd" on the entire Splunk install tree on the deployer. I had a similar error (and the SPL is probably my complaint) about vague error messages that was a caused by a single bad file in a completely unassociated directory. ... View more

The deployer is part of every full Splunk install. The deployer needs to be a separate instance than the cluster members. See http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/PropagateSHCconfigurationchanges ... View more

Ayn's answer contains the username, but has a lot of extra information that the user inherits from it's roles. Try: | rest /services/authentication/current-context | table username The result that isn't "splunk-system-user" is the current username. ... View more

Often a duplicate app gets created if Splunk has to modify a file in slave_apps and cannot -- I've seen this when, for example, the app in slave_apps contains an unencrypted password and Splunk wants to encrypt it. It can't write it back to slave_apps since it's read only, so it writes it to the normal apps directory. The solution is to have the app with the password be pushed to the slaves in a different (out of band) way, or to have a secondary app that is used for this. An example might be a cluster-bundled app (so it's pushed to slave_apps) called company_data that doesn't contain the plaintext password and a local apps/company_data_unique that is manually copied to all the indexers. ... View more

I would highly suggest avoiding having two IPs listed for the A record of the Cluster Master. Every time an indexer or searchhead tried to go to the one that was down you'd have issues. Instead, I would have a single A record with a very short TTL, so it's easy to switch to the backup Cluster Master if needed by changing DNS. For syncing between the Primary and Backup Cluster Masters I'd use either rsync or better a version control system (git, subversion) and do automated checkins/checkouts. ... View more

Yann is correct. If search heads aren't configured to use mounted bundles they default to the normal method (sending the bundle over the wire to the indexer.) ... View more

Or that you don't use the main index. Try doing | metadata index=* type=hosts | eval age = now()-lastTime | where age > 2 | sort age d | convert ctime(lastTime) | fields age,host,lastTime</code This will search all non-internal indexes and display any hosts that haven't reported in 2 seconds. ... View more

p10 and p90 return the 10th and 90th percentile values1 ... View more

You should copy the file to local and modify it there so an incremental release before the author fixes it doesn't wipe out your changes. I also suggest breaking your logfiles into as small of chunks as you can. We had to have our rsyslog server rotate every 15m to handle our ~400gb/day of logs, but that helped immensely. ... View more

The easiest way is to add a deploymentclient.conf to /opt/splunkforwarder/etc/system/local, but it's also the wrong way. The best way is to make an app on your Deployment Server (and is listed in the DS's serverclass.conf) that has a local/deploymentclient.conf file, then manually copy that app to your UFs at install. That way the app itself is controlled by DS, so if you ever need to update it on the UFs (for example, you change the IP of the DS or want to split them out) you can. For new installations, I've just made a bash shell script that unzips the deployment app files in the UF's etc directory after install, so that it phones to the DS on startup. The DS has a default serverclass.conf for all machines that just serves the deploymentclient app. ... View more

For ongoing knowledge of your indexers, I highly recommend Sanford Owing's Fire Brigade Fire Brigade App on Splunkbase. It will give you a ton of information on the size of indexes and individual buckets for existing indexes, which lets you plan for future growth. (Edited to link to Fire Brigade Version 2 for Splunk 5 & 6) ... View more

Splunk 6.0.3 has been released, which that patches Heartbleed and a separate TLS vulnerability ... View more