See also this excellent searchable reference for windows event codes. Primed with 540, one of the more useless ones: https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=540 And here is a quick guide on how to sift them from the mouth of the beast itself: https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/appendix-l--events-to-monitor Good starting point if Windows isn't your thing. As ever, work with system owners to make sure their requirements are taken into account--but start from a position of some knowledge at least. You'll waste a lot of licence quota on what is basically noise from Windows if you aren't firm with them. ... View more

Pretty much as I thought then. It seems to me that if you're using something like git to store your splunk configs--and you should be!--it will be simpler in most cases to assemble an app to splunk your git repo and search your configuration elements that way. I have actually done this myself years ago but neglected to retain the exact method when I left that engagement. What I do remember is that it involved using only one depth of the repo (i.e. current master versions for all configs) and storing the full path, the conf file path and name, and the specific element which was configured. Keyword searches for any item then showed you what sort of a thing it was, where it was defined and additionally where it was used. This was also handy for finding out whether you had an existing sourcetype you could use, or whether you had to create a new one. Might seem trivial, but it isn't when you have hundreds or thousands of data sources and sourcetypes in a large enterprise. Ah well, might need to figure out how to do it all over again! ... View more

In that case it looks like a permissions issue. If you want to use these aliases inside search, change the permissions for the aliases in local.meta so that they're exported system wide for read. And also any other objects inside the app that you might want to use from another app. I haven't loaded this app since I don't have a fortigate, but typically an installation will set the knowledge objects to be only visible within the originating app. Make sure, of course, that any objects you do this to, don't conflict with things exported from other apps (unlikely, as they should be tied to sourcetypes only applicable to fortigate, but worth checking nonetheless), or you won't get the results you expect. You can use btool to check the work. ... View more

Pretty sure you'll need to do this yourself then. It boils down to studying the applicable CIM topic area, working out what tags, eventtypes and field aliases are needed for your sourcetypes, and creating them. Sounds like you won't need to do any extractions or lookups, but I'm not familiar with Fortigate devices so you'd need to check this as well. Not terribly helpful help, but this is simply what happens sometimes. Cheers, Charles ... View more

You won't be able to run a SPARC V9 binary on Solaris X86. Get the X86 UF from here: https://www.splunk.com/en_us/download/universal-forwarder.html#tabs/solaris ... View more

OK since I don't have a solaris box any more this will be a bit tricky 🙂 The part of the scrpt that executes the solaris-specific commands is here: elif [ "x$KERNEL" = "xSunOS" ] ; then assertHaveCommand vmstat assertHaveCommandGivenPath /usr/sbin/swap assertHaveCommandGivenPath /usr/sbin/prtconf assertHaveCommand prstat CMD='eval /usr/sbin/prtconf 2>/dev/null | grep Memory ; /usr/sbin/swap -s ; vmstat -q 1 1 ; vmstat -s ; prstat -n 1 1 1' PARSE_0='/^Memory size:/ {memTotalMB=$3} (NR==5) {memFreeMB=$5 / 1024}' PARSE_1='(NR==2) {swapUsed=0+$(NF-3); swapFree=0+$(NF-1)}' PARSE_2='/pages paged out$/ {pgPageOut=$1} /pages swapped out$/ {pgSwapOut=$1}' PARSE_3='/cpu context switches$/ {cSwitches=$1} /device interrupts$/ {interrupts=$1} / v?forks$/ {forks+=$1}' PARSE_4='/^Total: / {processes=$2; threads=$4; loadAvg1mi=0+$(NF-2)}' MASSAGE="$PARSE_0 $PARSE_1 $PARSE_2 $PARSE_3 $PARSE_4 $DERIVE" You'll see that having executed a series of commands it runs them through PARSE 1-4 and then DERIVE, which can be found elsewhere in the script. If you're getting a negative vaiue it's probably in DERIVE which the only place where subtractions are occurring. You can debug it by echoing the values seen by DERIVE and seeing which one looks wrong. It should be simple enough. Don't forget the -S change to the second vmstat. vmstat manual link here: https://docs.oracle.com/cd/E23824_01/html/821-1462/vmstat-1m.html HTH ... View more

I wrote this app four years ago and it's no longer under active development; I don't have any solaris boxes left but I'll have a swing at it anyway! Can you please provide the output of vmstat.sh run from the command line, and also vmstat -q 1 1 which is the relevant command that it runs. With solaris there are usually better commands than the generic ones and the main reason I wrote the TA and app in the first place was that solaris offered much better ways to get the stats in most cases, and lots of extra information that the standard NIX app just doesn't know how to get. Perhaps vmstat itself is giving the wrong answer; perhaps it's being wrongly formatted by the app (more likely). I am suspecting a rounding error without having seen what you're getting. This script was one where I attempted to stick with the platform-indpendent style used in the Splunk official NIX TA. Later I gave up and rewrote the scripts wholesale as it was just getting too complicated, and solaris has diverged very far from generic NIX if you really want to know what's going on. You could most likely get much more accurate results by ditching vmstat altogether and re-implementing it with dtrace. This is no longer something I'm prepared to do. I see one problem already though, the next command in the script, vmstat -s, should have -S as the option. Cheers, Charles ... View more

Can this problem be solved using a standalone search peer which is just for this purpose? In some places I've worked putting the production cluster into maintenance mode to load old data is simply not an option. There must be a better way. It seems needlessly restrictive and complicated to have to restore the buckets on the cluster nodes it first came from. Enhancement needed? ... View more

Bumping this topic again. Why? Because Answers seems to be fairly evenly divided between use INDEXED_EXTRACTIONS and don't. Here is someone who has actually benchmarked them both: https://www.hurricanelabs.com/blog/splunk-case-study-indexed-extractions-vs-search-time-extractions So thank you Ryan for laying out the issues in the most detail I've seen so far. Upside of INDEXED_EXTRACTIONS: improved performance with some scenarios; note he uses the :: operator, not "=" Downside: increased resource usage and perhaps greatly increased disk usage (stands to reason); and, as I discovered, no HF routing for any sourcetype processed this way. So you need to consider your use case carefully. In the past, and in the absence of this information, I would have gone INDEXED_EXTRACTIONS because it's easy and very reliable. Now I think I would use KV_MODE for json/XML stuff and DELIMS for CSV, unless there was a compelling reason not to. It would be great if INDEXED_EXTRACTIONS allowed you to blacklist or whitelist fieldnames or patterns to conserve resources, especially disk. Right now it doesn't. ... View more

Yeah thought it might be something like that because I did notice that 'Thruput' wasn't listed. Interesting though, that it can be rounded even though it's hiding. Well, that's show business. ... View more

The docs are fine as far as they go, but the Auto drilldown for the map looks like this: sourcetype=customer | eval l_location=upper(location) | eval state="VIC" | lookup locations suburb as l_location state output lat lon | search lat>=-35.85938 lat<-35.15625 lon>=143.43750 lon<144.84375 So it is automatically generating a latitude and longitude range presumably corresponding to the radius of the cluster marker, and supplying this in the search parameters. I cannot see how to pass this range in custom tokens, since the properties of the cluster marker aren't available. The statistics table for the parent map has four fields: geobin, latitude, longitude and a count (of what I'm not sure, not customers certainly) if there's a way to group the lats and longs I don't see it. As an experiment I tried passing $click.lat.value$ into the child dashboard to see what I got, which was exactly one lat value--no range. So, what magic is going on here? ... View more

Seen this of course, and it's nice as far as it goes. Also had a look at what gets shipped with the search app: how complete is this? This topic is barely even touched in the current docs. As far as I can tell, there is no reference resource for the hundreds and hundreds of other css attributes used by splunk. People who do this a lot more than I do, seem to be spending lots of time messing around with an inspector of some kind, tracking through the attribute hierarchies, which may or may not be particularly human readable. Then there's the tedious procedure of edit, bump and reload to see if the knob you've twiddled is actually the right one, and what interesting side effects might happened. Seriously, is this the state of the art? Anyone recommend a good CSS tool that can take some of the pain out of this, and does not require you to be such an expert that you don't need it? Not built-in browser object inspectors... ... View more

You can customise what happens with frozen buckets as documented here: http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Automatearchiving Thus you can use any type of storage at all, and control how your stuff gets there, as long your indexer can get to it somehow. ... View more

The grub.conf solution, as someone else already pointed out, can be overwritten by a kernel update. What might help is a proper systemd-type service to do this as a dependency for splunk before it starts (and a lot of other software doesn't like THP either so this would be widely useful). Alternatively some proper THP controls created upstream in the distros, but I'm not holding my breath. hugeadm in ubuntu is a step in the right direction but that doesn't work properly either as I discovered when I tried to use it. It's not sticky IIRC or perhaps it was some other problem--I don't recall. And it doesn't help you with any other distro anyway. Yet another possiblity would be for splunk to actually address the problem and update the boot script apparatus. The ulimit stuff is out of date too. The only thing I've found that actually worked is here: http://stackoverflow.com/questions/39506149/ubuntu-16-04-systemd-redis-issues-with-ulimit and this is a lot of messing around to achieve what used to be easy but now isn't. This still leaves you with a problem if someone runs $SPLUNK_HOME/bin/splunk start or restart without a wrapper script. All in all quite exasperating and I freely admit I've created an ugly hack. ... View more

The only other time I've seen something like this is where you have a combination of INDEXED_EXTRACTIONS (which I use) and routing as documented here: http://docs.splunk.com/Documentation/Splunk/6.5.1/Forwarding/Routeandfilterdatad There is unfortunately SPL-98594 about this which from what Support tell me is very difficult to fix and has low priority. It's been open for three years now but I didn't know about it until after I'd written the TA. If you are using it in conjunction with routing, unexpected things may happen. I cannot suggest a workaround if this is the issue; we had to turn off routing. This may or may not be an option for you. The symptoms were much the same: some but not all sourcetypes being silently dropped on the floor. If this isn't what's happening, please discuss with me further by email as suggested above. ... View more

Putting it in rc.local is too late. Using the legacy sysV init script which splunk enable boot-start deploys, it's already started by then. What I do is modify /etc/init.d/splunk as follows: Add function: RETVAL=0 #existing # disable hugepages disable_huge() { echo "disabling huge page support" if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi } Use it: case "$1" in start) disable_huge splunk_start ;; stop) splunk_stop ;; restart) disable_huge splunk_restart ;; Tried all the other things here and in places like stackoverflow, seems to be the only thing that actually works. Yes we should all be using systemd methodology everywhere, but you know what? CBF learning yet another OS startup gadget, of which the woods seem to be full. ... View more

Let's take this offline until I've worked out what's going on. Please use the email in my profile. I will post the solution/new version when sorted. ... View more

I would also add that I have several other reasons for posting a root-only TA: 1. Modern data centres use automation (jumpstart, ansible, puppet, whatever) for everything so the question of user access to a solaris box should not come up. 2. I cannot even begin to guess at the policy framework in use for sites that are using the ACL mechanism. It may be that the list of required permissions might still fall foul of your policies. Not a discussion I want to engage in. 3. Matching the commands used to the required ACLs and creating the procedures to apply and test them is in itself not a lump of work I can justify the time for at the moment. 4. Running things setuid was out of the question. ... View more

So if I understand correctly you've changed the index used from solaris to ia. The searches used in dashboards do not specify the index (this is deliberate, so you can use any index) but you do need to make sure all indexes you have used are accessible by default, and are searchable by default for the role you are using to access the events. So check permissions on all the objects. Also check to see if maybe you missed some and there is an index called solaris with some contents. Your missing events might be there. Any errors resulting from script execution should show up in splunkd.log, search for ERROR ExecProcessor. I've kept it pretty simple, there doesn't seem to me a whole lot that could go wrong with it assuming SPLUNK_USER can execute the underlying commands in the first place. And this can also be a problem. For reasons best known to themselves, the people who wrote many of the OS commands I've used, only let root get access even to display the information, so unless splunk is running as root or you do a whole lot of messing around with ACLs it won't work properly. LDOMS is one example. Only root can run ldm at all. I am well aware that this is not an ideal situation, but I would suggest that if anyone can mess around with the scripts to do bad things, you already have a much larger problem than anything the TA might be able to cause. If you are making enhancements to any of it, please let me know and I'll include them if you want. I wrote this App and TA because A. I think the official *nix app has lost its way and B. In trying to be cross-platform the official stuff doesn't use any of the cool things that are solaris-only, and which tell you about virtual networks and zones/ldoms, zpool, zfs etc. etc. For that matter, it's all out of date wrt modern linux distros as well, and for all the annoying but shiny new UI in the *nix app, it is still using methods 10 years old to collect it. ... View more