Do you have an outputs.conf on your forwarder that tells it to send data to your indexer? If not, here's the basic documentation on doing that: http://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Configureforwardingwithoutputs.conf Here's the flow: The forwarder is installed on the host that generates nginx logs. It has an inputs.conf (defined as I described above) that tells it which log files to watch, and an outputs.conf that tells it what indexer to send all its data to. The search head will peer to the indexer (assuming you have distributed search configured) and allow you to query the indexer for your data. Now, to answer your specific questions: How should we make sure that 1 instance can only remain as an indexer. What is required to be changed in configuration ? You configure a general Splunk Enterprise instance to be an indexer by giving it an inputs.conf and telling it to listen for data. The basic instructions are here: https://docs.splunk.com/Documentation/Splunk/6.5.3/Forwarding/Enableareceiver By default, Splunk Enterprise has an index called main and some internal indexes, so all of the data from your forwarders will go into that main index. If you want to use custom index names, instructions are here: http://docs.splunk.com/Documentation/Splunk/6.5.3/Indexer/Setupmultipleindexes Do we require outputs.conf file on search-head to forward the data to the indexer ? Outputs.conf is required on any host that should be forwarding logs to another host (typically that receiving host will be an indexer). So in a typical configuration, you'll have an outputs.conf configured on all of your forwarders, as well as on your search heads (so they can send their own logs to indexers to be indexed). ... View more

This Splunk blog post has a trick that's fully supported and will reduce the size of your Windows events: https://www.splunk.com/blog/2014/11/04/splunk-6-2-feature-overview-xml-event-logs/. The post explains how to send your Windows logs in as XML, which is parsed perfectly by the Windows TA, but the events are a lot smaller because they have less duplicate data (in fact, the XML versions of the events have more useful information stuffed into fewer characters). According to the blog, it reduces event size by about 70%, but in practice it looked like a smaller reduction in my environment. ... View more

Is the KV updated by just a few well-maintained searches? i.e., will it only be updated in controllable, predictable ways? If so, you can do this by adding a last_modified field to the KV ( | inputlookup example_kv | eval last_modified=now() | outputlookup example_kv ), and then keep it updated by making sure that all of your updating queries modify that last_modified field as well (using now() as the value). If you do this, your updating query will go from something like this: | inputlookup example_kv | search _key=544948df3ec32d7a4c1d9755 | eval CustName="Marge Simpson" | eval CustCity="Springfield" | outputlookup example_kv append=True to something like this: | inputlookup example_kv | search _key=544948df3ec32d7a4c1d9755 | eval CustName="Marge Simpson" | eval CustCity="Springfield" | eval last_modified=now() | outputlookup example_kv append=True Then when you're exporting you can just search for ... | where last_modified>relative_time(now(), -1d@d) (adjust as needed). ... View more

Not a problem! If that's the case, there are two ways you can go: 1) Create a custom Python command sendslack that posts the results to Slack and use the same logic as Option 2 above. 2) Take a look at the third option I mentioned above. While it requires a scheduled search and two alerts, the two alerts are basically free in terms of performance/load. All you need is a very simply KVstore that has one field: last_count. So your main savedsearch would basically be ... | stats count | rename count | outputlookup alert_count_lookup . and each alert query would just be | inputlookup alert_count_lookup | search count>x count<y+1 . ... View more

I can think of two ways of doing this: 1) The easiest, but most inefficient, option is to set up two almost identical alerts. Both alerts would run the same search at the same time, but they'd have different actions. One alert would have an action with one range (x-y), and send to the first set of recipients, and the second alert would use the same query about have the other range ((y+1)-z), and send to the second set of recipients. The problem with this option, of course, is running twice as many searches as you need to. 2) Instead of an alert, create a saved search that uses the | sendemail command (that's basically what an alert is anyway). Basically what you can do is have an eval that sets a "to" field based on the ranges. Your query would end up looking roughly like ... | stats count | eval to=case(count>10, "address1, address2", count>5, "address3, address4", 1=1, "") | sendemail to="$result.to$" from="sender@example.net" subject="test" message="test2" . If the to field is empty, (i.e., count<5), nothing will send. Here's a Splunk Answers post that describes how to use sendemail this way: https://answers.splunk.com/answers/213340/how-to-get-splunk-sendemail-command-to-send-multip.html A third option is basically a more efficient version of option 1: make a kvstore that tracks state, and use a saved search to populate that kvstore. Then create two alerts that just check that kvstore (which is basically a zero-cost query). ... View more

Have you tried searching "All Time"? It would be easy for Splunk to mistakenly index this data with the wrong time, as you have the correct time field buried in the middle and aren't telling Splunk what time to assign. ... View more

What are you ultimately trying to accomplish? I only ask because I can't think of any reason you would need to create a lookup table manually in /opt/splunk/var/run/splunk . ... View more

The only potential issue I can think of is that "service splunk ..." requires elevated privileges while "/opt/splunk/bin/splunk ..." requires the privileges of whichever user owns /opt/splunk and its files. That's where I'd investigate first. ... View more

Hi Sumangala, What app are you in when you run "| makeresults | eval name="xyz" | eval token="12345"| outputlookup 12_april_lookup.csv append=True createinapp=True"? If it isn't the same app where you created the initial lookup, Splunk will create a new lookup in the app in which you ran that "| makeresults..." query. All you should have to do to fix this is switch over to appname in SplunkWeb and run the query again. ... View more

There is no way to do what you want by just using one inputs.conf file. The closest I could think of would be to have one primary inputs.conf app that defines all the monitoring stanzas and disables them, and then a set of secondary inputs.conf apps, each assigned to just one forwarder, that just enables the appropriate stanzas for that forwarder. But I don't recommend doing that, as it's just another layer of complexity to manage (so now when you change something you have to change it in at least two places instead of one). In the past, when I've had the need for very complex inputs.conf configurations, with hosts collecting data on behalf of thousands of other hosts, I've had a lot of success with building a spreadsheet that tracked all my inputs and then a Python script that processed the spreadsheet into the inputs.conf file(s) I needed. This had major benefits: I was just editing things in one place (the spreadsheet), which was a lot easier to read, sort, and search than an inputs.conf file; if I needed to change the way things were broken up, I could make the change in the Python script and have it automate the changes downstream; and much lower typo risk. I highly recommend this approach. ... View more

There's actually a lot to this, so you should start by heading to this link, which contains many docs that describe the process in detail: http://docs.splunk.com/Documentation/Splunk/6.5.3/Security/HowSAMLSSOworks There are actually three major sections in that document tree that may help: "Authentication using single sign-on with SAML", "Authentication using Proxy SSO", and "Authentication using single sign-on with reverse proxy". Which one you need will depend on how you plan to implement SSO. If you have a SAML solution (such as Microsoft ADFS) and plan to use LDAP/Active Directory group membership to set Splunk user roles, the SAML method is pretty easy to set up using the SplunkWeb (and coordination with your ADFS admin). If your setup is more complicated than that, you'll have to go with one of the proxy solutions, which are a good deal more complicated (but once you have them working you can largely set and forget). ... View more

Unfortunately the srchTimeWin setting is on a per-role basis, not a per-index basis. You give your roles access to a set of indexes; you give them a srchTimeWin; and then they can access all of those indexes as far back as that srchTimeWin allows. The only ways I can think of to do what you want would be ugly hacks. Example (I'm not 100% sure this will even work--it is unsupported, has unpredictable behavior, and if it goes wrong, it'll go really wrong. But I'm putting it here in the hope it'll spur other ideas): - Assume your indexers are running in Linux - Create two indexes: foo, bar, and baz - Foo will be the index that admins access to get at all the data. Bar will be the index that users access to get the hotwarm data. Baz will be the summary index. - Send all the new data to foo. Never send any data to bar. Set the index settings for bar to never expire or move any data--just have an unlimited size hotwarm. - Stop Splunk. Delete the baz hotwarm directory (likely located in $SPLUNK_HOME/var/lib). Recreate that directory as a soft link to the foo hotwarm directory. - Start Splunk - Give users permission to search bar and baz, not foo. Give admins permission to search foo and baz, not bar (at least not by default). Give users permission to search any time window. - Resulting behavior: Users can search any index as far back as they want. Splunk will see bar as being a full index (hotwarm only) made from the data in foo, but won't see that it has any data in cold storage. - The big problems here are the risk that Splunk double-maintains the foo/bar hotwarm data in a way that breaks, as well as the difficulty of managing these duplicate indexes. Depending on how often you need to search the cold data, another (supported!) option is to freeze all the data that would have hit cold and then thaw it whenever you need to. You can keep the frozen data on disk just like the cold so that it's fast and easy to thaw when you need it. ... View more

Hi David, The logs you posted show that the Splunk indexer doesn't think you have the _internal and _introspection indexes. These two indexes exist on every Splunk server by default, and they are configured in $SPLUNK_HOME/etc/system/default/indexes.conf. So I see a few possibilities: 1) You edited $SPLUNK_HOME/etc/system/default/indexes.conf. If you did that, the best thing you can do is figure out what else you've been changing in $SPLUNK_HOME/etc/system/default, pull out your changes and copy them into notes, and then reinstall Splunk from scratch. It's really important that you leave $SPLUNK_HOME/etc/system/default alone. 2) You created a local indexes.conf, either in $SPLUNK_HOME/etc/system/local or in an app, and in that indexes.conf your expressly disabled _internal and _introspection. 3) Splunk couldn't create the index directories (by default, these are in $SPLUNK_HOME/var/lib/splunk) due to a permissions issue. Maybe you installed Splunk as root but then ran it as a regular user without changing any permissions? If it is a permissions issue, you'd have to fix that. It might be as simple as stopping Splunk, recursively chowning the whole Splunk directory to a regular user, and then setting Splunk to start as that regular user (instructions here: http://docs.splunk.com/Documentation/Splunk/6.5.3/Installation/RunSplunkasadifferentornon-rootuser#Use_sudo_to_start_or_stop_Splunk_software_as_a_different_user). ... View more

What you want to do is create a local outputs.conf configuration on the Search Head that looks like this: [tcpout] forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_internal) This will replace the default outputs.conf entry, which sends more internal indexes to indexers. Other than the above, all you'll have to do is set outputs.conf to point at the indexer cluster. See this old answer for more detail on how the forwardedindex whitelists and blacklists work: https://answers.splunk.com/answers/339930/how-do-forwardedindex-whitelists-and-blacklists-wo.html ... View more

Here are options that may solve your problem: 1) Switch Splunk to SSO strict mode and connect it to a single sign-on solution so the web UI no longer allows users to manually enter a username/password (documentation here: https://docs.splunk.com/Documentation/Splunk/6.5.3/Security/ConfigureSplunkSSO). Then add all your WebUI users to the SSO solution, but add your admin/REST user to Splunk only--not the SSO solution. If that user ever tries to authenticate to the WebUI, the SSO solution will reject it. But when that user tries to authenticate on the management port, Splunk will bypass the SSO solution (you will have to add the source address that you expect REST queries to come from to the trustedIP list, and just for good measure you should have iptables specifically block the Splunk web port from that IP). 2) I bet you can do something similar to the above by setting up multifactor authentication using Duo (https://docs.splunk.com/Documentation/Splunk/6.5.3/Security/AboutMultiFactorAuth). My guess--I can't find any documentation that addresses it--is that the Splunk management port bypasses Duo just like it bypasses SSO. If that's the case, you'd just have to set up a second factor for the REST account and then destroy that second factor; that way that account can still use its username/password to login through the management port, but it won't have the second factor required to login to Splunk Web. A couple of things you should consider if you do what you're intending: If you're giving this account admin privileges, it can create another account that has admin and web UI privileges (SSO example above notwithstanding). You can create an alert to look out for that, but this account can also disable that alert. You can create an alert that looks for the account disabling that alert, but... turtles all the way down. So think carefully about those possibilities before you put your trust in this solution. ... View more

Yes! One way to do this is with a custom search command. Let's say your python script is called "testscript". You could rewrite as (or call it from) a Python script that Splunk recognizes as a custom search command, and then you could execute it as a Splunk search like this: "| testscript". Then you could call that search (adding arguments as needed) using the Splunk REST API. Here's a link that explains how to create custom search commands: http://dev.splunk.com/view/python-sdk/SP-CAAAEU2 Here's the basic documentation on running a search from Python using Splunk's REST API: http://dev.splunk.com/view/python-sdk/SP-CAAAEE5 ... View more