Even though the official docs are not very explicit on this aspect, TailingProcessor (for plain log files) and ArchiveProcessor (for archived logs) have slightly different implementations: both TailingProcessor and ArchiveProcessor will have the same behaviour if the CRC check validation outcome results is a "no match" -> new file. The log will be read and ingested by splunk in full. if the CRC check validation outcome is "match" then we have a difference between the 2 components: TailingProcessor will check also if the the size has changed and new events been added. If that is the case, then the new events will be read and ingested as well. ArchiveProcessor, once the CRC check validation outcome is confirmed to be a "match" (already known file), will assume that the file is "old" and already ingested and will just skip any other processing. For the time being these are the various possibilities I currently see to move this forward (I have tried to summarise them in 3 macro categories): A. sourcelog management logic change, for example: A.1. instead of archive the older logs, simply rotate or rename them as plain log file so that the ArchiveProcessor component is avoided. A.2. purely read archive files and not plain log files, so that all archive files are always seen as "new" and read and ingested in full by the ArchiveProcessor, with no change of missing events. A.3. extend the log management logic, so that it becomes aware of an ongoing splunk outage, also from a temporal perspective, so that, for example, archives being generated during a splunk outage, are being extracted once again immediately after spunk is back available, so that the TailingProcessor can finish off reading all new events which had been ignored/skipped by the ArchiveProcessor earlier. A.4. I am sure that there are other variants of things which could be done in order to change the sourcelog management on the UF servers in question. B. a custom input implementation, for example a custom scripted or a modular input. C. a Splunk Idea (new portal for ERs and FRs) to be raised for future evaluation/implementation based on the PM prioritisation. ... View more

What is happening here is that you are hitting a mongodb limitation, as described here: https://docs.mongodb.com/manual/core/index-multikey/#limitations This happens if more than one to-be-indexed field in the collection is an array. This potentially can happen if you first define a collection without any accelerated fields and then feed the collection with some SPL creating 2 or more multivalue fields like: | makeresults  | eval Field1="andy, matteo, john, brian", Field2="greg, daviann, zofnat, devon"  | makemv delim="," Field1  | makemv delim="," Field2  | outputlookup MG_Collection append=t  Once you define the field acceleration within collections.conf and then restart, you will hit the problem. If instead you define straight away the field acceleration in collections.conf and you attempt to save 2 or more multivalue fields within that KVstore collection, the search will fail and not store the data. Hope this helps. ... View more

Splunk Engineering believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service, as explained earlier. ... View more

You can use following approach in order to solve your issue: please check following directive: http://httpd.apache.org/docs/2.2/mod/core.html#allowencodedslashes If this is not set to "NoDecode", then the proxy will try to decode and, in case like this one when it has to deal with slashes, this will lead to a failure, because it doesn't manage it as a string as it should. You can verify your Apache version by using for instance the command: rpm -qa | grep httpd If you are using an Apache version older than 2.2.18, then please upgrade and change the directive accordingly. Here is an example about the Apache configuration after the change: <VirtualHost 10.10.10.22:80> ProxyPass / http://10.10.10.18:8000/ ProxyPassReverse / http://10.10.10.18:8000/ AllowEncodedSlashes NoDecode </VirtualHost> So please set AllowEncodedSlashes NoDecode under the appropriate VirtualHost. Adding it anywhere else will not solve the issue. Restart Apache afterwards in order to apply changes. Hope this helps. ... View more

Limits for controlling disk space in Splunk can be changed The relevant stanza and parameter of interest in server.conf is: [diskUsage] minFreeSpace = <num> For more details please look here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Setlimitsondiskusage This can be changed on any Splunk installations as explained on the online documentation: "for all installations, including forwarders, you must have a minimum of 5GB of hard disk space available in addition to the space required for any indexes." The default is 5000 and this value can be changed as explained before. For more details, please check here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Installation/Systemrequirements#Recommended_hardware Hope this helps. ... View more

The First Time Run wizard in Windows Infrastructure app 1.1.0/1.1.1 and Microsoft Exchange app 3.1.0/3.1.1 also internally checks if the SA-ldapsearch app is enabled. Could you please make sure to enable the app, restart Splunk and then try again? If you should still face the issue again, then please raise a support case (if not already done) and provide: the full developer tools output (for Chrome for instance) for the disabled "next button". a new diag of the SH. the exact browser versions involved. The Windows Infrastructure and Microsoft Exchange app known issues section have been amended as well: http://docs.splunk.com/Documentation/MSApp/1.1.1/MSInfra/Releasenotes#Current_known_issues http://docs.splunk.com/Documentation/MSExchange/3.1.1/DeployMSX/Releasenotes#Current_known_issues Hope this helps. ... View more

This can be solved by following one of the 3 possible approaches listed in this documentation article: http://docs.splunk.com/Documentation/Splunk/6.2.1/Knowledge/Aboutsummaryindexing i.e. Report acceleration - Uses automatically-created summaries to speed up completion times for certain kinds of reports. Data model acceleration - Uses automatically-created summaries to speed up completion times for pivots. Summary indexing - Enables acceleration of searches and reports through the manual creation of separate summary indexes that exist separately from your main indexes. Hope this helps. ... View more

Issue has been fixed and actually logs were not lost, also considering that useACK was enabled and no further connectivity issues were reported. The timestamp extraction configuration on a couple of new IDXs was wrong. These IDXs didn't get the right props.conf configuration as the necessary setting had neither been deployed manually nor put on the DS to be pushed out. After correcting props.conf on the involved IDXs and restarting them, the issue was solved. ... View more

Given csv files with 30k to 60k lines, the performance is pretty good here and there isn't anything special really in order to improve that. Lots of other customer have the same load times having csv files with just a few hundreds of lines in them. Hope this helps. ... View more

If you would like to use the "Subject Alternate Names" in SSL certificates, then Splunk provides following parameters which should cover the use-case you are referring to: outputs.conf: sslAltNameToCheck = <string> * Check the alternate name of the server's certificate against this name. * If there is no match, assume that Splunk is not authenticated against this server. * You must specify this setting if sslVerifyServerCert is true. sslVerifyServerCert = [true|false] * If true, you must make sure that the server you are connecting to is a valid one (authenticated). * Both the common name and the alternate name of the server are then checked for a match. * Defaults to false. More details here: http://docs.splunk.com/Documentation/Splunk/6.1.4/Admin/Outputsconf In addition in server.conf: sslAltNameToCheck = <alternateName1>, <alternateName2>, ... * If this value is set, and 'sslVerifyServerCert' is set to true, splunkd will also be willing to verify certificates which have a so-called "Subject Alternate Name" that matches any of the alternate names in this list. * Subject Alternate Names are effectively extended descriptive fields in SSL certs beyond the commonName. A common practice for HTTPS certs is to use these values to store additional valid hostnames or domains where the cert should be considered valid. * Accepts a comma-separated list of Subject Alternate Names to consider valid. * Items in this list are never validated against the SSL Common Name. * This feature does not work with the deployment server and client communication over SSL. * Optional. Defaults to no alternate name checking sslVerifyServerCert = true|false * Used by distributed search: when making a search request to another server in the search cluster. * Used by distributed deployment clients: when polling a deployment server. * If this is set to true, you should make sure that the server that is being connected to is a valid one (authenticated). Both the common name and the alternate name of the server are then checked for a match if they are specified in this configuration file. A certificiate is considered verified if either is matched. * Default is false. More details here: http://docs.splunk.com/Documentation/Splunk/6.1.4/Admin/Serverconf ... View more