What we have here is an internal identifier that we call the 'pipelinechannelset' and is used to ensure that data from a particular input stream is not mingled with data from another stream. This is primarily used for network inputs where we would have incoming streams from multiple sources via the same TCP port, 9997 by default. In the case of local file inputs, it's not necessary to have an identifier like this as our default parsing machinery already has the ability to keep data from different files separate - so that explains why you will sometimes see '/n' versus a number. The more incoming data-streams you have (i.e. the more forwarders in your deployment), the higher this number will be. ... View more

This event happens in the context of distributed search. It is coming from bundle replication, which is attempting to tar up all of your app files to push the search bundle to your indexers. In order to make this manageable, Splunk has a default limit of 50MB, which can be tuned with the following setting in distsearch.conf, in the [replicationSettings] stanza - concerningReplicatedFileSize = * Any individual file within a bundle that is larger than this value (in MB) will trigger a splunkd.log message. * Where possible, avoid replicating such files, e.g. by customizing your blacklists. * Defaults to: 50 However, the better solution here would be to simply blacklist these, and any other large files that are not necessary for searching on the indexers. Read the information here about controlling the size of your replicated bundles - http://docs.splunk.com/Documentation/Splunk/5.0.4/Deploy/Configuredistributedsearch#Limit_the_knowledge_bundle_size And then for any changes you want to make to white & blacklist settings, you can edit the distsearch.conf file - http://docs.splunk.com/Documentation/Splunk/5.0.4/admin/Distsearchconf ... View more

The instructions in the docs are for specifically resetting auto-sourcetyped data, but you have already set a manual sourcetype in inputs.conf, so it's never going to get overwritten again, unless you specifically use a props/transforms entry to re-write it completely, an example is posted here - http://docs.splunk.com/Documentation/Splunk/5.0/Data/Advancedsourcetypeoverrides#Example:_Assign_a_source_type_to_events_from_a_single_input_but_different_hosts Another alternative would be to remove ' sourcetype = syslog ' from inputs.conf and rely on a combination of auto-sourcetyping and other props.conf stanzas to set the sourcetypes on the non-syslog data. ... View more

http://wiki.splunk.com/Community:Run_multiple_Splunks_on_one_machine The link above will take you to instructions on how to successfully configure & run multiple instances on a single machine. The problem you are encountering is a known behaviour on Mac OS X. In running the install package for the second time, the OS inadvertently clobbers the original install, removing files and directories. Usually it will leave the '$SPLUNK_HOME/etc' and '$SPLUNK_HOME/var' directories intact, so your configurations & data should still exist, but the instance will not be in a usable state and the best option at this point is to move what is left aside, download the .tgx package, extract it to the desired location and move your configs and data back over. Assuming your 2nd instance installed succesfully when you ran the .dmg that caused this problem, you should now have 2 instances on one machine. Please read the steps in the article linked above to ensure that there are no conflicts between the instances. ... View more

You need to ensure that the User running Splunk (by default the 'Local System User' on a Windows instance) has full access permissions to the $SPLUNK_DB location. When Splunk starts up, it will run through a validation check on existing index directories to verify that it has the correct permissions to create & modify files in those locations. The user needs full permissions, read + write ... View more

This is actually an issue with the license functionality and the automatic expiration function. When working correctly, the license lifetime should start counting down from the first time you start up a new installation. We have refreshed most the packages on the download page but you can also access an updated enterprise trial license here To apply it, simply place the file in $SPLUNK_HOME/etc/licenses/download-trial/ and restart the Splunk instance. If your browser just displays the raw license .xml contents, just copy that text to a file called enttrial.lic in the above location. ... View more

In order to address this issue and spread the data evenly, use a regular (heavy) forwarder to collect the data and parse it before sending it to the indexer. With the Universal Forwarder, minimal parsing is performed on the forwarder side before sending the data onwards. This means that the UF has no idea where line-breaks occur between events, so in order to use auto-LB, it has to wait until there's a break in the data-stream before switching the output connection to a new indexer. The same behaviour would be observed if it was monitoring a file, and the logging application never stopped writing to that file. As long as data from a specific source keeps appearing fast enough, the UF will continue to send that data to a single indexer in order to avoid corruption of the index. A regular forwarder will parse the data fully parsed before being sending it to the indexers, making it easy to identify points where the connection can be switched. Note that using this instance will increase the resource usage on the host server, so if that box is running critical applications, we should advise using a separate, dedicated box for this purpose. ... View more

The quick answer is yes, you can modify the settings in the .../default/transforms.conf by creating a stanza of the same name in the `...local/transforms.conf' file. The default settings are based on what we expect syslog data to look like but it's not going to match every possible format out there. Just remember than any changes you make to files in the default directories may get overwritten on an upgrade, so make sure you always make your changes in the local directory. ... View more

We actually get this question quite a lot in the support team, and my usual response is: What kind of performance stats are you looking for? Splunk has 2 main operations, indexing and searching. Both of these operations are dependant upon the hardware resources available, the more resources, the faster Splunk will run. I'm not just referring to CPU and memory, Splunk is also very i/o intensive, so the speed of your storage volume is also very important. Further, if you intend to use RAID, that will also affect the performance numbers. Splunk recommends RAID 0 for best performance, and the recommended hardware config is detailed here The performance of your server is also dependant on the data you are indexing and searching on. If you are just interested in standard single-line syslog, containing key = value data, Splunk will handle that data like a champ, and eat it up as fast as you can feed it in, provided that your disk is fast enough. If all of your events are multi-line however, with varying lengths, data format etc, Splunk will be slower to index it and searching will also be impacted. The only way to know for sure how Splunk will perform with your data, is to run some tests with real data samples. There is an app on Splunkbase here that will help you with this, it's mainly a sequence of CLI commands that runs a test with a dataset you specify. As you can see, there's no easy answer to this question, as there are a lot of dependancies, but on a well-tuned, beefy server we would expect to see average indexing thruput of 4 - 7 Mb/sec. Anything higher than that would likely impact search performance. ... View more

Yes, it makes sense and yes, what you have heard is pretty accurate. Moving data from one architcture/file-system to another is usually not possible. We have seen some cases where moving data from Solaris to Linux has worked, if the Solaris box was x86 arch. If you have a SPARC box, then it's just not going to work. If you want to confirm, you can try moving just one bucket as a test. The easiest option for you is the one you have described, setting up an instance on both architectures to act as search peers until such time as that data is no longer needed. Another, more time-consuming option would be to export the data from the original index and import it into the new ones, using a method similar to the one described here Migrating data from one instance to another isn't complicated, and is simply a case of copying the buckets from one location to another, as described here ... View more

Have you heard of the bump endpoint ?? If you hit /info on any instance, and there's a link called static resource cache control near the bottom of the page. Click that and it'll allow you to increment the "bump" number on the instance. We cache static files very aggressively. Ctrl+ will refresh it for you, but not for your other users. Clicking the bump endpoint works for all because it changes all the static URL's for everyhing on the entire server ... View more