In general every thing is come to your user count,Max concurrence searches running, how many reports, how much data load all will come into picture. As general recommendation is below one Master distributed the data. 4xquad-core CPUsat2.0 GHz/12Gb ~ 16GB RAM/ 2x10kRPMlocal SAS drives in RAID 1 To test your load please make sure your run Bonee++ utility on linux servers. ... View more

first check from your network and firewall team to make sure splunk ports and firewall rules are enabled for splunk. some times this error messages point to SSL enable also on the indexer. Don't forget to add the SSL stanza in the inputs.conf on the indexer side. ... View more

couple of things you can try, after seeing that message it sounds like parsing Que issue. you can try increase maxQueueSize = 200MB in outputs.conf. some times the maxQueueSize setting causes events to be loaded into a queue in memory so Before increase check your memory and performance using top what ever OS command. Note: check out metrics.log and parsingQueue or TCPque is full etc errors ... View more

Thanks sounds like some direction to work. but i never worked on tokens, do we have any splunk app see here is my headers for those sheets. Arrival city Country Email First name From date Last name Mobile To date ... View more

Few things can you verify it from UF to Indexer server. Check the port connections as well firewall rules most of the time TCP error/Cooked data refer to these kind of issues. Telnet to port and IP. etc Try to troubleshoot problem from Network layer to Splunk you will find it. ... View more

Its depend on your server hardware model and type, Its all depend on how much memory you have on server how many users are running searches etc.Lets you have 8 core 24GRam right for 40 users one search head is enough. max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches To fix the problem more generally, you can tweak some configuration nobs in limits.conf as follows: a) max_searches_per_cpu While increasing this could fix the dashboard issue where searches are fairly cheap to run, this could lead to performance degradation if you've scheduled a large number of expensive searches. b) dispatch_quota_retry This is the number of retries the back end will attempt before throwing the quota/limit error. The back end here does an exponential back-off starting with 100ms and doubling that every time it retries. c) dispatch_quota_sleep_ms The initial sleep time for retries. Instead of increasing max_searches_per_cpu you can set the dispatch_quota_retry to 10 which will instruct the back-end to retry dispatching a particular search for about 100 seconds before throwing the quota/limit error. ... View more

Thanks for your help i have checked that already. im running with spluk 6.1 verison ... View more

This is not working for me |metadata type=hosts | table host lastTime | where lastTimerelative_time(now(),"-30d@d") | convert timeformat="%Y%m%d" ctime(lastTime) AS LastTime ... View more

check this url, i had similar issues then i found version mismatch on SH peers. http://answers.splunk.com/answers/192592/search-process-did-not-exit-cleanly-exit-code255-d.html#answer-195456 ... View more

Few places you can check. Search peers are running with inconsistency version. Check search peers status . ask user and that job/lookup table not had right permissions or not exist all over the places. ... View more

tried too not working [monitor://D:\Debug\Log\Forms\Rollingfile*] recursive = true crcSalt = sourcetype = log4net index = forms_int ... View more

Thanks i was thinking same way ... View more

I have similar issues can some one tell me what was the fix for this ... View more

i just found my issues why this is hitting again On indexer servers on problem server we have 5.x version splunk running on rest of the server as well SH v4 version running , im assuming on that V4 version on SH not able to copy the Knowledge objects on proper way that is the reason that is giving issues.It make sense to me. http://docs.splunk.com/Documentation/Splunk/5.0.3/Deploy/Whatisdistributedsearch What search heads send to search peers When initiating a distributed search, the search head replicates and distributes its knowledge objects to its search peers. Knowledge objects include saved searches, event types, and other entities used in searching across indexes. The search head needs to distribute this material to its search peers so that they can properly execute queries on its behalf. The set of data that the search head distributes is called the knowledge bundle. The indexers use the search head's knowledge bundle to execute queries on its behalf. When executing a distributed search, the indexers are ignorant of any local knowledge objects. They have access only to the objects in the search head's knowledge bundle. The process of distributing knowledge bundles means that indexers by default receive nearly the entire contents of all the search head's apps. If an app contains large binaries that do not need to be shared with the indexers, you can reduce the size of the bundle by means of the [replicationWhitelist] or [replicationBlacklist] stanza in distsearch.conf. See "Limit knowledge bundle size" in this manual. The knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers/ directory on each search peer. Because the search head distributes its knowledge, search scripts should not hardcode paths to resources. The knowledge bundle will reside at a different location on the search peer's file system, so hardcoded paths will not work properly. By default, the search head replicates and distributes the knowledge bundle to each search peer. For greater efficiency, you can instead tell the search peers to mount the knowledge bundle's directory location, eliminating the need for bundle replication. When you mount a knowledge bundle, it's referred to as a mounted bundle. To learn how to mount bundles, read "Mount the knowledge bundle". ... View more

No luck some times it is working some times it is not. ... View more

thanks but i have multiple files with same format like 2014-11-23 ----2013-10-12 etc * is not working ... View more