When I search for index=summary in search head, the result only shows one of the server in splunk_server field. But I have 3 indexers. Shouldn't I see all the indxers name in the splunk_server field? ... View more

How to disable this message on splunk 4.3? https://quickdraw.splunk.com/images/banner_4_3_free_trial.gif ... View more

Apr 25 17:13:28 www2 sshd[27718]: [ID 800047 auth.debug] debug1: no match: WinSCP_release_4.3.2 [..within 5 secs..] Apr 25 17:13:29 www2 sshd[27718]: [ID 800047 auth.info] Failed none for john from 10.2.43.186 port 1358 ssh2 So if winscp is below 4.0.4 get an alert saying john is using older release. Because that is sshd2 process 27718 belongs to john. How do I correlate between two events, 5+ secs apart, based on the process id and then generate an appropriate alert if a the number portion of the string is below 4.0.4? Alert will always go to sysadmin@example.com. ... View more

root@splunk:/usr/src# dpkg -i splunk-4.2-96430-linux-2.6-amd64.deb Selecting previously deselected package splunk. (Reading database ... 48824 files and directories currently installed.) Unpacking splunk (from splunk-4.2-96430-linux-2.6-amd64.deb) ... Setting up splunk (4.2-96430) ... ---------------------------------------------------------------------- Splunk has been installed in: /opt/splunk To start Splunk, run the command: /opt/splunk/bin/splunk start To use the Splunk Web interface, point your browser at: http://apa-splunk-04:8000 Complete documentation is at http://www.splunk.com/r/docs root@apa-splunk-04:/usr/src# /opt/splunk/bin/splunk start SPLUNK SOFTWARE LICENSE AGREEMENT [..] "Splunk Developer API" means the documentation and functionality enabling Do you agree with this license? [y/n]: y This appears to be your first time running this version of Splunk. Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'. Generating RSA private key, 1024 bit long modulus [..] The Splunk web interface is at http://splunk:8000 root@splunk:/usr/src# /opt/splunk/bin/splunk enable boot-start -user splunk bash: splunk: command not found First-time-run has not finished. Ignore this error when previewing migration - exiting. Any idea why it is failing? It is also giving wrong error message "bash: splunk: command not found" root@splunk:/usr/src# /opt/splunk/bin/splunk --version Splunk 4.2 (build 96430) root@splunk:/usr/src# getent passwd splunk splunk:x:1001:1001:Splunk Server:/opt/splunk:/bin/bash ... View more

According to my Deployment monitor app one of my indexer shows backed up. I need help find out if it is some thing due to slow disk or some complex regex. I am providing the following logs as evidence to my issue indexer's splunkd.log 11-16-2010 17:23:14.625 INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying... 11-16-2010 17:23:39.674 INFO TailingProcessor - ...continuing. 11-16-2010 17:23:43.608 WARN DateParserVerbose - Accepted time (Fri Oct 22 05:29:30 2010) is suspiciously far away from the previous event's time (Tue Nov 16 17:11:52 2010), but still accepted because it was extracted by the same pattern. Context="source::/var/genesys/GVHC/GVHC_Stat_Server2.log.20101116_163632_193.log|host::tuk1cc-g2|genesys_statserver_log|remoteport::37189" 1041 similar messages suppressed. First occurred at: Tue Nov 16 17:18:38 2010 11-16-2010 17:23:43.608 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::/var/genesys/QwestHSI/QwestHSI_UR_Server2.log.20101116_045930_983.log|host::cer1cc-g2|genesys_urserver_log|remoteport::47624" Text=" AttributeCustomerID 'QwestHSI'..." 25777 similar messages suppressed. First occurred at: Tue Nov 16 17:18:38 2010 11-16-2010 17:23:43.608 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::/var/genesys/QwestHSI/QwestHSI_UR_Server2.log.20101116_045930_983.log|host::cer1cc-g2|genesys_urserver_log|remoteport::47624" Text=" AttributeANI '606837491'..." 11-16-2010 17:23:43.608 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::/var/genesys/QwestHSI/QwestHSI_UR_Server2.log.20101116_045930_983.log|host::cer1cc-g2|genesys_urserver_log|remoteport::47624" Text=" AttributeDNIS '8665313546'..." indexer's metrics.log 11-16-2010 17:26:41.086 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size=1000, filled_count=11, empty_count=7307, current_size=1000, largest_size=1000, smallest_size=1 11-16-2010 17:26:41.086 INFO Metrics - group=queue, name=typingqueue, blocked=true, max_size=1000, filled_count=4, empty_count=15508, current_size=1000, largest_size=1000, smallest_size=1 11-16-2010 17:27:35.067 INFO Metrics - group=queue, name=aggqueue, blocked=true, max_size=1000, filled_count=1, empty_count=0, current_size=1000, largest_size=1000, smallest_size=817 11-16-2010 17:27:35.067 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size=1000, filled_count=0, empty_count=0, current_size=1000, largest_size=0, smallest_size=1000 11-16-2010 17:27:35.067 INFO Metrics - group=queue, name=typingqueue, blocked=true, max_size=1000, filled_count=0, empty_count=0, current_size=1000, largest_size=0, smallest_size=1000 forwarder's metrics.log 11-16-2010 17:27:16.324 INFO Metrics - group=queue, name=typingqueue, blocked=true, max_size=1000, filled_count=4, empty_count=1132, current_size=1000, largest_size=1000, smallest_size=1 11-16-2010 17:27:16.324 INFO Metrics - group=tcpout_connections, apa-splunk, blocked=true, current_entries_count=1000, queue_size=1000 11-16-2010 17:28:18.903 INFO Metrics - group=queue, name=aggqueue, blocked=true, max_size=1000, filled_count=0, empty_count=0, current_size=1000, largest_size=0, smallest_size=1000 11-16-2010 17:28:18.903 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size=1000, filled_count=0, empty_count=0, current_size=1000, largest_size=0, smallest_size=1000 11-16-2010 17:28:18.904 INFO Metrics - group=queue, name=parsingqueue, blocked=true, max_size=1000, filled_count=0, empty_count=0, current_size=1000, largest_size=0, smallest_size=1000 11-16-2010 17:28:18.904 INFO Metrics - group=queue, name=tcpout_apa-splunk, blocked=true, max_size=1000, filled_count=0, empty_count=0, current_size=1000, largest_size=0, smallest_size=1000 11-16-2010 17:28:18.904 INFO Metrics - group=queue, name=typingqueue, blocked=true, max_size=1000, filled_count=0, empty_count=0, current_size=1000, largest_size=0, smallest_size=1000 11-16-2010 17:28:18.904 INFO Metrics - group=tcpout_connections, apa-splunk, blocked=true, current_entries_count=1000, queue_size=1000 Here is an snapshot of the DM status http://picpaste.com/pics/indexer-ObvwPnn9.1289928881.png Here is little bit more details of the queue on the indexer http://picpaste.com/pics/splunk-1vsSa4RV.1289929280.png ... View more

11-09-2010 00:00:57.985 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/logs/current/cisco-cc'. 11-09-2010 00:01:01.012 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/logs/current/cisco-cc'. 11-09-2010 00:01:01.015 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/logs/current/cisco-cc'. 11-09-2010 00:01:04.248 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 00:01:28.441 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 00:03:13.104 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 00:04:37.773 INFO LicenseManager - Recalculating persistent statistics 11-09-2010 00:13:04.069 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 18:20:36.170 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 18:20:36.429 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 18:20:36.433 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/splunk/metrics.log'. 11-09-2010 18:20:36.521 INFO WatchedFile - Will begin reading at offset=24999692 for file='/opt/splunk/var/log/splunk/metrics.log.1'. 11-09-2010 19:29:32.445 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 23:59:07.098 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 23:59:07.119 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/var/logs/current/shasta'. 11-09-2010 23:59:07.119 INFO WatchedFile - Will begin reading at offset=0 for file='/var/logs/current/shasta'. 11-09-2010 23:59:31.663 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-09-2010 23:59:31.687 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/var/logs/current/cisco'. 11-09-2010 23:59:31.687 INFO WatchedFile - Will begin reading at offset=0 for file='/var/logs/current/cisco'. 11-10-2010 00:00:08.491 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/logs/current/cisco-qmoe'. 11-10-2010 00:00:26.188 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-10-2010 00:00:26.227 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/var/logs/current/juniper'. 11-10-2010 00:00:26.227 INFO WatchedFile - Will begin reading at offset=0 for file='/var/logs/current/juniper'. 11-10-2010 00:00:55.118 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/logs/current/arbor'. 11-10-2010 00:00:57.158 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/logs/current/dial'. 11-10-2010 00:00:57.168 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/logs/current/alcatel'. 11-10-2010 00:01:00.167 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/var/logs/current/dial'. 11-10-2010 00:01:04.414 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-10-2010 00:01:09.452 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-10-2010 00:03:19.371 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. 11-10-2010 00:03:30.937 INFO LicenseManager - Recalculating persistent statistics 11-10-2010 00:05:01.064 ERROR DatabaseDirectoryManager::Bucket - Unable to find a directory for id = 734 when attempting to validate a sentinel. I am seeing lots of these error messages. What does it mean? Something I need to worry about? ... View more

* | rex "(?<fpc>fpc\d+) (?<ichip>ICHIP\(\d+\)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?<err>\d+)" How do I get the max(err) and min(err) for each combination of host, fpc, ichip ? err is an integer number. ... View more

I have a router with multiple FPCs and each FPC has multiple ICHIPs. An ICHIP can produce pktwr drops and that number of drops is logged periodically. So, within a span of time, say 720 minutes, I want to pick out the minimum number of drops (per host per FPC per ICHIP). This would be the start value for that ICHIP. Also, I want to pick out the maximum number of drops; this is the finish value. Then I want to subtract the minimum value from the maximum value to find the total. Then I want to take that total and divide by 720 minutes to get a rough drop rate. Then I want to sort the results by the highest values first. The raw logs look like this: Aug 11 19:01:24 a-priv-03.example.net fpc2 ICHIP(0):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 6477315 Aug 11 19:03:20 c-priv-01.example.net fpc1 ICHIP(3):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 56030 Aug 11 19:11:07 t-priv-03.example.net fpc2 ICHIP(0):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 6477325 Aug 11 19:20:50 c-priv-03.example.net fpc1 ICHIP(3):Packet drop in Ichip pktwr,rate: %PFE-3: 1, total: 56130 I had put this together and used some eval and min and max statements to produce some undesirable results: source="/var/logs/current/juniper" Packet drop in Ichip pktwr | rex "(?<fpc>fpc\d+) (?<ichip>ICHIP\(\d+\)):Packet drop in Ichip pktwr,rate: %\S+: \d+, total: (?<err>\d+)" | timechart span=12h eval((max(err) - min(err)) / 720) by host How do I sort by multiple fields? I am not sure if my logic is correct in the search above. I want the results to look like this: host fpc ichip pktwr_drops rate c-priv-03 1 3 100 .14 t-priv-03 2 0 10 .01 ... View more