Dashboards & Visualizations

Real-time dashboard rtsearch connection terminated

markrobinson734
Explorer

version 4.2.3

Once or twice in a 24hr period we get a gray notification on the Splunk dashboard with regards to rtsearch timing out or being terminated.

rt_admin_admin_search_Q0hBUlQgSUlTIFdlYnNoaXRzIGJ5IEhUVFAgc3RhdHVz_rt_1319177401.105

This causes the real time dashboard to stop working. In our NOC this isn't very useful.

We run the Splunk dash in a Chrome browser which connects over a VPN to our DC. Could this be a connection fault causing these breaks? No other monitoring tools have the same issue.

Also, the dashboard is running several searches in realtime, saving 1hr of historic data each.

Here are our Splunkd.log for the time of which the last RTSEARCH CONNECTION TERMINATED in the dashboard.

Please let me know if any other information is required. It has been so far the only issue we have had with Splunk.

10-21-2011 09:05:23.228 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND blocked index::main source::d:\\apps\\webknight\\* ]', active_streams = 8

10-21-2011 09:05:26.582 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 7

10-21-2011 09:05:29.422 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 6

10-21-2011 09:22:25.131 +0000 INFO  WatchedFile - Will begin reading at offset=24999901 for file='D:\APPS\Splunk\var\log\splunk\audit.log.1'.

10-21-2011 09:22:26.379 +0000 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\APPS\Splunk\var\log\splunk\audit.log'.

10-21-2011 09:22:26.379 +0000 INFO  WatchedFile - Will begin reading at offset=0 for file='D:\APPS\Splunk\var\log\splunk\audit.log'.

10-21-2011 09:43:01.768 +0000 WARN  DateParserVerbose - Failed to parse timestamp for event.  

Context="source::D:\IISWEB\Logs\Logfiles\W3SVC4\u_ex111021.log|host::CODSCL01|iisw3c|remoteport::22081" Text="#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-usern..."
10-21-2011 10:05:36.802 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 5

10-21-2011 10:05:36.802 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 4

10-21-2011 10:05:37.738 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND 40* index::main sourcetype::iisw3c ]', active_streams = 3

10-21-2011 10:05:37.816 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 2

10-21-2011 10:05:40.109 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::webknight ]', active_streams = 1

10-21-2011 10:05:46.599 +0000 INFO  IndexProcessor - rtsearch connection terminated, filter = '[ AND 50* index::main sourcetype::iisw3c ]', active_streams = 0

10-21-2011 10:08:48.648 +0000 WARN  DateParserVerbose - Failed to parse timestamp for event.  Context="source::D:\IISWEB\Logs\Logfiles\W3SVC4\u_ex111021.log|host::CODSCL02|iisw3c|remoteport::22209" 

markrobinson734
Explorer

This is what generates the Pie Chart.

sourcetype="iisw3c" | geoip c_ip | dedup c_ip | top c_ip_country_name

0 Karma

markrobinson734
Explorer

Splunkweb errors.

06:55:56.127 2011-10-21 07:55:56,127 ERROR [4ea1258c1c8c46908] utility:59 - name=javascript, class=Splunk.Error, lineNumber=3958, message=getConfigValue - SERVER_ZONEINFO not set, no default provided, fileName=http://x/en-GB/static/@105575/js/common.min.js

21/10/2011 06:54:25.969 2011-10-21 07:54:25,969 ERROR [4ea12531f889242e8] utility:59 - name=javascript, class=Splunk.Error, lineNumber=884, message=Unspecified error., fileName=http://x/en-GB/static/@105575/js/common.min.js

20/10/2011 16:29:04.105 2011-10-20 17:29:04,105 ERROR [4ea05a5fc77886cc0] admin:944 - uiHelper processValueEdit operator failed for endpoint_path=saved/searches/PieChart IP's by Geographical Location elementName=spl-ctrl_summary_index: argument of type 'NoneType' is not iterable

20/10/2011 16:29:04.105 2011-10-20 17:29:04,105 ERROR [4ea05a5fc77886cc0] admin:944 - uiHelper processValueEdit operator failed for endpoint_path=saved/searches/PieChart IP's by Geographical Location elementName=spl-ctrl_script_enable: argument of type 'NoneType' is not iterable

20/10/2011 16:29:04.105 2011-10-20 17:29:04,105 ERROR [4ea05a5fc77886cc0] admin:944 - uiHelper processValueEdit operator failed for endpoint_path=saved/searches/PieChart IP's by Geographical Location elementName=spl-ctrl_rss_enable: argument of type 'NoneType' is not iterable

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...