About doctorjw

doctorjw · ‎06-25-2021

We are using splunk cloud 8.2.2105.2 (Build: 164754a2784c). When we came into work this morning, we found a number of our dashboards are suddenly not rendering correctly, and we're at a loss to explain why. It would see some centering is no longer happening correctly, and font sizing has changed. You can see this through a very simple dashboard that I put together: <dashboard hideChrome="true" hideFilters="true" theme="light"> <label>Basic Dashboard</label> <row> <panel> <single> <search> <query> | makeresults count=1 | streamstats count | eval msg = 5 | stats count</query> </search> <option name="underLabel">A field we should see</option> <option name="height">50</option> </single> </panel> </row> </dashboard> When this is rendered, I get: You can see how the text ("1") is large, and cut off, and the label ("A field we should see") does not show. Whatever is going on has affected a number of our production dashboards; again, things were fine last night but this morning .... the rendering is bad. We have not had to mess with style sheets in the past to get the rendering correct, and don't think we should have to here ... What changed, and how can we get our dashboards rendering properly again? Help? Thx john

doctorjw · ‎09-02-2020

Okay, yes, this is solved. Using a yaml file to put the report in place, our default value of 5m for the allow_skew comes into play, regardless of what I set in my yaml file. I haven't found a way to reset that value this way; I can manually edit a report outside of our report area to set the allow_skew, though. To get things to work, I just need to be much more particular about the cron, and use a cron expression outside of what allow_skew will adjust ... and that's fine.

doctorjw · ‎09-01-2020

Well ... I take this back. When I do a manual edit of my scheduled job and set allow_skew to 0, I get what I expect. However, when our pipeline deploys a report (so I can't edit it directly) but my yaml file has allow_skew 0 .. I'm not getting the changes I expect. I feel I'm closer but ... not there yet.

doctorjw · ‎09-01-2020

This does look like the issue; we have a default allow_skew of 5m for some reason. Testing with an allow_skew of 0 fixes this, and ... my data lines up with what I expected to have.

doctorjw · ‎09-01-2020

Could the answer be that the saved search has an "allow_skew" of 5m? 8^/ dr_j

doctorjw · ‎09-01-2020

Okay, running this query: index=_internal sourcetype=scheduler savedsearch_name=dip-summary-revenue-by-minute status=success I get this result: 9/1/20 9:24:24.796 AM 09-01-2020 14:24:24.796 +0000 INFO SavedSplunker - savedsearch_id="nobody;dip-summary-revenue-by-minute", search_type="scheduled", user="admin", app="myapp", savedsearch_name="dip-summary-revenue-by-minute", priority=default, status=success, digest_mode=1, scheduled_time=1598970000, window_time=-1, dispatch_time=1598970264, run_time=0.497, result_count=5, alert_actions="", sid="scheduler__admin__myapp__RMD5a5372a05036e6d57_at_1598970000_39293", suppressed=0, thread_id="AlertNotifierWorker-0", workload_pool="" When I run my query to get the latest data right after running the above query: index=prod-summary-revenue source="summary_revenue_by_minute" | convert ctime(info_min_time) AS minTime | convert ctime(info_max_time) AS maxTime | convert ctime(info_search_time) AS searchTime | sort by -_time | table _time minTime maxTime searchTime _raw I get these results: 2020-09-01 09:18:00 09/01/2020 09:14:00.000 09/01/2020 09:19:00.000 09/01/2020 09:24:24.104 09/01/2020 09:18:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=171, revenueByMinute="909.96" 2020-09-01 09:17:00 09/01/2020 09:14:00.000 09/01/2020 09:19:00.000 09/01/2020 09:24:24.104 09/01/2020 09:17:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=167, revenueByMinute="882.49" 2020-09-01 09:16:00 09/01/2020 09:14:00.000 09/01/2020 09:19:00.000 09/01/2020 09:24:24.104 09/01/2020 09:16:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=150, revenueByMinute="830.47" 2020-09-01 09:15:00 09/01/2020 09:14:00.000 09/01/2020 09:19:00.000 09/01/2020 09:24:24.104 09/01/2020 09:15:00 -0500, search_name="dip-summary-revenue-by-minute", search_now=1598970000.000, info_min_time=1598969640.000, info_max_time=1598969940.000, info_search_time=1598970264.104, numOrdersByMinute=122, revenueByMinute="606.37" Confirming that instead of the generated data being in the 1-6 minute old range, it's 6-11 minutes old (offset 5 minutes in the past from what we expected). btw, not sure it's that important to mention, but this is with splunk cloud. Dr_j

doctorjw · ‎08-31-2020

I have a report scheduled to run every 5 minutes (*/5 .....). This report gathers summary data from 6 minutes ago, to 1 minute ago, like this (I've removed the index & search criteria, etc., as they aren't germane): index=<index> <search_criteria> earliest=-6m@m latest=-1m@m .... When I run this with a "collect' with testmode=true, I get exactly what I'd expect ... If the current time is 18:00, I get data for 17:54, 17:55, 17:56, 17:57, and 17:58 The same query run as a report, though, with testmode=false, apparently the offset times are changed by 5 minutes and I get data for 17:49, 17:50, 17:51, 17:52, and 17:53 Is there something in the configuration that would have report offsets be altered by 5 minutes? Something I'm missing dealing with reports somehow? Running interactively vs running in a report is clearly offsetting things by 5 minutes. I've used this query to verify the times in my report query: index=<summary index> source="summary_revenue_by_minute" | convert ctime(info_min_time) AS minTime | convert ctime(info_max_time) AS maxTime | convert ctime(info_search_time) AS searchTime | sort by -_time | table _time minTime maxTime searchTime _raw I'm scratching my head & reading docs to no avail. Thoughts?

doctorjw · ‎08-03-2015

We're facing this same situation. A simple query basically doing just a "select 1" is taking 8 seconds (on rare occasions it will return in 4). Our more complex query will also take 8 seconds. When running the real query via MS Sql Server Studio, the query itself returns in sub-second time. The query is pretty basic --- selecting maybe a half dozen or so fields from the join of a few tables across user ids. The tables themselves are not very large --- and as evidenced by the sql server took, the query itself runs pretty quickly. I've inspected the splunk job, and dbquery is listed as taking ... you guessed it ... roughly 8 seconds. No further detail to indicate where the holdup is. I, too, doubt this is some environmental factor. We looked into connection pooling, and it does look like a pool is configured and is being used, so it doesn't seem to be from connecting/disconnecting from the database. Thoughts? Ideas? Currently our simple dashboard with 3 of these queries takes what seems to a user to be a very long time.

Posts	8
Solutions	0
Karma Given	0
Karma Received	0
Member Since	‎11-30-2012

Online Status	Offline
Date Last Visited	‎06-25-2021 03:46 PM

Rendering change overnight .... what happened?

Report time offsets are off by 5 minutes?

Rendering change overnight .... what happened?

Re: Report time offsets are off by 5 minutes?

Re: Report time offsets are off by 5 minutes?

Re: Report time offsets are off by 5 minutes?

Re: Report time offsets are off by 5 minutes?

Re: Report time offsets are off by 5 minutes?

Report time offsets are off by 5 minutes?

Re: Splunk DB Connect 2: Is there a known search p...