All Apps and Add-ons

Help optimizing an advanced Splunk DB Connect search

griffinpair
Path Finder

The query below takes approximately 20 minutes to run and I need help optimizing it. The point of the query is to gather the number of problematic data conditions from each client. The conditions are:
1. Where PROCESSEDFLAG='N' (from the VA table)
2. Where WORKFLOWSTATUS is null (from the CA table)
3. Where PROCESSEDFLAG='N' (from the VS table).

Our current setup has each client having their own database and clients split throughout servers. Given our setup, this is why I have formatted the search to have a different sub-search for each problematic condition for each client. What solutions are there to fix the current speed of the search or optimize the current query?

|dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID1\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-126" | stats count as ALPVCA 
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID2\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-126" | stats count as ARTVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID3\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-126" | stats count as CEGVCA] 
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID4\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-126" | stats count as CFUSVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID5\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-116" | stats count as IRMVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID6\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-116" | stats count as MARVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID7\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-116" | stats count as USBIVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID8\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-116" | stats count as WHIVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID9\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-124" | stats count as WOSVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID10\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-124" | stats count as POWVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID11\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-124" | stats count as NABVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID12\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-125" | stats count as NDQVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID13\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-127" | stats count as WEDVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID14\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-127" | stats count as HLMVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID15\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-127" | stats count as ICBCVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID16\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-128" | stats count as CFUKVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID17\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-128" | stats count as MEDVCA]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID18\".\"dbo\".\"VA\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-128" | stats count as SPIVCA]

| appendcols [dbxquery query="SELECT ClientID FROM \"clientID1\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-126" | stats count as ALPWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID2\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-126" | stats count as ARTWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID3\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-126" | stats count as CEGWS] 
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID4\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-126" | stats count as CFUSWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID5\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-116" | stats count as IRMWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID6\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null  connection="connection-116" | stats count as MARWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID7\".\"dbo\".\"CA\"WHERE WORKFLOWSTATUS = Null connection="connection-116" | stats count as USBIWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID8\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null  connection="connection-116" | stats count as WHIWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID9\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null  connection="connection-124" | stats count as WOSWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID10\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null  connection="connection-124" | stats count as POWWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID11\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null  connection="connection-124" | stats count as NABWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID12\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null  connection="connection-125" | stats count as NDQWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID13\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null  connection="connection-127" | stats count as WEDWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID14\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-127" | stats count as HLMWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID15\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null  connection="connection-127" | stats count as ICBCWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID16\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-128" | stats count as CFUKWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID17\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-128" | stats count as MEDWS]
| appendcols [dbxquery query="SELECT ClientID FROM \"clientID18\".\"dbo\".\"CA\" WHERE WORKFLOWSTATUS = Null connection="connection-128" | stats count as SPIWS]

| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID5\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-116" | stats count as IRMVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID6\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-116" | stats count as MARVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID7\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-116" | stats count as USBIVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID8\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-116" | stats count as WHIVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID9\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-124" | stats count as WOSVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID10\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-124" | stats count as POWVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID11\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-124" | stats count as NABVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID12\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-125" | stats count as NDQVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID13\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-127" | stats count as WEDVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID14\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-127" | stats count as HLMVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID15\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-127" | stats count as ICBCVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID16\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-128" | stats count as CFUKVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID17\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-128" | stats count as MEDVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID18\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-128" | stats count as SPIVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID1\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-126" | stats count as ALPVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID2\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-126" | stats count as ARTVSEC] 
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID3\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) "  connection="connection-126" | stats count as CEGVSEC]
| appendcols [dbxquery query="SELECT PROCESSEDFLAG FROM \"clientID4\".\"dbo\".\"VS\" WHERE PROCESSEDFLAG = 'N' and ADDEDDT >DATEADD(hour, -24, GETDATE()) " connection="connection-126" | stats count as CFUSVSEC]  

| eval Totals= ARTVCA+CEGVCA+CFUSVCA+IRMVCA+MARVCA+USBIVCA+WHIVCA+WOSVCA+POWVCA+NDQVCA+WEDVCA+HLMVCA+ICBCVCA+CFUKVCA+MEDVCA+SPIVCA+ARTVSEC+CEGVSEC+CFUSVSEC+IRMVSEC+MARVSEC+USBIVSEC+WHIVSEC+WOSVSEC
+POWVSEC+NDQVSEC+WEDVSEC+HLMVSEC+ICBCVSEC+CFUKVSEC+MEDVSEC+SPIVSEC+ARTWS+CEGWS+CFUSWS+IRMWS+MARWS+USBIWS+WHIWS+WOSWS+POWWS+NDQWS+WEDWS+HLMWS+ICBCWS+CFUKWS+MEDWS+SPIWS
| table Totals
0 Karma

david_muegge
New Member

Have you tried to use SQL unions and construct a single SQL query? The data would then be assembled at the DB server. This would reduce the network round trips drastically. Just a thought.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...