The actual errors are as following 2020-08-10 16:04:00,520 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Closing tlsio from a state other than TLSIO_STATE_EXT_OPEN or TLSIO_STATE_EXT_ERROR' 2020-08-10 16:04:00,519 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'wait_for_connection failed' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/socketio_berkeley.c':'socketio_open':770) 2020-08-10 16:04:00,519 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Failure: select failure.' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/socketio_berkeley.c':'wait_for_connection':370) 2020-08-10 16:03:50,453 INFO pid=19259 tid=MainThread file=connection.py:_state_changed:177 | Connection '88ceb760-92f6-4ee2-92d7-3013adb58efa' state changed from <ConnectionState.START: 0> to <ConnectionState.START: 0> 2020-08-10 16:03:50,452 INFO pid=19259 tid=MainThread file=connection.py:_close:136 | Connection shutdown complete '88bfa644-3d73-42f5-98af-bc38aa8173d7'. 2020-08-10 16:03:50,452 INFO pid=19259 tid=MainThread file=connection.py:_close:129 | Shutting down connection '88bfa644-3d73-42f5-98af-bc38aa8173d7'. 2020-08-10 16:03:50,452 INFO pid=19259 tid=MainThread file=error.py:_handle_exception:233 | u'eventhub.pysdk-c7722623' has an exception (AMQPConnectionError('Unable to open management session. Please confirm URI namespace exists.',)). Retrying... 2020-08-10 16:03:48,850 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Failed opening message receiver' ('/data/src/vendor/azure-uamqp-c/src/amqp_management.c':'amqp_management_open_async':981) 2020-08-10 16:03:48,850 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Link attach failed' ('/data/src/vendor/azure-uamqp-c/src/message_receiver.c':'messagereceiver_open':362) 2020-08-10 16:03:48,850 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Begin session failed' ('/data/src/vendor/azure-uamqp-c/src/link.c':'link_attach':1154) 2020-08-10 16:03:48,849 INFO pid=19259 tid=MainThread file=connection.py:_state_changed:181 | Connection with ID '88bfa644-3d73-42f5-98af-bc38aa8173d7' unexpectedly in an error state. Closing: False, Error: None 2020-08-10 16:03:48,849 INFO pid=19259 tid=MainThread file=connection.py:_state_changed:177 | Connection '88bfa644-3d73-42f5-98af-bc38aa8173d7' state changed from <ConnectionState.START: 0> to <ConnectionState.END: 13> 2020-08-10 16:03:48,849 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Opening the underlying IO failed' ('/data/src/vendor/azure-uamqp-c/src/connection.c':'connection_open':1344) 2020-08-10 16:03:48,849 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'xio_open failed' ('/data/src/vendor/azure-uamqp-c/src/saslclientio.c':'saslclientio_open_async':1097) 2020-08-10 16:03:48,849 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Failed opening the underlying I/O.' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c':'tlsio_openssl_open':1258) 2020-08-10 16:03:48,849 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Invalid tlsio_state. Expected state is TLSIO_STATE_OPENING_UNDERLYING_IO.' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/tlsio_openssl.c':'on_underlying_io_open_complete':760) 2020-08-10 16:03:48,848 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Closing tlsio from a state other than TLSIO_STATE_EXT_OPEN or TLSIO_STATE_EXT_ERROR' 2020-08-10 16:03:48,848 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'wait_for_connection failed' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/socketio_berkeley.c':'socketio_open':770) 2020-08-10 16:03:48,848 INFO pid=19259 tid=MainThread file=mgmt_operation.py:__init__:65 | 'Failure: select failure.' ('/data/src/vendor/azure-uamqp-c/deps/azure-c-shared-utility/adapters/socketio_berkeley.c':'wait_for_connection':370) 2020-08-10 16:03:38,781 INFO pid=19259 tid=MainThread file=connection.py:_state_changed:177 | Connection '88bfa644-3d73-42f5-98af-bc38aa8173d7' state changed from <ConnectionState.START: 0> to <ConnectionState.START: 0> I have make sure the ports are opened,still no luck. I have one question Is event hub has to be separate for each resource group, we have created a event hub name space for one resource group and enabled the WAF logs for another resource group . Is that the problem here. ... View more

Hi @jconger  I am still not able to get the success even after i follow all your troubleshooting steps, getting the following errors in splunk when i check with the different keywords. 08-10-2020 09:12:37.876 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" response = self._management_request(mgmt_msg, op_type=b'com.microsoft:eventhub') 08-10-2020 09:12:37.905 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" Unable to open management session. Please confirm URI namespace exists. 08-10-2020 09:12:37.876 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" raise error 8-10-2020 09:12:37.876 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" File "/opt/splunk/etc/apps/TA-MS-AAD/bin/azure/eventhub/error.py", line 196, in _handle_exception 08-10-2020 09:12:37.876 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" _handle_exception(exception, retry_count, max_retries, self) Could you please assist as this app is not supported by splunk. Thank you in advance. ... View more

Even i am facing the same issue when i am trying to establish a connection between MSSQL database and splunk HF. can anyone help ... View more

attaching image for your reference ... View more

@jkat54 i have tried the followin in props.conf and try create fields using that [pench_test] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true SEDCMD = y/:/=/ category = Custom pulldown_type = true sedcmd = y/:/=/ But its not creating most of the necessery fields to write the SPL query and seeking help. ... View more

@jkat54 @woodcock @sideview @vnravikumar can you please help ... View more

yes i did, the following is the full XML code for your reference. Autosys Cluster Autosys Jobs Running and started This DashBoard gives the list of autosys jobs running or started in each 1CDH cluster at times. <unset token="earliest"></unset> <unset token="latest"></unset> <input type="time" token="field1"> <label>Select Time range</label> <default> <earliest>-24h@h</earliest> <latest>now</latest> </default> </input> <input type="dropdown" token="appid"> <label>Select the APPID</label> <choice value="*">All</choice> <fieldForLabel>AppID</fieldForLabel> <fieldForValue>AppID</fieldForValue> <search> <query>|inputlookup appidvscluster.csv |dedup AppID |table AppID</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> </input> <input type="dropdown" token="clustername"> <label>Select the Cluster</label> <fieldForLabel>Cluster</fieldForLabel> <fieldForValue>Cluster</fieldForValue> <search> <query>|inputlookup appidvscluster.csv where AppID ="$appid$" |dedup Cluster |table Cluster -24h@h now All Select Host All host host |inputlookup "datalakenodeslist.csv" where cluster=$clustername$ | dedup "host" | table host -24h@h now Select Job <choice value="">All * Job Job |inputlookup alljobs.csv |rex mode=sed field=host "y/\w{5}\d{2}v\d{4}/\w{5}\d{2}a\d{4}/" | lookup datalakenodeslist.csv host OUTPUT cluster | search cluster="$clustername$" AND host="$host$" | dedup Job | table Job -24h@h now Select Job status Starting Running RUNNING <panel> <table> <search> <query>index=infra_apps sourcetype=ca:atsys:edemon:txt EventCode=40245 Status=$status$ AppID=$appid$ Machine=$host$ Job=$job$ | dedup _raw | lookup datalakenodeslist.csv host as Machine OUTPUT cluster | search cluster=$clustername$ | timechart count span=2m <earliest>-24h@h</earliest> <latest>now</latest> </search> <option name="drilldown">none</option> </table> </panel> <panel> <title>$jobstatus$ Jobs Trend</title> <chart> <search id="CDH_Cluster_jobs_trend"> <query>index=infra_apps sourcetype=ca:atsys:edemon:txt EventCode=40245 AND (Status=Failure OR Status=Terminated OR Status=Running OR Status=Success) AppID=$appid$ Machine=$host$ Job=$job$ | dedup _raw | lookup datalakenodeslist.csv host as Machine OUTPUT cluster | search cluster=$clustername$ | eval running=if(Status="RUNNING","1","0"),status=if(Status="RUNNING","start","stop"), time=_time+status | bin span=2m _time | stats max(running) as running, earliest(time) as first, latest(time) as last by Job,_time | xyseries _time Job running first last | makecontinuous span=2m _time | streamstats window=2 global=f earliest(last*) as last* | reverse | streamstats window=2 global=f earliest(first*) as first* | reverse | foreach running* [ eval <<FIELD>>=if(isnull('<<FIELD>>') AND like('first<<MATCHSTR>>',"%start"),"0",if(isnull('<<FIELD>>') AND like('first<<MATCHSTR>>',"%stop"),"1",if(isnull('<<FIELD>>') AND like('last<<MATCHSTR>>',"%start"),"1",if(isnull('<<FIELD>>') AND like('last<<MATCHSTR>>',"%stop"),"0",'<<FIELD>>'))))] | fields - first*, last* | filldown * | reverse | filldown * | reverse | addtotals fieldname=Running | fields _time,Running $field1.earliest$ $field1.latest$ line all progressbar $click.value2$ $click.value$ ($click.value$ + 300) strftime($earliest$, "%m/%d/%Y %I:%M:%S %p") strftime($latest$, "%m/%d/%Y %I:%M:%S %p") <panel depends="$jobcount$"> <title>List of Jobs from $earliest1$ to $latest1$</title> <table> <search id="CDH_Cluster_jobs_table"> <query>index=infra_apps sourcetype=ca:atsys:edemon:txt | search Job=$job$ | rename hostname as host | eval t1=strftime(_time,"%Y-%m-%d %H:%M:%S") | eval time_epoch=strptime(t1,"%Y-%m-%d %H:%M:%S") | fields Job host Autosysjob_time Status _time time_epoch t1 | lookup datalakenodeslist.csv host OUTPUT cluster | mvexpand cluster | table Job Status host cluster _time time_epoch | search cluster=$clustername$ AND host=$host$ | sort + time_epoch | stats count by _time Job Status host cluster time_epoch | bin span=2m _time | makecontinuous span=2m _time | filldown _time Job Status host cluster count time_epoch | table _time Job Status host cluster | search Status=$jobstatus$ | table _time Job host cluster Status $earliest$ $latest$ 10m delay 10 none progressbar ... View more

@vnravikumar thanks for the reply.....Its showinf search is waiting for input. the following is the piece of code i have incorporated in the xml code. <label>Select Job status</label> <choice value="STARTING">Starting</choice> <choice value="RUNNING">Running</choice> <default>RUNNING</default> </input> <panel> <table> <search> <query>index=infra_apps sourcetype=ca:atsys:edemon:txt EventCode=40245 Status=$status$ AppID=$appid$ Machine=$host$ Job=$job$ | dedup _raw | lookup datalakenodeslist.csv host as Machine OUTPUT cluster | search cluster=$clustername$ | timechart count span=2m <earliest>-24h@h</earliest> <latest>now</latest> </search> <option name="drilldown">none</option> </table> </panel> ... View more

I just got a query derived for no of running jobs at each time as follows index=XXX sourcetype=YYYY EventCode=40 AND (Status=Failure OR Status=Terminated OR Status=Running OR Status=Success) AppID=$appid$ Machine=$host$ Job=$job$ | dedup _raw | lookup datalakenodeslist.csv host as Machine OUTPUT cluster | search cluster=$clustername$ | eval running=if(Status="RUNNING","1","0"),status=if(Status="RUNNING","start","stop"), time=_time+status | bin span=2m _time | stats max(running) as running, earliest(time) as first, latest(time) as last by Job,_time | xyseries _time Job running first last | makecontinuous span=2m _time | streamstats window=2 global=f earliest(last*) as last* | reverse | streamstats window=2 global=f earliest(first*) as first* | reverse | foreach running* [ eval < >=if(isnull('< >') AND like('first< >',"%start"),"0",if(isnull('< >') AND like('first< >',"%stop"),"1",if(isnull('< >') AND like('last< >',"%start"),"1",if(isnull('< >') AND like('last< >',"%stop"),"0",'< >'))))] | fields - first*, last* | filldown * | reverse | filldown * | reverse | addtotals fieldname=Running | fields _time,Running But i find it too difficult to just understand the query :(, and not able to edit this query to have single query for both RUNNING and STARTING statuses. Can you please help to have a single query for both the statuses. ... View more

@sideview thanks for such detailed explanation. The following the is the latest query i am using index=infra_apps sourcetype=ca:atsys:edemon:txt | search Job=* | rename hostname as host | eval t1=strftime(_time,"%Y-%m-%d %H:%M:%S") | eval time_epoch=strptime(t1,"%Y-%m-%d %H:%M:%S") | fields Job host Autosysjob_time Status _time time_epoch t1 | lookup datalakenodeslist.csv host OUTPUT cluster | mvexpand cluster | table Job Status host cluster _time time_epoch | search cluster=EDL-PRD-M01 AND host=* | sort + time_epoch | stats count by _time Job Status host cluster time_epoch | bin span=2m _time | makecontinuous span=2m _time | filldown _time Job Status host cluster count time_epoch Sorting the resutls before stats command with _time was not working for me , thats why i had to use time_epoch. And i tried using | bin span=2m time_epoch and | bin span=2m _time but these are giving me different results at times , i still confused what to use on this line. P S: Example of how _time being extracted in this case is 2020-03-09 10:00:03.863 ... View more

@woodcock @jkat54 @somesoni2 @sideview can you please help ... View more

index=infra_apps sourcetype=ca:atsys:edemon:txt | search Job=* | rename hostname as host | eval time_epoch=strftime(_time,"%Y-%m-%d %H:%M:%S") | fields Job host Autosysjob_time Status _time time_epoch | lookup datalakenodeslist.csv host OUTPUT cluster | mvexpand cluster | table Job Status host cluster _time time_epoch | search cluster=EDL-PRD-M01 AND host=* | sort + time_epoch | stats count by _time Job Status host cluster time_epoch | bin span=2m _time | makecontinuous span=2m _time | filldown _time Job Status host cluster count time_epoch | stats count by _time Status this is giving me the correct result, but i am not able have a filter in the query like search Status=RUNNING for example, When i add the status filter its giving me different result. @woodcock @jka can you please help. ... View more

@richgalloway this is not helping.... I just want to give you some back ground about this..I have a single event for each job status in splunk But to make it continuous i am inserting 2 mins interval using bin and makecontinous command and filling those intervals with the last status it updated, For example if its running status at 9 am and success status at 10 am, i am creating 2m interval between that 1 hour gap and filling it with Running status. I suspect that is one of the reason i am not able to get that in graph as i am expexting. PS: I am getting the correct result till the filldown command, after that i am struggling. ... View more

its not working as expected. ... View more

@nickhills i can not apply dedup SOR as it nullifying all other results for other dates as well, following is there what i have developed so far, but its not working as expected. |inputlookup MBDA_SLA_stats.csv | dedup SOR feed timestamp | eval status=if(timestamp_epoch>Expected_time_epoch,"Missed SLA","Met SLA") | chart last(status) by Business_Date SOR useother=f limit=50 | fillnull value="Not Run" |sort - Business_Date |rename Business_Date as "Business Date" ... View more

@kamlesh_vaghela here is the query i am using, But this is not working as expected |inputlookup MBDA_SLA_stats.csv | dedup SOR feed timestamp | eval status=if(timestamp_epoch>Expected_time_epoch,"Missed SLA","Met SLA") | chart last(status) by Business_Date SOR useother=f limit=50 | fillnull value="Not Run" |sort - Business_Date |rename Business_Date as "Business Date" ... View more

@nickhillscpl thanks for the comment. I have few data feeds that share common SOR name. For example if any ONE data feed have status as Missed SLA , I want to calculate that whold SOR that data feeds belongs to as Missed SLA. If all of the datafeeds have Met SLA , i want to calculate that SOR as Met SLA. ... View more

Finally, i solved my own puzzle, Thanks to everyone for spending their time here Correct Answer: index=infra_apps sourcetype=ca:atsys:edemon:txt | stats count by _time Job Status hostname | bin span=5m _time | makecontinuous span=5m _time | filldown _time Job Status | chart count(Job) OVER _time BY cluster ... View more

Finally, i solved my own puzzle, Thanks for everyone for spending their time here Correct Answer: index=infra_apps sourcetype=ca:atsys:edemon:txt | stats count by _time Job Status hostname | bin span=5m _time | makecontinuous span=5m _time | filldown _time Job Status | chart count(Job) OVER _time BY cluster ... View more

I am trying filter it with one job...its showing values in the legend <<>JobName>> : RUNNING <<>JobName>> : SUCCESS NULL And its not showing the continuous line for Running status as expected, Could you please correct this and also i dont need to woryy about success status now...i need data for only how many jobs are in starting state and how many are in running state at each 5 mins ... View more