Yes I did figure it out in the end but nobody responded to my issue on here so hence why the solution was not posted. This is how I manually deleted my Data Lab Inputs from DB Connect File to edit to manually delete DB Connect Inputs if deleting in GUI fails with error SPLUNK_HOME/etc/apps/search/local/db_inputs.conf NOTE: SPLUNK_HOME = Splunk Home Directory in my case I have a windows Splunk Enterprise server so my full path was C:\Program Files\Splunk\etc\apps\search\local To manually remove a DB Connect input edit the file above Highlight and delete all related entries for the DB Connect input Example: in my file there was a Test DB so to delete that DB remove the title line and the 6 lines referencing settings for the DB that are immediately below the Title line. See Below [test] coldPath = $SPLUNK_DB\test\colddb enableDataIntegrityControl = 0 enableTsidxReduction = 0 homePath = $SPLUNK_DB\test\db maxTotalDataSizeMB = 10240 thawedPath = $SPLUNK_DB\test\thaweddb Save the file Restart Splunk Confirm in the GUI Console that the Inputs are no longer there   NOTE: the file above is how I resolved the issue in my environment but just in case the databases are not listed in that file the file below is another location where the DB Connect App Databases are shown so potentially they could be removed there as well. C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\local\db_inputs.conf ... View more

Hello Giuseppe Thanks so much for your help. I edited my search to add the index and more important to add your suggested command and it works a treat. really appreciate your taking the time to help me get this sorted. have a great day! Will ... View more

Hello Giuseppe thanks so much for your help. unfortunately I still get events during the ) 0200 - 0400 times  changed my search to below sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" NOT (time_hour=2 OR time_hour=3) | rex "(?<json>{.*})" | spath input=json systemEvents{} output=systemEvents | stats values(_time) as _time by systemEvents | spath input=systemEvents | fields - systemEvents | eval _time=strptime(timestamp,"%s") | search message=*DOWN |sort - _time | table _time,eventCode,metaData.server,metaData.listener,eventCode,message,moduleName,severity | rename metaData.listener TO Site,metaData.server TO Server results of new search with your suggestion below (note the events at 0210?) _time eventCode Server Site message moduleName severity 2020-08-27 07:13:27 30302 Server69 Carson_MDCM_Servers A Loadbalancer Server Status is changed to DOWN vShield Edge LoadBalancer Informational 2020-08-27 02:10:14 30302 Server81 WT_MDCM_Servers A Loadbalancer Server Status is changed to DOWN vShield Edge LoadBalancer Informational 2020-08-27 02:10:03 30302 Server62 WT_MDCM_Servers A Loadbalancer Server Status is changed to DOWN vShield Edge LoadBalancer Informational 2020-08-27 01:08:59 30302 Server69 Carson_MDCM_Servers A Loadbalancer Server Status is changed to DOWN vShield Edge LoadBalancer Informational 2020-08-26 20:41:08 30302 Server62 WT_MDCM_Servers A Loadbalancer Server Status is changed to DOWN vShield Edge LoadBalancer Informational 2020-08-26 20:34:53 30302 Server69 Carson_MDCM_Servers A Loadbalancer Server Status is changed to DOWN vShield Edge LoadBalancer Informational 2020-08-26 17:12:13 30302 Server69 Carson_MDCM_Servers A Loadbalancer Server Status is changed to DOWN vShield Edge LoadBalancer Informational ... View more

hello to4kawa thanks so much for your help. that did the trick. using your search as a base I simply added a table to eliminate a few of the fields we do not need and renamed the fields to simpler ones for our needs and it worked great! very much appreciated. I do wish I could give some points to spitchika as well. the solutions offered by spitchika (though a different approach from your solution) really got me close. but in the end yours really worked great and displays everything accurately and just what we needed. thanks so much for your help. I have created a lot of alerts for our business but still learning a LOT as regex is very hard to get my head around. kind regards and thanks again! Will ... View more

Hello Spitchika you are AWESOME! it is really close but I still have some pieces not showing accurate data. to clarify your latest suggested search DOES fix and display ALL of the message values correctly. So I believe you definitely fixed the issue for that field but I seem to have inaccurate data in the other two that needs sorting out. unfortunately the server and the site table column data is not matching the correct server or site in the event to the NOW CORRECT message in the new TotalMessage column NEW SEARCH sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" | rex field=_raw max_match=0 "message\":\"(?<TotalMessage>[^\"]+)" |sort - _time |fieldformat _time = strftime(_time, "%b %d, %Y - %H:%M") |mvexpand TotalMessage |table _time,server,site,TotalMessage NEW RESULT (highlighted green = data correct-matches log, red = data does not match log) Aug 07, 2020 - 05:28 Server69   Carson_MDCM_Servers   A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 05:28 Server69   Carson_MDCM_Servers   A Loadbalancer Server Status is changed to UP Aug 07, 2020 - 03:01 Server69   Carson_MDCM_Servers     A Loadbalancer Server Status is changed to DOWN BOTH IN RED BELOW (Server69 should be Server81 & Carson MDCM Servers should be WT_MDCM Servers) Aug 07, 2020 - 03:01 Server69   Carson_MDCM_Servers     A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 03:01 Server69   Carson_MDCM_Servers     A Loadbalancer Server Status is changed to UP Aug 07, 2020 - 03:01 Server69   Carson_MDCM_Servers     A Loadbalancer Server Status is changed to UP Aug 07, 2020 - 02:46 Server62   WT_MDCM_Servers           A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 02:46 Server62   WT_MDCM_Servers           A Loadbalancer Server Status is changed to UP Aug 07, 2020 - 01:51 Server69   Carson_MDCM_Servers     A Loadbalancer Server Status is changed to UP Aug 07, 2020 - 01:50 Server69   Carson_MDCM_Servers     A Loadbalancer Server Status is changed to DOWN From the results it seems like the only discrepancy if from the data being parsed from the Aug 7 03:01:12 event which is the only one that contains event values for TWO different Servers (Server69 and Server81) all of the other events being extracted during the searches timeframes only contain events for a single servername So it looks like it is not pulling the correct server from the two middle events in the following log  Aug 7 03:01:12 NSX-Edge03-0 MsgMgr[2349]: [MDCM]: payload len:1019 data:{"systemEvents":[{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30302", "message":"A Loadbalancer Server Status is changed to DOWN", "timestamp":1596794458, "metaData":{ "listener" : "Carson_MDCM_Servers", "server" : "Server69" } },{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30302", "message":"A Loadbalancer Server Status is changed to DOWN", "timestamp":1596794458, "metaData":{ "listener" : "WT_MDCM_Servers", "server" : "Server81" } },{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30301", "message":"A Loadbalancer Server Status is changed to UP", "timestamp":1596794468, "metaData":{ "listener" : "WT_MDCM_Servers", "server" : "Server81" } },{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30301", "message":"A Loadbalancer Server Status is changed to UP", "timestamp":1596794478, "metaData":{ "listener" : "Carson_MDCM_Servers", "server" : "Server69" } }]} ... View more

Hello Spitchika thanks so much for your help. I tried your suggestion but I get the same results I posted before. so here is what I have now search I am using now which is applying your suggestions, please advise if I formatted it correctly. sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" | rex field=_raw max_match=0 "message\":\"(?<TotalMessage>[^\"]+)" |sort - _time |fieldformat _time = strftime(_time, "%b %d, %Y - %H:%M") |mvexpand message |table _time,server,site,message   the resulting table from the search above Aug 07, 2020 - 05:28      Server69     Carson_MDCM_Servers     A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 03:01      Server69     Carson_MDCM_Servers     A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 02:46      Server62     WT_MDCM_Servers            A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 01:51     Server69     Carson_MDCM_Servers     A Loadbalancer Server Status is changed to UP Aug 07, 2020 - 01:50     Server69     Carson_MDCM_Servers     A Loadbalancer Server Status is changed to DOWN   The ACTUAL Logs indicate the events for this search time period should are missing the following events Aug 7 02:46:12 should show the following additional event Server62     WT_MDCM_Servers            A Loadbalancer Server Status is changed to UP   Aug 7 03:01:12 should show the following additional events Server81     WT_MDCM_Servers            A Loadbalancer Server Status is changed to DOWN Server81     WT_MDCM_Servers            A Loadbalancer Server Status is changed to UP Server69     Carson_MDCM_Servers     A Loadbalancer Server Status is changed to UP   Aug 7 05:28:12 should show the following additional event Server69     Carson_MDCM_Servers     A Loadbalancer Server Status is changed to UP   actual logs are below Aug 7 01:50:12 NSX-Edge03-0 MsgMgr[2349]: [MDCM]: payload len:272 data:{"systemEvents":[{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30302", "message":"A Loadbalancer Server Status is changed to DOWN", "timestamp":1596790245, "metaData":{ "listener" : "Carson_MDCM_Servers", "server" : "Server69" } }]} Aug 7 01:51:12 NSX-Edge03-0 MsgMgr[2349]: [MDCM]: payload len:270 data:{"systemEvents":[{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30301", "message":"A Loadbalancer Server Status is changed to UP", "timestamp":1596790256, "metaData":{ "listener" : "Carson_MDCM_Servers", "server" : "Server69" } }]} Aug 7 02:46:12 NSX-Edge03-0 MsgMgr[2349]: [MDCM]: payload len:515 data:{"systemEvents":[{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30302", "message":"A Loadbalancer Server Status is changed to DOWN", "timestamp":1596793558, "metaData":{ "listener" : "WT_MDCM_Servers", "server" : "Server62" } },{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30301", "message":"A Loadbalancer Server Status is changed to UP", "timestamp":1596793578, "metaData":{ "listener" : "WT_MDCM_Servers", "server" : "Server62" } }]} Aug 7 03:01:12 NSX-Edge03-0 MsgMgr[2349]: [MDCM]: payload len:1019 data:{"systemEvents":[{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30302", "message":"A Loadbalancer Server Status is changed to DOWN", "timestamp":1596794458, "metaData":{ "listener" : "Carson_MDCM_Servers", "server" : "Server69" } },{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30302", "message":"A Loadbalancer Server Status is changed to DOWN", "timestamp":1596794458, "metaData":{ "listener" : "WT_MDCM_Servers", "server" : "Server81" } },{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30301", "message":"A Loadbalancer Server Status is changed to UP", "timestamp":1596794468, "metaData":{ "listener" : "WT_MDCM_Servers", "server" : "Server81" } },{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30301", "message":"A Loadbalancer Server Status is changed to UP", "timestamp":1596794478, "metaData":{ "listener" : "Carson_MDCM_Servers", "server" : "Server69" } }]} Aug 7 05:28:12 NSX-Edge03-0 MsgMgr[2349]: [MDCM]: payload len:523 data:{"systemEvents":[{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30302", "message":"A Loadbalancer Server Status is changed to DOWN", "timestamp":1596803281, "metaData":{ "listener" : "Carson_MDCM_Servers", "server" : "Server69" } },{ "moduleName":"vShield Edge LoadBalancer", "severity":"Informational", "eventCode":"30301", "message":"A Loadbalancer Server Status is changed to UP", "timestamp":1596803291, "metaData":{ "listener" : "Carson_MDCM_Servers", "server" : "Server69" } }]} ... View more

thanks for that spitchika that works to present the two messages but it only shows them once. my fault maybe I should have given more details of what I am trying to accomplish. I have created 3 field extractions with the following field names: (data I am trying to extract is noted to the right of the field name below) message (to extract the "message": values: "A Loadbalancer Server Status is changed to DOWN" OR "A Loadbalancer Server Status is changed to UP" entries ) server (to extract the "server" : values:  "Server69") site (to extract the "listener" : values:  " Carson_MDCM_Servers" OR "WT_MDCM_Servers") I want a search to display the results in a table showing the time of the event and the values from the server, site and message fields extracted above. I am currently using the following search and it does work BUT it is not grabbing every instance of the values listed above. if an event has FOUR instances of the same event with a different site, server and message my search only returns the first instance or sometimes the first two instances but ignores the other two. so my search is no accurate as it is not parsing all of the occurrence's from a single event when there are three or more of them in a single event from the same time as in my original example. My Search is currently sourcetype=syslog_nsxedge host="NSX-Edge03-0" server!="NULL" |sort - _time |fieldformat _time = strftime(_time, "%b %d, %Y - %H:%M") |table _time,server,site,message   the search above returns the following Aug 07, 2020 - 05:28      Server69      Carson_MDCM_Servers      A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 03:01      Server69      Carson_MDCM_Servers      A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 02:46      Server62      WT_MDCM_Servers             A Loadbalancer Server Status is changed to DOWN Aug 07, 2020 - 01:51      Server69      Carson_MDCM_Servers      A Loadbalancer Server Status is changed to UP Aug 07, 2020 - 01:50      Server69      Carson_MDCM_Servers      A Loadbalancer Server Status is changed to DOWN   it looks correct but when compared against the RAW syslog information it is clear that there are missing events not listed in the table ... View more

Hi Chrisyoungerjds, thanks so much for your help, I chose the answer from woodcock because he clarified the correct syntax but I wish I could put you both as your suggestion (once I used woodcocks syntax) did work perfectly. I really appreciate your help and I gave you points in lieu of choosing as the answer. thanks again your answer got me on the right track and with the syntax provided by woodcock it worked and did exactly what I was looking for. ... View more

Hi Woodcock, awesome that worked a treat! thanks so much for your help clarifying the command syntax. I just tried it in my search and it worked exactly as I wanted! I am awarding points to chrisyoungerjds as well for this. you guys both really helped me out here. very much appreciated! ... View more

Hello, when using syslog-ng with splunk you can control the index and sourcetype etc in the app inputs.conf that is pushed to your forwarder. of course this implies you are using a splunk enterprise on prem deployment server to push apps to your endpoints forwarders. for example firewall sends its log to syslog-ng syslog-ng.conf has filters configured for each type of device sending logs, it checks through those filters until it finds a match for the log received from the firewall. lets say it is using subnet to identify this is a firewall. once it matches the filter it then checks the log rule which tells it which destination rule to use to place the log in the folder structure and naming convention that has been configured in the destination rule for this device type lets say it puts the log in "/home/syslog/logs/firewall/$HOST/$YEAR-$MONTH-$DAY-firewall.log" you will note that in the path that is configured in the destination rule for this log the hostname will be the 5th folder in the path so in your inputs.conf file for whichever app you use for this devicetypes logs you would need to tell it that the 5th position is where the hostname is. splunk will then be able to find the logs using that name with the host=host name example in my example the syslog_firewall_inputs\local\ inputs.conf file would look like this Firewall Logs sent to syslog-ng [monitor:///home/syslog/logs/firewall//.log] host_segment = 5 sourcetype = fgt_log index = firewall disabled = false note that the sourcetype was one we created ourselves for our environment so be sure all settings are for your own environment. as you can see it is in this file where you can specify which index and which sourcetype splunk UF will associate with the log. I hope this is helpful. your description was not 100% clear so I am hoping this is what you need ... View more