I think I've read this in its entirety 4 times now over the past week.  I am having difficulty understanding what the problem is.  Let me walk through it and see if writing it down helps... You work in IST which is +10.5 hours from CST/DST. You have alert, which the cron schedule says to fire at 1 PM (13:00) in CDT.  That's 11:30 PM (23:30) IST.  You maybe mistyped "11:00 PM" for that, and maybe that's the issue? Disregarding the 11:00/11:30 issue, the second thing I think you mentioned is that the alert didn't actually come until 11:44, which is a 14 minute delay.   The search itself is a lightweight, it should run practically instantly and run-time shouldn't be an issue.  The most obvious reason for the 14 minute delay is because your server is too busy at 1 PM CDT to get this out any faster.  You should check into that - there's a lot of resources available inside Splunk to see what might be going on, but my guess is just that it's a busy time of the day, coupled with possibly too many "heavy" searches that trigger then.  You could also increase the priority of that search, though this doesn't address the core problem and may actually make things *worse* and not better.  I mean, maybe better for this one search, and being so fast that's probably OK, but still, it's just trying to hide the bigger problem.   Anyway, hope that helps and happy Splunking! -Rich   ... View more

How do you know that xa goes with x1 and xc goes with x2?  Like literally I'm looking at your example and I don't know how you would have come up with that correlation. Can you tell me how you'd do this manually?  Unless we know this, I have no idea how we'd know how to make it work programatically. ... View more

3.6 is old and you should update, then the current documentation would work.  🙂 But, it also had a simpler system of checkpoints, storing those in files on disk. I cannot remember exactly where those were, but maybe something like $SPLUNK_HOME/var/lib/splunk/modinputs/dbx_input...?  I don't know, I have that directory and it seems familiar, but it's empty on my system because I try to keep up to date.  (Hehehe, you knew I was going to say that, didn't you... ?  🙂   ... View more

Normally, I've found using the command line (CLI) to set up being a deployment client is easier.  It's a single command on your Splunk Universal Forwarder.  https://docs.splunk.com/Documentation/Splunk/9.2.1/Updating/Configuredeploymentclients Receiving is, as you noticed, on the receiving page.  As long as that's configured (obligatory Splunk Docs link https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Enableareceiver) then it'll accept data from other systems. Now keep in mind those two systems are separate, use different ports and are managed differently/separately, and in fact you can have either one without the other (though some combinations are a bit silly or pointless).  And there's three concepts. Receiving is set up on an indexer (technically, on any full Splunk instance), and enabling it enables that server to accept data (usually over port 9997) from any other Splunk instance, UF or HF (that's "Universal Forwarder" and "Heavy Forwarder", in case you didn't know). Forwarding is the other side of Receiving - Forwarding is set up on the machines you want to have send their data to that receiving system.  Enabling forwarding on a UF tells it to send all the data that it reads/whatever into that receiving system. Deployment client/server is a *management* system.  So you set up receiving and forwarding which is the plumbing of how the data moves around.  Then the deployment server (which is, iirc, enabled by default on all full Splunk instances so does not need to be separately enabled) is listening on port 8089 by default.  Once you have the forwarding and receiving in place, you tell your UF to be a deployment client of the deployment server via the docs I supplied above.  After you've done that, when you go on the server to add data you'll have new options in the 'add data' section to have it get the data from the UF.  Here's another docs link for that.  https://docs.splunk.com/Documentation/Splunk/latest/Data/Forwarddata   Hope that helps!  I know it's got some new concepts and ideas and it's not quite like many other systems.  But it all generally does make sense once you get over the hump. -Rich ... View more

Ah, backups.  Splunk has this documented, so I'll just point you to their docs on "Backup and restore Splunk DB Connect version 3.10.0 or higher" Hope that helps! -Rich ... View more

In this case I suspect starting at the end and working backwards might be helpful. WMI - While it's not terrible for some small testing, I'd suggest not using it because it's *far* more difficult to set up, manage, and deal with than using a Universal Forwarder on the actual endpoint.  The UF installs easily, is tiny and efficient, and *also uninstalls easily and completely too*.  And don't take my word for it, Splunk also has docs for this.  I know, it'll sound like they're "pushing the UF for some nefarious reason" but there's nothing nefarious about it, it's just better in nearly every way than using WMI.   https://docs.splunk.com/Documentation/Splunk/9.2.1/Data/ConsiderationsfordecidinghowtomonitorWindowsdata Even neater is to spend the few minutes - it's not terribly hard! - to set up the forwarders to use your splunk as a deployment server. Then on your Splunk you *can* create remote inputs, but instead of being some unreliable "pull" over wmi, it'll be configs sent to the UF to tell it how to collect them locally and send in those logs. And with those changes, all your complaints about WMI will disappear.  I mean, you may have new  complaints, but they won't be about WMI.  🙂 "Could not find userBaseDN on the LDAP server" is just generally just 'incorrect configuration'.  Some time in ADSI Edit and the various AD tools may help here. And network devices - it truly depends on your familiarity with syslog etc, but even having had been a Windows admin I found getting network device data into Splunk was at least as easy as getting Windows data in.  You literally started with what I think is the hard part.  🙂  There's one or two extra moving parts, but they're all simple, isolated parts in the device->syslog->UF->Splunk path that are easily understood and worked with, vs. the "magic" and weird stuff that the Windows event logs can sometimes conjure up. And a note - we're all 100% volunteers here.  I'm sure the comment about "no time wasters" was just frustration speaking, and that's understandable.  But it did come off as somewhat unkind and I'm sure you would have gotten something of an answer much quicker without that.  No one here that I've ever seen wants to waste your time.  We're all spending our free time trying to help people.  ... View more

Maybe just try to upgrade it to the oldest 8.x version you can get on the downloads page, then uninstall it after that?   ... View more

It's terrible, they're not easily accessible except through the UI.  It's a big ... sore spot for some of us who need to use these in a more programmatic way. But, there is a way using the REST interface from cURL. curl -k -u <username>:<password> https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input Obviously fix the username and password to an admin one, and your hostname if it's not on localhost.  You might want to pipe that through jq to 'pretty print' it if you have jq installed because otherwise it's all smashed together and hard to read: curl -k -u <username>:<password> https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input | jq .  You can also see only an individual one if you append the _key's value for the one you want to the end.  (The _key comes from the output of one of the earlier commands.) curl -k -u <username>:<password> https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input/6452ce6e55102d0ad735ec31 | jq . You can also delete them or edit them, though ... obviously be careful and do this in a test environment at first! curl -k -u <username>:<password> https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input/6452ce6e55102d0ad735ec31 -X DELETE And I've not found a good way to "edit" them, but it's pretty trivial to just edit the JSON you get from an individual entry, and load that back in wholesale. curl -k -u <username>:<password> https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input -d '{ "inputName" : "newEntryforMyDB", "value" : "200", "appVersion" : "3.16.0", "columnType" : 4, "timestamp" : "2024-03-21T13:11:41.633-05:00", "_user" : "nobody", "_key" : "65fc6ce1764e95450b0d98e1" }' -H "Content-Type: application/json" Which would overwrite entry 65fc6... with that new information. Happy Splunking, Rich   ... View more

Your *exact* example doesn't make much sense - why would y-d be y1 instead of y2? But at least some of this may be as simple as "makemv" and/or "mvexpand". In your example, it appears as if abcde are all multi-value fields (the "mv" in the two above commands).  If that's so, ... | mvexpand parameter should make the original into 13 rows.  Once they're separated, perhaps there's some other eval/conditionals you can use to get each output row to include the correct value? If that doesn't work, you may need something like ... ... | makemv delim=" " parameter | mvexpand parameter In any case I think you'll be two steps closer and we can iterate from there.   happy Splunking, Rich ... View more

1) is a giant question.  🙂  The shortest story here is probably to understand the Admin differences - what you will no longer be able to do yourself and will need a ticket for.  The second is to understand the licensing and billing you will be using and how that may affect things.  A lot of that is covered in the Splunk Cloud Platform Migration Success Guide. 2 and 3 both) It's generally best to send from the UFs direct to cloud, that way all your indexers will equally participate in receiving the data.  Ditto with your syslog servers - they already have a UF/HF on them, I'd suspect, to grab the data sent in by syslog and send it into your on-prem instance so you just need to reconfigure those to forward data to your cloud instance instead of on-prem instance.   In your cloud instance you'll find an app called the (or some variation of) Splunk universal forwarder credentials package.  Click that and it has instructions and a little app to install on your forwarders to teach them how to talk to your cloud instance. You could send your syslog directly in to cloud too, using the SC4S app from Splunk. 4) I believe Splunk Cloud only accepts encrypted streams (https) so the encryption is enforced by the Splunk universal forwarder credentials package you can download from your cloud instance to set up your forwarders.  Compression is not necessary.   I hope that helps! -Rich ... View more

There's a few ways to go about this and none of them are really easy.  There's a specific command 'delta' but it only works for one field, so we'll have to go a bit off road. I just used my firewall data "by transport" instead of "by user" (== tcp, udp, etc), but I'm sure you can adapt it to yours (only lines 1 and 2 need changing) index=fw | timechart span=10m count by transport | streamstats window=2 first(*) as first_* last(*) as last_* | foreach *_* [eval delta_<<MATCHSEG2>> = first_<<MATCHSEG2>> - last_<<MATCHSEG2>> ] | fields delta*  So what that does: Line 1 and 2 are more or less like you have them.  Start by running just these two lines and adapting until you get your timechart data coming out OK. In line 3 we use streamstats to build groups of two of those events, and though it looks messy with the wildcards and underscores, it'll build new fields like first_tcp, last_tcp and so on. Line 4 is foreach and says for every field with a _ in the middle, like first_tcp, make a new field delta_tcp which is first_tcp - last_tcp. The last line just trims out all fields except our delta_* fields, because that's the only one we want. Try those - get the first two working, then add one line at a time and watch what each one adds/does.   Happy Splunking! -Rich ... View more

The Universal Forwarder has no UI.  Splunk Web isn't available on it, so there's no http site to go to. https://docs.splunk.com/Documentation/Splunk/9.2.0/Forwarding/Typesofforwarders To get a UI like that *on a forwarder* you'll need to install a Heavy Forwarder which is actually just the full Splunk server installation - what makes it a HF instead of just "Regular Splunk" is how it's configured later to both receive and 'forward' data in to another Splunk instance. Or just an install of the full Splunk installation. ... View more

The search you provided isn't in the format where using Trellis makes sense.  Turn off Trellis for the Single Value visualization you are using and the "distinct_count" will disappear.   ... View more

Sweet, I was probably typing (got distracted) when you were posting.  Glad we had the same answer.  🙂 ... View more

I have some ideas about this. First, you'll need your dates/times in datetime format, convertible to unix epoch time.  Here's my mockup that I'm working with - run this and take a close look at the output. | makeresults format=csv data="time_start, time_end, person 2024-03-07T07:00:00-0600, 2024-03-07T11:00:00-0600, Rich 2024-03-08T07:30:00-0600, 2024-03-08T15:00:00-0600, Rich" | eval _time = strptime(time_start, "%Y-%m-%dT%H:%M:%S%Z"), time_end_unix = strptime(time_end, "%Y-%m-%dT%H:%M:%S%Z") | append [| makeresults format=csv data="incident_time, incident 2024-03-07T06:50:00-0600, blizzard 2024-03-07T11:23:00-0600, hurricane 2024-03-08T13:44:00-0600, tornado 2024-03-08T18:03:00-0600, dust_storm" | eval _time = strptime(incident_time, "%Y-%m-%dT%H:%M:%S%Z") ] I'm just making it up though - hopefully your search will look more like (index=timetracking (sourcetype="user:clockin" OR sourcetype="user:clockout") OR index=incidents) | eval end_time_unix = strptime(<my time format string>, <my end time field>)   You'll need to install the Timeline visualization from https://splunkbase.splunk.com/app/3120 Now that we have that data, the Time viz requires a specific format of data, so let's do a little work... You need a duration, right?  (And it has to be in milliseconds, so in most sane environments you'll just want to multiply the "seconds" by 1000).  So to the end of the above search, add this: | eval duration = if(time_end_unix>0, (time_end_unix - _time) * 1000, 0) Now, take a look at those results.  The important fields are _time, duration, and incident and person. But incident and person are two different things and the visualization won't like that a lot. So let's put them together.  Add to the end of *that* ... | eval item = coalesce(person, incident) Now, added to the output is "item" with either "Rich" or some random weather event. Good enough!  At least for pass #1.  Let's add a table command to the end of *that*... | table _time, item, duration The click the visualization tab, change it to the Timeline, and ... well it's close!  Unfortunately, I don't like the blizzard, hurricane, tornado and dust_storm all being on their own lines like in this screenshot. How can we fix that?  Well, one way would be to make all the incident's "names" just be "incident" so it'd show up on its own line.  To do that, change the "eval item =..." line to | eval item = if(isnotnull(person), person, "incident") and leave the ending | table command.  Now it looks like this I honestly think that as long as you may have multiple people's schedules, this is as good as you can get.  Example with another person added in: But if you absolutely know there will never be another person, you could make them sit on top of each other by any mechanism to just make "person" always be "Rich".  Here I use a fillnull command   Anyway, my suggestion is to play with my examples until you understand them. Once you do, you might be able to make your own data work like this.  But if you still have difficulties after all that, post back and we'll see if we can help! Also it's entirely possible someone else will come up with a completely different type of answer.  I don't know, it'll be interesting if they do - there's a lot of smart folks around here and we don't all think alike! Happy Splunking, Rich   ... View more

As to the missing results - sure, because your TOTAL field appears empty.  You should just debug that for a start.  All the extra conditional logic you want can be implemented later once you get this core piece working.  Here's how I'd approach it: Temporarily comment out (triple backticks before and after them) or remove all the fieldformats and the trailing table command so you can see what values all fields actually have.  When you put them back in, I suggest doing those "pretty it up" tasks as one of the last steps after all actual "work" has been done.  Also this makes it easier to follow the code because it'll be structured better - first get your data, then do your calculations, lastly make things pretty. Then just backtrack.  Divide and conquer.  Remove all the stuff after the stats command where TOTAL is calculated.  If there's no result for TOTAL, figure out why.  Since TOTAL is the sum of time_difference, take out everything from the stats onward and see what time_difference is in the events.  If it's blank, then work backwards one more step and see where it comes from - incident review time and notable time - so what are the values for *those* fields?  At some point you'll see what I'm sure is a facepalm somewhere in there. Once you have all that straightened out, add back in the extra stuff one step at a time, confirming the results at each step.  You'll have a lot better understanding of the data you are working with and also how all this works, too. THEN. There's likely to be no reason at all to separately do only "medium" severity.  I suspect if you remove the "where" way up near the top, then do all your stats "by severity" you may be able to just calculate the answers for all severities in one pass.  But again, baby steps. Get it working first, then we can modify it to do that.   ... View more

Easiest is to just have two alerts.  There's practically *zero* downsides to just building a new search (you can start with your existing one!) and then creating an alert out of it once it appears like you've got the search all sorted out. That being said, I think you might just need to change source = "/tmp/unresponsive" sourcetype=cmi:gems_unresponsive to be able to do both at once.  I'm not sure what you need to change that to though. MAYBE - if /tmp/unresponsive is the source for either server, maybe all it needs is source = "/tmp/unresponsive" ( sourcetype=cmi:gems_unresponsive OR sourcetype=<whatever the sourcetype is for the other servers> ) And honestly, I'd go back to that core piece of the search (index=foo, source=bar, sourcetype=baz) and *find the events* first.  It should make it more obvious how to get their data in there too. ... View more

What's the actual regex you are using to capture it? And if you are sure you are using the right syntax because you copy pasted it from some working one - you could try to add | eval state = "up" as the last command in the search to force it to be "up" and see if that works.  If it doesn't, then I'd say there's something else wrong with that syntax. ... View more

I don't think that's the right app to read those events.  In any case, the app you have installed had its latest release in 2018 and references no Splunk version higher than 7.1, so it looks abandoned. Instead, it looks like the  "Cisco Secure eStreamer Client Add-On for Splunk" (https://splunkbase.splunk.com/app/3662) might extract fields from records with FTD in them.  It seems like it focuses on events 430001, 430002, 430003 and 430005.  Still, it's worth a shot. Indeed, right now you could see if you have those - try a search like index=<your cisco index> FTD (430001 OR 430002 OR 430003 OR 430005) If that returns a few items (or lots), then the app I mention above should turn that into useful fields. If that search does NOT return any events, ... well, widen the time frame.  These seem like they might be less common events, not run of the mill "every tcp session makes 42 zillion of them" so it's possible there's only a few per day or something. In any case, happy splunking and I hope you find what you need! -Rich     ... View more

I've taken a few stares at that search, and I honestly think you are trying to do way too much in one search. There are several giveaways.  a) There's different binning/bucketing for one of them - though I think the entire thing could be binned to 1s intervals to no actual detriment, b) results are joined together using 'join' which is rarely the right command to use because Splunk Is Not A Relational DB, and c) but you sort of *have* to use join simply because of how convoluted it would be to pile all these results into one table. But it's not necessarily the worst solution if (and only if!) there's only a single or only a few uri/fi's to actually do this for.  E.g. if it's a 1-1 mapping for a few dozen items, ... well, this is still inefficient but it's not going to be the end of the world.  A slight improvement would be to use append to get all the outputs into one table, then a final "| stats ... by fi" (or uri?)  to get the three sets of results back into a single row per fi/uri. Next best might be to rework the entire thing into one much bigger pile o' data, with a more complex stats that does all this work at once.  You'd likely have to do some conditional counting inside the stats, but though it may look a little odd, it's actually still going to be about a hundred times more efficient than your existing search, especially as it scales up to lots of uri/fi's.  https://docs.splunk.com/Documentation/Splunk/latest/Search/Usestatswithevalexpressionsandfunctions But I think best would be three simple searches with their results placed on a dashboard.  You'll have simpler searches that are much faster, far more easily edited/adjusted as things change, and likely way more robust as well. You'll also have a LOT more ways you could display the results. So a few other suggestions - I'll bet you are a DBA, are you not?  This looks similar to what someone familiar with any common RDBMS dropped into Splunk-land would do.  That's not a complaint or any slight, it's where lots of folks start.   In that case, you might want to start here: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/SQLtoSplunk But even in the docs, Splunk still suggests join in Splunk to handle the left join from SQL and that I think is a disservice.  See Nick Mealy's talk  https://conf.splunk.com/watch/conf-online.html?search=mealy (and even better if you can find the 2016-2018 version of that talk which I think is even better and more detailed) ... View more

Changing MAC address shouldn't probably affect anything, but changing IP addresses might. In any case, I'd start with your firewall - how is it configured to send syslog, to what address specifically? Is it actually doing so?  You basically just need to follow the path the data is supposed to take and find out where it's failing. That may lead directly to Splunk Cloud, with Splunk Cloud listening on a network port.  https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/HowSplunkEnterprisehandlessyslogdata In that case you might have to adjust the IP allow list in Splunk Cloud.  https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/Config/ConfigureIPAllowList It's also possible the device is sending its logs to a local syslog server, which has a splunk forwarder installed and which then sends the logs in to Splunk.  If that's the case, then the problem is most likely with the firewall - either as mentioned earlier it's not actually sending syslog, or I guess it's also possible in the firewall swap a rule or two has been missed. And if that's the case (FW -> syslog locally, Splunk forwarder -> Splunk Cloud) it's not likely anything in the last half of that is broken but you could check for something simple like a forwarder that just got all jammed up and confused and needs a restart. But as mentioned - start with your firewall's syslog settings and work your way through the syslog data flow and I'm sure you'll find it. ... View more

IMO this might get a lot more fiddly than you think.  If the system is sending in logs all the time (like, let's use for example they're Apache web servers), then even if you restart one, it'll come back online and catch up almost immediately.  You can take that farther - even if you turn off the forwarder for a week, when it gets turned back on it'll catch back up (possibly in just a couple of minutes!).    A week of "not reporting" just gets swept under the rug. If you actually have *gaps* in your logs, this is ... an entirely separate issue and you should solve that, because that's not a thing that should really happen and it's generally fixable. A bit better might be to use some shenanigans with the _indextime field, but it's also unlikely to be easily converted to a proper "wasn't sending in data" type report.  It might be closer, but it's still a lot of extra work. Even better than that might be to read some of the _internal indexes (the metrics one comes to mind) to find out there which 30 second periods it wasn't responding in (or whatever).  That would be more accurate. But possibly best when it comes to that sort of information might be just getting the forwarder's messages that say "I'm restarting" and "I've restarted".  THOSE would be easier to calculate a proper "downtime" from.  You can start here: index=_internal source=*splunkd.log* ("shutdown complete" OR "Splunkd starting") As long as your internal retention is long enough, that should get you each stop/start sequence - you'll have to then eval some fields, do some stats with a 'by host' and so on, but it should get close.  You could also use the above by host and | collect it to a summary index to keep just that information around for longer than retention on _internal would normally allow. It will NOT tell you if the host was actually offline though - if you disconnect the network cable... well, you should try it and see what logs show up for that!  I don't think it can tell you it disconnected, only that it reconnected but maybe there's a "seconds I was unable to talk to you" in that message too.  (Don't think so). Anyway, I'm sure that's a lot more words than you were expecting, but I wanted to explain the pitfalls of some of the first attempts people make for this sort of answer, since they don't often work that well.  🙂 Also happy to continue helping once you've explored a bit on your own, so if you get stuck ... post again in here! ... View more

I'm not completely clear on what the question is asking, but for the sake of a) getting discussion started to determine that and b) maybe this is what you are looking for anyway, ... The big search for total time - those last three lines, try changing them to the following 5. ... | stats sum(time_difference) AS TOTAL, count as COUNT | eval AVERAGE_TIME_PER_EVENT = TOTAL / COUNT | fieldformat TOTAL = strftime(TOTAL, "%H:%M:%S") | eval AVERAGE_TIME_PER_EVENT = tostring(AVERAGE_TIME_PER_EVENT, "duration") | table COUNT, TOTAL, AVERAGE_TIME_PER_EVENT  I don't think I have any typos in the above, but maybe they exist.  🙂 Anyway, all I do differently is add the count into your stats, eval a new field that's the total / count.  Then I leave your fieldformat the same, but right after that I introduce you to "tostring" with a type of duration, that should do what you want just like the fieldformat (H:M:S) but will ALSO add "days" to the front when appropriate (like 1+14:21:54 for 1 day plus 14 hours, blah blah.).  You can change both to eval or use fieldformat for both - it's fine either way. Lastly I added the two new fields to the table. If this is no where near what you want the answer to, ... well, maybe clarify your question a bit!  🙂 Happy Splunking, Rich ... View more

A guess! I see nowhere where you specify the date or time of the contents of the files, so it's being "automatically figured out" and that process often goes wrong. Try searching index=admin over *all time* for any fairly unique string that is in one of the files.  I'll bet you'll find them 6 days in the future, or somehow ingested as if they were from 2015. If that's not the problem then it might be helpful to have a snippet of the first bits of one of those files. Happy Splunking! -Rich ... View more