curl -k -u username:password https://127.0.0.1:8089/servicesNS/admin/search/saved/searches/abc/acl The servicesNS API (services namespace) explicitly uses user/app contexts (namespaces) to fetch results. servicesNS/<<user>>/<<app>> In order for the abc search to be shown, it must be visible to that <<user>> in that <<app>>. You can use the servicesNS/-/-/ endpoint to return all namespace contexts e.g: curl -k -u username:password https://127.0.0.1:8089/servicesNS/-/-/saved/searches/abc/acl    ... View more

Hi @klim, Unfortunately, not. | rest can only be used to perform GET requests. You cannot use any other HTTP method. If you want to do this with a Splunk search (and not a different program like curl) then I would consider a custom command / alert action. There is some old stuff kicking around on Splunkbase that might be able to help but I haven't seen anything that is supported on Splunk 9.x.   ... View more

I'll do some testing and get back to you later. ... View more

Sure no problem. This validation error pertains to the fact some of your files outside of the bin directory have execute permissions set. The way to fix this is to remove the execute permissions for any file that does not need to be executed. WARNING: This command will change file permissions so make sure that you only run this in your app directory (do not run this command elsewhere on your OS), firstly lets find all of the non-bin files in the app because these will not need to have execute permissions.   cd my_app find -type f -not -path "./bin/*"     Once you're happy that this is finding the correct files, we can then add the permission change:   find -type f -not -path "./bin/*" -exec chmod 644 {} +     Then we'll need to deal with the bin directory, if your bin is entirely Python based, then because Python is an interpreted language (e.g. the Python binary executes and reads the Python scripts) we can remove the execution permissions from the bin contents too:   find -type f -path "./bin/*" -exec chmod 644 {} +     The only thing in your bin that will need execution permissions are shell scripts and other executables. You can explicitly set these individually with a chmod (e.g. allow all users to execute and read my_executable):   chmod 755 ./bin/my_executable     ... View more

Your metrics indicate that your events are arriving, it's peculiar that you can't see them. @yeahnah calling out _time extraction is a great shout. Definitely explore those ideas. Could your events be older than your frozenTimePeriodInSecs? Queue routing is defined in transforms.conf and applied via props.conf. I'd check what props.conf apply to your sourcetype, host:: or source:: and make sure that there are no TRANSFORMS calls modifying the queue or index names. TCP routing is defined in inputs.conf but your post has ruled out those issues. The disabled/deleted index message is an actual warning presented within the Splunk GUI (not logged), look for a blue circle with a number in at the top of the GUI. Another thing to check is: have you blown your license for today? Seems trivial but its worth checking off. Finally, does your user have permissions to read that index? If nothing turns up, I would also suggest a restart of Splunk Enterprise just to rule it out. ... View more

How are you updating the geoip files currently? ... View more

That's a lot of errors... Start with the Cloud Appinspect Criteria. Hopefully something obvious jumps out. If that doesn't help then it's going to be very hard to help debug this without some more details about what your app contains. Feel free to share the (obfuscated) config within the app and the high-level structure too. If you (understandably) don't want to share this on here then I would raise a ticket for Splunk support. ... View more

Your deployment and subsequent troubleshooting steps all look absolutely spot-on so far! You're right in thinking that you don't need the local inputs.conf on your Splunk Enterprise instance. I suspect the issue here could be down to a slight misconfiguration or typo. Is the index definitely enabled? Could the UF be forwarding to a different Splunk Instance? i.e. Have you got any other outputs.conf servers on your UF? Have you got any routing/index renaming props/transforms applied on Splunk Enterprise? Could this be a typo, on your Splunk Enterprise messages can you see this warning? "Received event for unconfigured/disabled/deleted index" If so make sure your index name matches your inputs.conf exactly. ... View more

This error has came up before a few times on the Splunk community. How to get WMI data collection by providing to spl... - Splunk Community Solved: Getting Error When trying to read windows event l... - Splunk Community The TL;DR is that the service account running the splunkd service on the forwarder host needs to have specific permissions set (on both the local and domain level). This Splunk Documentation does a good job of taking you through the required permissions and also the appropriate security considerations. ... View more

Hi @sudderthc, First and foremost, this add-on is supported by Tenable so I would raise a case with them if you can't get this going. Looking at tenable_account_validation.py that error is unfortunately just as vague as it looks. Funnily enough, the ConnectionError exception that's raised to trigger that exception is decently verbose but for some reason the TA Devs have decided to mask it with that generic error you're seeing. If you're comfortable with it, you could temporarily modify the TA to include the verbose exception details. I've also included some more generic troubleshooting steps to help if you'd rather not touch the code. Making the TA Validation More Verbose: Modify tenable_account_validation.py lines 221-223 From:   except Exception: msg = "Please enter valid Address, Username and Password or configure valid proxy settings or verify SSL certificate." raise Exception(msg)   To:   except Exception as e: msg = "Please enter valid Address, Username and Password or configure valid proxy settings or verify SSL certificate. Verbose Details: " + repr(e) raise Exception(msg)   Then try to add your account again. It should now include the verbose details about what went wrong. High-Level Troubleshooting: If I were troubleshooting this on the Splunk host, I would step-through the following: 1. Establish basic connectivity:   curl -k https://tenablesc_host:tenablesc_port/rest/system   If you're behind a proxy you can use the --proxy curl switch. Do you get a response? If not then you'll need to facilitate this connectivity first. 2. Validate SSL Settings: (This is only applicable if you're using verify_ssl=true, if not then skip to 3.)   curl https://tenablesc_host:tenablesc_port/rest/system   Does the same command work with SSL Verification (no -k)? If not, then you're probably using a self-signed certificate for your Tenable SC instance. Tenable does not support self-signed certificates on their SDK, but you could modify the TA slightly to support it. A quick look through the code shows that it leverages the restfly library so you could update the APISessions base class to support your self-signed certificate. Do this as your own risk, it will be a trade-off between SSL verification and upstream support. 3. Validate Credentials If you've managed to establish connectivity then I would test if your credentials are working properly.   curl https://tenablesc_host:tenablesc_port/rest/system -H "X-APIKey: accessKey=123; secretKey=123;"   (Replacing 123 with the appropriate keys) does this return the same content as the previous commands? If not and you receive a login denied message instead, validate that your credentials are correct and API key login is enabled on the SC instance. Good Luck, let me know how you get on! ... View more

Hi Jason, Unfortunately, add-ons are typically not built with this application-layer redundancy in mind and the Splunk Add-On for Cisco UCS is no exception. Your best bet would either be to manually build this resiliency into the TA itself, or sync the configuration between your forwarders using some kind of automation tool. ... View more

Hi @redhonda03_2  Just to add to this, the reason it's a struggle to get the regex going, is probably the backslashes giving you grief. The backslashes within search regex need to be escaped at the search layer and at the regex layer too. You need to triple escape the backslashes. This solution will work with both conventional lettered-drives and also UNC paths:   | rex field=FilePath "(?i)(?<parent>(?:[A-Z]\:|\\\\{2}[^\\\\]+)\\\\[^\\\\]+\\\\)"   Using an | eval like in @yeahnah's solution is definitely more readable and probably more practical too, just be mindful for any UNC paths. ... View more

Hi @adsquaired, I understand what you're saying. I'm not necessarily suggesting that you change the span value in the mstats command. The | mstats is calculating the average rate on a 5-minutely basis. To aggregate the averages how you suggest, you'll need to use an additional command. | timechart is going to be the one for this. For example, this will give you the maximum average rate across 5 minute windows for each week.   | mstats rate_avg("octets.*") WHERE index="network" chart=true host="device-*" span=5m by host | fields - _span* | rename "rate_avg(octets.rx): *" AS "in * bit/s" | rename "rate_avg(octets.tx): *" AS "out * bit/s" | foreach * [eval <<FIELD>>='<<FIELD>>' * 8 ] | timechart span=1week max(*) as *   When layering statistics on top of each other like this, it's important to bare in mind what they're doing. This is not the maximum rate at any given moment, it's the maximum average. ... View more

Hi @klay824, Your example looks good but going back to my previous post, the message.payload's won't be overriding each other because they are in totally independent searches. With this search, if message1 and message2 occurred at the exact same _time (down to the millisecond) then they will be grouped together. Try this search, which uses a where at the end to filter down to where message1 and message2 are populated at the same _time: index="example" TERM(STOP) | rename message.payload as message1 | table _time, message1 | append [| search index="example" TERM(index) | rename message.payload as message2 | table _time, message2 ] | stats values(message1) as message1, values(message2) as message2, count by _time | where isnotnull(message1) AND isnotnull(message2)   ... View more

Can you share the output of:   ls -l /bin/sudo   Sudo needs to have the suid permission set to run as root. Could this have been unset? Whilst you're there (as the Splunk user), does your sudo command execute properly when ran directly on the CLI (instead of via the scripted input). Also can you share some more details about the OS please? ... View more

Yes, This can be achieved in both Dashboard Studio and Simple XML Dashboards via a drill-down. Select the Visualisation > Scroll-Down the Configuration Menu > Add Drill-Down > On-Click >Link to Dashboard. You can read more here. ... View more

Yes, you've hit the nail on the head when you talk about summarising the data. You'll need to decide how to aggregate or roll-up your search into larger spans. For example do you want to display the average for that range? Or the max? Either way, this can all be decided dynamically using a | timechart and subsearch in tandem: In this example, the avg() will be taken per week if the search is a year long, or per day if the search is a month long (otherwise the default is 5min). | timechart [| makeresults | addinfo | eval search_range = (info_max_time - info_min_time)/(24*3600) | eval search = "avg(*) as * " + case(search_range>=365, "span=1week", search_range>=30, "span=1day", 1=1, span="5min") | table search]   ... View more

I've done some more testing, this time on CentOS and I'm experiencing similar symptoms to you again. $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes is exiting with a 141 (just like tar was on Ubuntu) so it's failing to start. There must be a niche difference between the way Splunk is starting on the two OSs that explains why it started without a problem on Ubuntu. That being said, the change to wrapper.sh from my original answer (redirecting the STDOUT and STDERR away from Splunk) solved this issue again. Based on your original Scripts, here is what is working for me: wrapper.sh #!/bin/bash ( /opt/splunkforwarder/etc/apps/splunk_upgrade_lin_v8/bin/upgrade.sh & ) > /dev/null 2>&1   upgrade.sh #!/bin/bash # set splunk path SPLUNK_HOME=/opt/splunkforwarder # set desired version NVER=8.2.2 # determine current version CVER=`cat $SPLUNK_HOME/etc/splunk.version | grep VERSION | cut -d= -f2` if [ "$NVER" != "$CVER" ] then echo "Upgrading Splunk to $NVER." $SPLUNK_HOME/bin/splunk stop tar -xvf $SPLUNK_HOME/etc/apps/splunk_upgrade_lin_v8/static/splunkforwarder-8.2.2-87344edfcdb4-Linux-x86_64.tgz -C /opt $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes fi   If you still can't get it to work, let's add some debugging to a /tmp/ file to check what's going on. Here is a modified upgrade.sh that will output some more verbose logs. upgrade.sh (with logging) #!/bin/bash LOG=/tmp/splunk_upgrade.log echo "Starting Splunk Upgrade." >> $LOG # set splunk path SPLUNK_HOME=/opt/splunkforwarder # set desired version NVER=8.2.2 # determine current version CVER=`cat $SPLUNK_HOME/etc/splunk.version | grep VERSION | cut -d= -f2` echo "Current Version: $CVER, Target Version: $NVER" >> $LOG if [ "$NVER" != "$CVER" ] then echo "Proceeding with upgrade." >> $LOG echo "Upgrading Splunk to $NVER." echo "Stopping Splunk." >> $LOG $SPLUNK_HOME/bin/splunk stop 2>>$LOG echo "Stopping Splunk returned exit code: $?." >> $LOG echo "Unpacking Splunk." >> $LOG tar -xvf $SPLUNK_HOME/etc/apps/splunk_upgrade_lin_v8/static/splunkforwarder-8.2.2-87344edfcdb4-Linux-x86_64.tgz -C /opt 2>>$LOG echo "Unpacking Splunk returned exit code: $?." >> $LOG echo "Starting Splunk." >> $LOG $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes 2>>$LOG echo "Stopping Splunk returned exit code: $?." >> $LOG fi echo "Complete." >> $LOG   Please share a copy of these logs and we'll see if there's anything different happening on your system. ... View more

Your best bet is going to be the splunkd_access sourcetype. index="_internal" sourcetype="splunkd_access" host="<<SPLUNKHOST>>" | stats values(user) as user | mvexpand user That being said, if you're auditing a SH, you're going to see lots of traffic from splunkweb. To address this you could filter out the Splunk user agent (the risk with this is that user-agents can be modified):   index="_internal" sourcetype="splunkd_access" host="<<SPLUNKHOST>>" | regex useragent!="Splunkd?\/[\d\.]+ \(" | stats values(user) as user | mvexpand user    Or filter out any localhost connections: index="_internal" sourcetype="splunkd_access" host="<<SPLUNKHOST>>" clientip!="127.0.0.1" | stats values(user) as user | mvexpand user​ ... View more

Ah-ha, I understand. Depending on exactly what you're trying to acheive. If you truly are looking to use TERM in the conventional sense to group the message.payload properties, then you can let Splunk natively take care of the different TERMs via a subsearch:   index="example" TERM(STOP) | eval message1 = src_ip | table _time, message1 | append [| search index="example" TERM(index) | eval message2 = src_ip | table _time, message2 ] | stats values(message1) as message1, values(message2) as message2, count by _time If you know that your TERMs will always have a consistent prefix/suffix, e.g. always surrounded by a space, then you could use the LIKE command as follows:   index="example" (TERM(STOP) OR TERM(index)) | eval message1=if(LIKE(_raw,% STOP %), message.payload, null()) | eval message1=if(LIKE(_raw,% index %), message.payload, null()) | stats values(message1) as message1, values(message2) as message2, count by _time     ... View more