How Splunk indexers operations works when it comes into manual detention state ? We are migrating from RHEL 6 - RHEL 8 and here can't do OS upgrade on the same machine , so we will be getting new REHL 8 machines and then we need to install Splunk there and work accordingly. Current setup : 5 indexers (RHEL 6) in indexer cluster , S.F -1 , R.F -1. Our plan is to add new 5 indexers (RHEL 😎 in indexer cluster and route all sources to send data to newly added indexers , once that is complete will make sure any of the source is not sending anything to old indexers. Enable manual detention on old indexers so that they dont replicate their data to other indexers (newly added), was going through manual detention documentation (below) and found only four operations are concerned  if peer is in manual detention. stops replicating data from other peer nodes. optionally stops accepting data from the ports that consume external data, causing the peer to no longer index most types of external data. continues to index internal data and stream the data to target peer nodes. continues to participate in searches. https://docs.splunk.com/Documentation/Splunk/8.0.5/Indexer/Peerdetention But I would like to know if peer is in manual detention will it be rolling over the stored indexes data as per retention policy to frozen DB ? Or except above 4 points, all other operation will be as usual even if peer is in detention mode. Why am I concerned because in my case we have hot DB (local mount - 700 GB data on each peer) and cold DB (NAS mount - 2.3 TB each peer) and we are going to share  same mount for new peers as well with creating separate subfolders from NAS (for new peer). Second thing how my S.F and R.F behaves since its 1:1 , because most of the data will be with old peers and those would be in detention mode , are S.F & R.F gonna met , is there any impact in searches. Final step would be run command splunk offline --enforce-count to decommission old peer one by one , but that all depends on above questions(if peer is in manual detention will it be rolling over the stored indexes data as per retention policy to frozen DB ?) , if peer (enabled detention mode) is not purging the data as per retention period then we would not have enough space in NAS storage (cold DB) to store the data which is coming after rolling over form hot DB from new peer.  If old peers (enabled detention mode) continue to purge their data as per retention period in that way will have space cleared from one side and new data getting added to NAS (cold DB) from another side. If all this works , i dont know if R.F is set to 1 means each peer will have one copy for data, if some peer doesnt have , i think they will create new replicate copies when we do decommission. Please suggest          ... View more

configure webhook to receive event from third party application. Configured it from data inputs , but we are not receiving any data. Provided Splunk WebUI and port 8081 to azure team to configure webhook action and send events but it's not working  any suggestions ? ... View more

how much storage we can save by enabling frozen bucket and roll over data from cold bucket. We have tons of data coming in and instead of storing data in cold db for 1 year ,we want to store it in cold db for 3 months max and older than that we will move to frozen bucket , but by enabling that we would like to understand how much storage we can save if we store 9 months data in frozen and retrieve it whenever required. Example: you can take 3 TB coming in per day ,we have 14 indexers ,multi site cluster (R.F-2, S.F-1). We would like to keep data till 3 months (max) in hot\warm\cold DB then we would store that frozen DB. what would be compression factor if we do that , is it gonna save some storage or it will consume same storage that ... View more

microsoft azure add-on for Splunk is unable to pull ad risky sign-on logs if we look for internal logs , getting below mentioned events frequently , didn't see any issue but still we are not seeing any data. 2019-12-23 14:44:18,779 DEBUG pid=28967 tid=MainThread file=connectionpool.py:new_conn:809 | Starting new HTTPS connection (1): graph.microsoft.com 2019-12-23 14:44:18,772 INFO pid=28967 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled! 2019-12-23 14:44:18,772 DEBUG pid=28967 tid=MainThread file=base_modinput.py:log_debug:286 | _Splunk Getting proxy server. 2019-12-23 14:44:18,772 DEBUG pid=28967 tid=MainThread file=base_modinput.py:log_debug:286 | Splunk nextLink URL (@odata.nextLink): https://graph.microsoft.com/beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+gt+2019-12-22T14%3a42%3a44.517899Z+and+createdDateTime+le+2019-12-23T14%3a35%3a44.819576Z&$skiptoken=**************************_17000 2019-12-23 14:44:18,134 DEBUG pid=28967 tid=MainThread file=connectionpool.py:_make_request:400 | https://graph.microsoft.com:443 "GET /beta/auditLogs/signIns?$orderby=createdDateTime&$filter=createdDateTime+gt+2019-12-22T14%3a42%3a44.517899Z+and+createdDateTime+le+2019-12-23T14%3a35%3a44.819576Z&$skiptoken=*****************************_16000 HTTP/1.1" 200 None ... View more

Splunk DB connect is not working for DB2 database. Installed drivers for DB2 : db2jcc.jar db2jcc_license_cisuz.jar db2jcc_license_cu.jar PROGRESS_DATADIRECT_JDBC_DB2_ALL.jar Getting license error : [ibm][db2][jcc][10333][11649] No license was found. An appropriate license file db2jcc_license_*.jar must be provided in the CLASSPATH setting. I tried to update classpath for license file but then Splunk restart , nothing is working, please help... export CLASSPATH="$SPLUNK_HOME/etc/apps/splunk_app_db_connect/drivers/db2jcc-libs/db2jcc_license_cu.jar" Tried for another license file as well but again error is same ... View more

Splunk cluster indexers are consuming high memory. Memory usage on indexer server is always at 99% used, after restarting splunk it's coming down but within one minute again reaching at 99%. Nothing coming in logs which indicates if anything causing this. Also on same indexers internal_db is filling so quickly, are both issues related to each other. Any suggestions? We have 23 GB memory aligned to each indexer (total 5 in cluster) and we are logging around 400 -500 GB data on this environment. Splunk version 7.2.3. One more thing , is this know issue after upgrading to 7.x.x from 6.x , because while env were on 6.5.3 then we didn't face memory related issue but on that time we were logging around 300 GB data and memory aligned was 12 GB per indexer. ... View more

after upgrading forwarder to 7.2.6 it's not getting controlled by Splunk user(specifically aligned to Splunk only (non-root user)) while restarting service. We upgrade Splunk UF to 7.2.6 from 6.x.x , everything is working as expected but while stop\start splunk service it's asking for authentication (mentioned below). And this message coming only once we enable boot start for Splunk user so that It can auto start after reboot. If we disable boot start then I am not getting these messages. [user@servername ~]$ /usr/splunk/splunkforwarder/bin/splunk restart Stopping splunkd... Shutting down. Please wait, as this may take a few minutes. ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Multiple identities can be used for authentication: And you will get multiple user identities here after this above line, these are the user who's ID is synced with root user. And if I ask them to do they are able to restart Splunk but they have to choose their username and password , so to add splunk user here in identities list what we need to do. Is there a way to get rid of this. Splunk UF version - 7.2.6 OS version - Red Hat Enterprise Linux Server release 7.6 (Maipo) Do we need to tweak splunk configuration or make any entries in sudoer files on OS side. ... View more

Splunk is not generating alert for normal stats count output 7.0.0. index=my_index "Response code -401" | stats count - after that I am saving this as result and keeping TH when it's count is greater than 10 , I need alert, but it's not working , not triggering alert after breaching TH. all other conditions are configured properly. Is this common issue with stats command that we can't have alert for any string count ? ... View more

Range map is not working properly in Splunk 6.5.3. Below is my search query, When value is above 90 then it says green color which is correct, and when value is below 90 then other two defined (elevated & severe) are not working, if its (health field) range is between the two values then it's color getting merged, like green color is default there and when red color should come then it's getting merged and color showing is black my search|rangemap field=Health low=90-100 elevated=70-89 severe=0-69 please do needful, is this due to decimal values which is coming for health field? I tried to remove decimal values and kept above same range but still didn't work. ... View more

I have report scheduled to run at 12 AM EST to search for last 7 days and just provide stats count for user ID's like below: my search | stats count by date_wday -it will display week day field and then count of events corresponding to that. But for last one day it's giving exact count and and for other 6 days it seems to be average values not exact count. one more thing each count can be between 10000 - 20000 , is this limitation in Splunk that we can have it for one day only and after that it will be taking average for other days. ... View more