Hi, I'm trying to forward data into my Splunk indexer, but when I do a "./splunk list forward-server", it shows up under "Configured but inactive forwards". When I checked "splunkd.log", I see   AutoLoadBalancedConnectionStrategy [1676 TcpOutEloop] - Cooked connection to ip=<indexer_ip>:9997 timed out <-- 5 times, every 20 seconds TcpOutputProc [1675 parsing] - The TCP output processer has paused the data flow. Forwarding to host_dest=<indexer_ip> inside output group default-autolb-group from host_src=<forwarder hostname> has been blocked for blocked_seconds=8100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.   My forwarder is version 8.2.3 (on Ubuntu 20.04), my indexer is version 7.1.4 (on RHEL 7.4). I'm using a non-root account on my forwarder, and every time I run splunk or access the monitored folders, I need to run the commands using "sudo". I have tried to telnet from the forwarder to the indexer. On port 8089, I immediately get a "connection refused" message. On port 9997, there is no immediate response, the prompt just seems to wait for a long time at "Trying <indexer IP>...". I don't expect the telnet connection to be successful as my forwarder is behind a firewall that is very strict about which ports are allowed. But port 9997 (as destination) should be allowed on the network-level firewall. I also tried to check my "ufw". But I don't think it's running. > sudo ufw status Status: inactive > sudo systemctl status ufw.service ... Active: active (exited) since Monday 2021-12-06; 2h 59min ago ... > sudo ufw show listening tcp: 22 <forwarder IP> 8089 * udp: ... On the indexer, I have "/opt/splunk/etc/system/local/inputs.conf":   [default] host = <indexer hostname> [splunktcp://9997] disabled = 0   What is wrong with my setup? ... View more

Hi, I have set up a Splunk Enterprise instance (version 8.2.1) and a Universal Forwarder instance on Docker on the same machine, and I'm trying to forward data into the Splunk indexer. Here's what I have so far: On the Splunk Enterprise instance (1.1.1.1): Created an app named "abc" Created an index named "abc_idx" on app "abc" Created a sourcetype named "abc_data" on app "abc" On the Splunk forwarder: Added the indexer: "./bin/splunk add forward-server 1.1.1.1:9997" My very next command was "./bin/splunk add monitor /splunk_forward/log" Then I realized I wanted the monitored logs to be added to the index "abc_idx" and using the sourcetype "abc_data", so I removed the monitor, and then restarted the container. This is when I see the events appearing in the "main" index, so I believe the files did get forwarded. I then ran on the Splunk forwarder: ./bin/splunk add monitor /splunk_forward/log -index abc_idx -sourcetype abc_data But I did not see any event on the index "abc_idx". However, if I run the "oneshot" command, the events show up in the index "abc_idx" Is Splunk refusing to (re)index the same files again, even though they are going to different indexes? Also, I thought the commands I typed would end up in "/opt/splunkforwarder/etc/system/local/inputs.conf"? But I only see "[splunktcp://9997]" in it, not the folder I'm monitoring. Am I looking at the wrong file? However, I see the following in "/opt/splunkforwarder/etc/system/local/outputs.conf": [tcpout:default-autolb-group] server = 1.1.1.1:9997 [tcpout-server://1.1.1.1:9997] So why did my indexer configuration become part of the config file? Preferably, I would like to configure the forwarder using the config files, but I'm not sure exactly which ones to modify - local/inputs.conf and anything else? Thank you. ... View more

Hi, I wish to create text notifications for certain alerts, and email notifications for other alerts. Therefore, I would need to configure a SMS server and a SMTP server. However, it seems like only 1 email server is allowed to be configured under the "Mail server settings".  Is there a way to configure more than 1 mail server? Thank you. ... View more

Hi all, If I mouseover any charts, the default background color of the "popup" is black and the text is white/other colors. However, that makes it quite difficult to read what the text is sometimes. Is there a way to change the background color to white and the white text to black? Similar to the "popup" when I mouseover a choropleth map? Thank you. ... View more

Hi all, I'm trying to set the search period such that "earliest" is a specific day, and "latest" is 7 days after that. For this example, I chose "earliest" to be 159595200, which is Jul 29, and I would like the search period to be 7/29/20 12AM to 8/5/20 12AM. My query is     index=test earliest="1595952000" | eval latest=relative_time(earliest,"+7d") | ... | bin _time span=7d | dedup _time fieldA | stats count by _time fieldB     Splunk searches from 7/29/20 to 8/11/20 (i.e. now). It seems to ignore "latest". I've also tried      index=test | eval earliest="159595200" | eval latest=relative_time(earliest, "+7d") |...     In this case, Splunk searches from 7/29/20 to 7/30/20, i.e. only for 1 day. What is wrong with my query, and why is my "latest" ignored? Thank you. ... View more

I'm trying to create a timechart showing the count of events over 6 months. The query is index=itemdb `macrotest` (name != "*itemA" AND name != "*itemB") | eval category = case(...) | eval fields = split(name,"_") | eval mname = mvindex(fields,1) | search category = "promo" | dedup f_1 f_2 | timechart count by id span=1mon The goal is to dedup within that month only, not across all 6 months. For example, if the same values of f_1,f_2 appear in all 6 months, I should get 1 count of f_1,f_2 in each of the 6 months, not only in the last month. However, it seems like the f_1,f_2 values will be dedup across all 6 months, and appear only in the last month. Can I bin events by the months they appear in, then dedup within that month only to achieve this? Or is there another way? ... View more

I'm trying to create a table over a 6-month period, showing the number of items sold in each month (e.g. 10 ItemA & 20 ItemB in Jan, 15 ItemA & 10 ItemB in Feb, etc). However, when doing a comparison between the number of events returned and the values in the table, I found that there is a difference between the two. For example, the table shows 10 ItemA & 20 ItemB in Jan, but if I only query for Jan events, the actual number of events is 15 ItemA & 25 ItemB. My query is index=itemdb `macrotest` (name != "*itemA" AND name != "*itemB") | eval category = case(...) | eval fields = split(name,"_") | eval mname = mvindex(fields,1) | search category = "promo" | search mname = "itemC" | dedup f_1 f_2 | stats count by id _time My query above is searching only for a specific item, in an attempt to troubleshoot. If my time range is set to "Last 6 months", I get the following values in the table (showing only Oct, Nov and Dec here): Oct = 18, Nov = 10, Dec = 11 If my time range is set to "During X 2019", where X is Oct, Nov and Dec, i.e. I search specifically during that month only, I get the following number of events: Oct = 26, Nov = 14, Dec = 11 Missing events for Nov : 4 events on 30 Nov Missing events for Oct : 5 events on 31 Oct, and 3 events on 27 Oct I'm not sure why I get different results using the same query, if I change my time range. ... View more

I'm building a dashboard that shows a stacked column chart of different items sold in the last 6 months (using timechart ). For example, in Nov, there would be 2 Item A, 3 Item B, etc in the column for Nov. Then when I click on the block for Item A, I would get a table that shows the details of the 2 Item A in Nov only (using stats ). Part of my simple XML is as follows. <chart> <search> <query>... | timechart count by item span=1mon </query> <earliest>-6mon@mon</earliest> <latest>now</latest> </search> <drilldown> <set token="trend_item_earliest">$earliest$</set> <set token="trend_item_latest">$latest$</set> </drilldown> <chart> ... <table> <search> <query>... | stats count by item </query> <earliest>$trend_item_earliest$</earliest> <latest>$trend_item_latest$</latest> </search> </table> To troubleshoot this, I've reduced the queries to almost exactly the same for the 2 scenarios, except for the ...|timechart count by item and ...|stats count by item at the end. Running these 2 queries in separate searches (I simply clicked on the Magnifying glass icon on each panel to open the search separately), where the time range for timechart query is "Last 6 months", and the time range for stats query is "during Nov 2019", still gave me different results. However, from the table I got from the timechart query, if I click on one of the cells with discrepancy, e.g. the cell for Nov and Item A, and clicked "View Events", the corresponding search gave the correct number of events. Why am I getting different results? ... View more