Hello to everyone, on my indexers I just configured Splunk as a service with systemd, start command works fine but stop command (systemctl stop Splunkd), instead, returns some errors: [root@pe-sec-idx-02 system]# systemctl status Splunkd ● Splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start' Loaded: loaded (/etc/systemd/system/Splunkd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2022-01-12 15:18:32 CET; 9s ago Process: 1462 ExecStop=/opt/splunk/bin/splunk _internal_launch_under_systemd (code=exited, status=1/FAILURE) Process: 31484 ExecStop=/bin/sleep 10 (code=exited, status=0/SUCCESS) Process: 31225 ExecStop=/sbin/runuser -l splunk -c /opt/splunk/bin/splunk edit cluster-config -manual_detention on -auth admin:D1c3mbr3Sec (code=exited, status=0/SUCCESS) Process: 20750 ExecStartPost=/bin/bash -c chown -R 1001:1001 /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS) Process: 20746 ExecStartPost=/bin/bash -c chown -R 1001:1001 /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS) Process: 20643 ExecStartPost=/sbin/runuser -l splunk -c /opt/splunk/bin/splunk edit cluster-config -manual_detention off -auth admin:D1c3mbr3Sec (code=exited, status=0/SUCCESS) Process: 19156 ExecStartPost=/bin/sleep 60 (code=exited, status=0/SUCCESS) Process: 19155 ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd (code=exited, status=52) Main PID: 19155 (code=exited, status=52) Jan 12 15:12:20 pe-sec-idx-02 splunk[19155]: All installed files intact. Jan 12 15:12:20 pe-sec-idx-02 splunk[19155]: Done Jan 12 15:12:21 pe-sec-idx-02 splunk[19155]: Checking replication_port port [9887]: 2022-01-12 15:12:21.354 +0100 splunkd started (build 7651b7244cf2) Jan 12 15:13:17 pe-sec-idx-02 systemd[1]: Started Systemd service file for Splunk, generated by 'splunk enable boot-start'. Jan 12 15:18:03 pe-sec-idx-02 systemd[1]: Stopping Systemd service file for Splunk, generated by 'splunk enable boot-start'... Jan 12 15:18:16 pe-sec-idx-02 systemd[1]: Splunkd.service: control process exited, code=exited status=1 Jan 12 15:18:16 pe-sec-idx-02 splunk[19155]: 2022-01-12 15:18:16.021 +0100 Interrupt signal received Jan 12 15:18:34 pe-sec-idx-02 systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'. Jan 12 15:18:34 pe-sec-idx-02 systemd[1]: Unit Splunkd.service entered failed state. Jan 12 15:18:34 pe-sec-idx-02 systemd[1]: Splunkd.service failed. Despite the output, service stops successfully. As you can see, I added some instructions in the service unit file to put the indexer (which is part of a cluster) in manual detention before stopping it, and also it turns manual detention off once Splunk is started. I say again that stop/start commands work good, but in any case I get the above error messages when I stop the service. Am I doing something wrong? This is my service unit file: #This unit file replaces the traditional start-up script for systemd #configurations, and is used when enabling boot-start for Splunk on #systemd-based Linux distributions. [Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network.target [Service] Type=simple Restart=always ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd ExecStartPost=/bin/sleep 60 ExecStartPost=/sbin/runuser -l splunk -c '/opt/splunk/bin/splunk edit cluster-config -manual_detention off -auth admin:D1c3mbr3Sec' ExecStop=/sbin/runuser -l splunk -c '/opt/splunk/bin/splunk edit cluster-config -manual_detention on -auth admin:D1c3mbr3Sec' ExecStop=/bin/sleep 10 ExecStop=/opt/splunk/bin/splunk _internal_launch_under_systemd LimitNOFILE=64000 LimitNPROC=16000 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 User=splunk Delegate=true CPUShares=1024 CPUQuota=1400% MemoryLimit=30G PermissionsStartOnly=true ExecStartPost=/bin/bash -c "chown -R 1001:1001 /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R 1001:1001 /sys/fs/cgroup/memory/system.slice/%n" KillMode=mixed KillSignal=SIGINT TimeoutStopSec=10min [Install] WantedBy=multi-user.target   ... View more

Hello to everyone, my issue is that when I use sendemail in a scheduled search to send results via email in csv format, columns in the csv are not in the same order I tabled them in the search. For example: <some_search> | table field1 field2 field3 | outputcsv TestInvioMail_searchOutput.csv | stats values(recipient) AS emailToHeader | mvexpand emailToHeader | map search="|inputcsv TestInvioMail_searchOutput.csv | where recipient=$emailToHeader$ | sendemail sendresults=true sendcsv=true server=<my_email_server_address> from=<sender_server_address> to=$emailToHeader$ subject=\"Some object\" message=\"Some message\"" | append [|inputcsv TestInvioMail_searchOutput.csv] I need to use also map command because I have to send different results to different recipients, since inserting recipient token in the Splunk's mail alert panel doesn't work. Sendemail works fine, every recipient receives the correct results, but they receive a csv in which fields are in a different order respect to the one specified in table command (for example in the csv column order is field2 field3 field1). I also tried to add width_sort_columns=<bool> parameter in sendemail command (after sendcsv=true) but without success. Do you have any suggestion? Thanks in advance. ... View more

Hi all, tabled results from a scheduled search are sent via email as a csv attached. Some rows could be very long so in some cases, when I open that csv file with Excel, I find some "split rows", I would expect one unique line per row but instead sometimes I have half line positioned in the second column (as in the screenshot below). I'd like to obtain only one entire line per row,  so every event only in the first column of the Excel. The source search finds some events and tables some fields as result. Thanks in advance for any hint. ... View more

Hello to everyone, We'd like to set up a Splunk Test environment using the same license we are currently using in Production environment. The reason we want to do so is we want Test environment be representative of Production environment, so we need a distributed environment in Test too. Test environment will be on a different VLAN respect to the one on which Production is located. Is it possible to do so? Is it enough to only open firewall from test servers to IP and port (8089) of License Manager? Thanks in advance for any advice. ... View more

Hello to everyone, as in the title, we'd need to run scripts for putting Splunk search heads in manual detention (but also running other management CLI commands), and we're looking for a way to do that without writing admin's password in plain text inside the command, or leaving it written in user's history. Is there any way to do that? The only option I found right now is using -auth parameter, while instead the use of rest endpoint is not very clear and I'm not sure it fulfills my requests. Any help? Thanks in advance! ... View more

Hi all, we are monitoring some log files in a Windows directory; we'd like to keep only events containing the word FAILURE and discard the rest, so we set a filter in our two heavy forwarders but it doesn't work, we ingest all data and nothing is discarded. Here our configuration files: props.conf     [bpm_metastorm] DATETIME_CONFIG=CURRENT SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=UTF-8 disabled=false TRANSFORMS-set=bpm_null,bpm_parsing     transforms.conf     [bpm_null] REGEX = . DEST_KEY = queue FORMAT = nullQueue [bpm_parsing] REGEX = FAILURE DEST_KEY = queue FORMAT = indexQueue     There aren't other stanzas with same names as above, so it shouldn't be a name conflict problem; and we set another similar filter to another sourcetype, that worked while this new one not. Do you have any suggestions? Thanks in advance ... View more

Hi all, I have a cluster with 2 indexers, plus a cluster master in a different server. For some reasons that I don't know (since we've inherited the environment), the two indexers store hot/warm buckets locally and cold buckets in a NAS (same NAS for both indexers). We have to perform an activity on that NAS, so that it will be unavailable for a period of 3-4 hours.  My question is, putting the indexers in maintenance mode do I stop cold buckets from being generated? Or do you have any suggestion to perform this activity avoiding any data loss? Sorry but from Splunk documentation I can't find sure informations about this activity on both indexers at the same time. Thanks in advance. ... View more

Hi all, we have a Splunk Enterprise clustered environment, with a cluster of 3 search heads. For many reasons, a lookup file is updated once a day in only one of these search heads (the first one). To update this lookup file also in the other two search heads, we set up a scheduled search with the following string:     | inputlookup my_lookup_table.csv | outputlookup my_lookup_table.csv     Since if this search is run from a different search head than the number one the lookup is not updated, is it possible to run it always from the same search head? I know we could send the lookup via SFTP to the other search heads servers, but if possible we'd like to avoid it. Thanks in advance. ... View more

Hello to everyone, I have Splunk 7.3.1 with jQuery library version 2.1.0 and jQuery-ui version 1.10.4. Since they are affected from security bugs, is it possible to upgrade those libraries globally? I mean, I'd like that every file with a reference to those libraries will use the upgraded ones....possibly without modifying every single file 🙂 Thank you in advance for your answers/advices. ... View more