I have syslog events being written to a HF locally via syslog-ng - these events are then consumed via file reader and the IP address in the log name is extracted as host. I now want to run an ingest_eval on the ip address and use a lookup to change the host If i run the cmd from search i get the required result: index=... | eval host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value") this replaces host with "host_value" I have this working on an AIO instance with the following config below: Now adding to HF tier : /opt/splunk/etc/apps/myapp/lookups/lookup.csv lookup has global access and export = system host,host_value 1.2.3.4, myhostname props.conf: [mysourcetype] TRANSFORMS-host_override = host_override transforms.conf: [host_override] INGEST_EVAL =host=json_extract(lookup("lookup.csv",json_object("host",host),json_array("host_value")),"host_value") When applied on the HF (restarted)  i see some of the hostnames are changed to "localhost" the others remain unchanged (but this is due to the config not working OR the data coming from another HF with the test config not applied Any ideas - thx ... View more

I'm looking to add some column formatting to some table in dashboard studio - but the option is greyed out saying the column is an array, why is this ? and can i re-factor my search to make it work? index=test AND host="test" sourcetype=test | stats latest(state) latest(status) by host name state status | stats list(name) as NAME list(state) as STATE list(status) as STATUS by hos ... View more

am i missing something with how dbconnect works ? i have dbc on a HF and if i create a search that creates a lookup - that lookup will reside on the HF - there is no mechanism that moves it into the SHC so it can be used ? Similarly i can't run a | dbxquery from the shc - right ? gratzi ... View more

Is it possible to monitor whether processes are running using metrics data and SAI ? I want to push out a config via UF to say monitor these X processes - and alert should any of them stop ? gratzi ... also - i have a single UF reporting to Splunk - my SAI "Overview" dashboard - looks like this (last 15 mins) UPTIME(h:m:s) top 10547 root 0 0.2 00: 00: 00 top 10817 root 0 0.2 00: 00: 00 top 11789 root 0 0.2 00: 00: 00 top 13036 root 0 0.2 00: 00: 00 top 14295 root 0 0.2 00: 00: 00 top 17779 root 0 0.1 4+18: 58: 48 What is this telling me - i only have one instance of top running - which shows as 4days+ - where are all the other pids coming from? ps -ef | grep -i top root 17779 24995 0 Feb21 pts/1 00:03:02 top root 30528 26798 0 12:27 pts/0 00:00:00 grep --color=auto -i top # ... View more

I have 23000 (yes 000) directories in /opt/splunk/var/tmp/data I can't find info in docs as to what this is - how best to find out ? each directory has files in such as : part-00000.gz part-00001.gz part-00002.gz part-00003.gz part-00004.gz part-00005.gz ... View more

I have some blob storage and in there are different files that I need to ingest and apply different source types to. eg. some are error.log files some are web access logs some are other logs How do I do this ? Gratzi. ... View more

It's working fine in an AIO scenario .. Moved to a cluster and the monitors are running on the Heavy Forwarder and populating my indexes however nothing populating the app on the Search Head Cluster. All my inputs are on the HF. I feel I'm missing a silly step - please help! gratzi. ... View more

Trying to replicate thresholds from a legacy tool in ITSI that are configured over time periods How would you create a KPI which alerts if CPU is over 95% for 15 minutes? gratzi ... View more

I have a syslog file and none of the default sourcetypes give me what i want - so i have: any advice on best approach for props.conf Apr 15 16:54:01 HOSTNAMEX Group CfgSrvc: hd[0]: cfgcore: WritePhase2(Security,system,security.authentication.accounts[0].adminaccount.failedlogincount) value update "9234" => "9235" Apr 15 16:54:01 HOSTNAMEX Group CfgSrvc: hd[0]: cfgcore: callback DynamicValidate(Security,"807847",system,{security[0].authentication[0].accounts[0].adminaccount[0].failedlogincount[0]/local,"9235"},"PlatCfgS/5/01-0"[17]) Apr 15 16:54:01 HOSTNAMEX Group Security: hd[0]: SecurityService: login login failed, increment # of failed logins Apr 15 16:54:01 HOSTNAMEX Group Security: hd[0]: SecurityService: login login failed, setLoginResult 6 Apr 15 16:54:01 HOSTNAMEX Group Security: hd[0]: SecurityService: If loginSuccess is false ( Apr 15 16:54:01 HOSTNAMEX Group Security: hd[0]: SecurityService: LocalAuthenticator::login, role 3 loginSuccess 0 Apr 15 16:54:01 HOSTNAMEX Group Security: hd[0]: SecurityService: SecurityServiceLoginRequest(): username: admin Apr 15 16:54:01 HOSTNAMEX Group logcat: hd[0]: UAppSvcs(2132): (legacyapi) API:OUT:Password: Apr 15 16:54:01 HOSTNAMEX Group logcat: hd[0]: UAppSvcs(2132): (legacyapi) API:OUT: -- password failed, retry -- gratzi ... View more

At this time if i run this over 4 months the x axis shows Month, day - all i want to see is month. And can this search be improved ? gratzi index=response source=responsetimes | table _time, ACTION_TIME | sort - ACTION_TIME | convert rmcomma(ACTION_TIME) | eval ACTION_TIME = (ACTION_TIME/1000) | timechart avg(ACTION_TIME) as "Average" span=1d | append [search index=response source=responsetimes | table _time, ACTION_TIME | sort - ACTION_TIME | convert rmcomma(ACTION_TIME) | eval ACTION_TIME = (ACTION_TIME/1000) | eventstats p5(ACTION_TIME) as top5perc | where ACTION_TIME < top5perc | timechart avg(ACTION_TIME) as "Top 5%" span=1d] | append [search index=response source=responsetimes | table _time, ACTION_TIME | sort - ACTION_TIME | convert rmcomma(ACTION_TIME) | eval ACTION_TIME = (ACTION_TIME/1000) | eventstats p95(ACTION_TIME) as bottom5perc | where ACTION_TIME > bottom5perc | timechart avg(ACTION_TIME) as "Bottom 5%" span=1d] | timechart first(*) as * ... View more

OK so its not supported - but have a handfull of servers that i'd like to get a fwd on .. installed the latest version for 2000 5.0.18 - its talking to the IDX but seeing a few errors for the DS : 02-13-2019 01:11:07.396 -0500 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: not_connected host = win2000 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd 13/02/2019 17:10:50.827 02-13-2019 01:10:50.827 -0500 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: not_connected host = win2000 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd 13/02/2019 17:06:13.201 02-13-2019 01:06:13.201 -0500 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: not_connected host = win2000 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd 13/02/2019 17:06:18.328 02-13-2019 01:06:18.328 -0500 INFO HttpPubSubConnection - Could not obtain connection, will retry after 60 seconds. host = win2000 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd 13/02/2019 17:06:25.208 02-13-2019 01:06:25.208 -0500 WARN DeploymentClient - Unable to send handshake message to deployment server. Error status is: not_connected host = win2000 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd IS this because the old UF uses TLS 1.0 wheres as we are now on TLS 1.2 This looks similar to : https://answers.splunk.com/answers/713063/if-i-upgrade-to-splunk-enterprise-70-can-i-recieve.html Is it possible ? Gratzi ... View more

Hi Some of the dashboards are missing from the previous versions Billing, Azure AD & the nice Topology feature ? Can these be re-added ? Or access to the previous versions as i over wrote it? gratzi ... View more

I have the template installed on a SH and the Azure collection apps (MSCS, Azure Monitor etc) are all installed on on a HF If i run the search to list the subscriptions on the HF all good as thats where the config is .. but running on the SH displays no answer. and an error Failed to fetch REST endpoint uri=https://127.0.0.1:8089/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/configs/conf-mscs_azure_resource_inputs?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API. Learn More Failed to fetch REST endpoint uri=https://127.0.0.1:8089/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/configs/conf-mscs_azure_audit_inputs?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API. Learn More search is : | rest /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/configs/conf-mscs_azure_resource_inputs | append [ | rest /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/configs/conf-mscs_azure_audit_inputs] | stats count by subscription_id So i tried adding the HF as a remote host but i still do not get any subscriptions showing : | rest splunk_server=remote_HF /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/configs/conf-mscs_azure_resource_inputs | append [ | rest splunk_server=remote_HF /servicesNS/nobody/Splunk_TA_microsoft-cloudservices/configs/conf-mscs_azure_audit_inputs] | stats count by subscription_id ... View more