In forwarder management I get a message stating there are 6 clients with "DEPLOYMENT ERRORS" but cannot find the issue. Searched the _internal index but still do not see what the errors are.
Where can I find the client errors?
Here is a dashboard I have made to find these types of error.
<form version="1.1" theme="dark">
<label>Deployment status</label>
<!--
1.0
1.1 change name 19.12.2019
-->
<search id="base_search">
<query>
index=_internal OR index=*_internal
sourcetype=splunkd
host="$Host$"
name="$Server$"
sc="$Stansa$"
app="$App$"
result="$Result$"
action=Download
| table _time host name sc app result
</query>
</search>
<fieldset submitButton="false">
<input type="time">
<label></label>
<default>
<earliest>-24h@h</earliest>
<latest>now</latest>
</default>
</input>
<input type="dropdown" token="Host">
<label>Deployment server</label>
<search base="base_search">
<query>
| eval data=host
| stats count by data
| eval info=data." (".count.")"
| sort -count
</query>
</search>
<choice value="*">Any</choice>
<fieldForLabel>info</fieldForLabel>
<fieldForValue>data</fieldForValue>
<default>*</default>
</input>
<input type="dropdown" token="Server">
<label>Server</label>
<search base="base_search">
<query>
| rex field=name "bit_(?<server>[^_]+)"
| eval data=name
| stats count by data server
| eval info=server." (".count.")"
| sort -count
</query>
</search>
<choice value="*">Any</choice>
<fieldForLabel>info</fieldForLabel>
<fieldForValue>data</fieldForValue>
<default>*</default>
</input>
<input type="dropdown" token="Stansa">
<label>Stansa</label>
<search base="base_search">
<query>
| eval data=sc
| stats count by data
| eval info=data." (".count.")"
| sort -count
</query>
</search>
<choice value="*">Any</choice>
<fieldForLabel>info</fieldForLabel>
<fieldForValue>data</fieldForValue>
<default>*</default>
</input>
<input type="dropdown" token="App">
<label>Application</label>
<search base="base_search">
<query>
| eval data=app
| stats count by data
| eval info=data." (".count.")"
| sort -count
</query>
</search>
<choice value="*">Any</choice>
<fieldForLabel>info</fieldForLabel>
<fieldForValue>data</fieldForValue>
<default>*</default>
</input>
<input type="dropdown" token="Result">
<label>Result</label>
<search base="base_search">
<query>
| eval data=result
| stats count by data
| eval info=data." (".count.")"
| sort -count
</query>
</search>
<choice value="*">Any</choice>
<fieldForLabel>info</fieldForLabel>
<fieldForValue>data</fieldForValue>
<default>Fail</default>
</input>
</fieldset>
<row>
<panel>
<chart>
<search base="base_search">
<query>
timechart count by name limit=10
</query>
</search>
<option name="charting.chart">column</option>
<option name="charting.chart.stackMode">stacked</option>
</chart>
</panel>
</row>
<row>
<panel>
<table>
<search base="base_search">
<query>
stats count by host name sc app result
| sort result
| rename host as "Deplyment server" name as Server sc as Stansa app as Application
</query>
</search>
<option name="count">100</option>
<format type="color" field="Deplyment server">
<colorPalette type="sharedList"></colorPalette>
<scale type="sharedCategory"></scale>
</format>
<format type="color" field="Server">
<colorPalette type="sharedList"></colorPalette>
<scale type="sharedCategory"></scale>
</format>
<format type="color" field="Stansa">
<colorPalette type="sharedList"></colorPalette>
<scale type="sharedCategory"></scale>
</format>
<format type="color" field="Application">
<colorPalette type="sharedList"></colorPalette>
<scale type="sharedCategory"></scale>
</format>
<format type="color" field="result">
<colorPalette type="map">{"Fail":#DC4E41,"Ok":#53A051}</colorPalette>
</format>
</table>
</panel>
</row>
</form>
This is my way of finding out who is that has issue:
1st , search this in deployment server:
index=_internal sourcetype=splunkd record (New OR Updating) result=Fail | head 100
You should be able to see name of the client along with application and server class.
you can get the system name of the server, by Settings > Forwarder Management > Clients Tab, then paste name of the client.
You could continue your troubleshooting from there.
This works. Thanks!
Thanks!!! Splunk should implement this...
This answer greatly helped, thanks.
This is one reason I am starting to NOT like Splunk many unanswered questions. I too am having this problem.
run the search
index=_internal sourcetype=splunkd fail
The return will have information to narrow the search for the clients that have problems.