Splunk Deployment Monitor 4.2.2 is not working anymore

[LCM]

I simulate client, forwarder and indexer on the same machine (different ports and so) -> REHL 5.6

Somehow now, my Splunk Deployment Monitor 4.2.2 (installed on the indexer part) is not working anymore! I guess something's wrong with the summary indexing, since all summary* buckets are empty.

The license is an Enterprise Trial one and still valid (info from Manager -> Licensing)

I appreciate any hints, thanks

[sdwilkerson · Accepted solution]

LCM,

The Deployment Monitor runs many searches and then writes summaries of these searches out to a handful of special summary indexes. Using it in a Splunk Distributed environment (as you are doing) is very useful but you must consider a few things.

The Deployment Monitor should be running on just the search head. If this Search Head has a default configuration, it will be storing the summary data locally (on indexes that are on the search head and were created by the indexes.conf that ships with Deployment Monitor). There is nothing wrong with doing it this way, but there are other ways (mentioned in a minute).
If your system is setup this default way, then you should check the summary indexes on your search head to see if they contain fresh data. If they do contain fresh data, that means it is being collected, but you just aren't seeing it and that is a different issue. If they do not contain fresh data then your app is not collecting this data. If it is not collecting the data then look into why this might be: Check the scheduled searches specific to the Deployment Monitor and see if any are scheduled to run, If any searches ran recently click on the "view recent" and see if there are results there, Look at one of the queries made by the deployment monitor that it uses to build the summaries and see if you get any results when this runs.

An alternate configuration that I prefer to use for a normal distributed Splunk environment is to have your search head forward its events on to your Indexers (or preferably a cluster of Heavy Forwarders for large-scale deployments). In this environment, your Search Head would run the query against the indexers, then it would generate the data that should be indexed in the summaries and forward this data on to the system configured in your outputs.conf. This configuration would allow the data collected to be sent across all of your indexers and no data would be sitting on your search head. If you do this for the Deployment Monitor app, you must remember to create these special summary indexes on your indexer otherwise when the data gets to the indexer, it will not be prepared for it. Here are very granular details.

Best,
Sean

[LCM]

Hi Sean,

Thanks for your hints so far: I investigated the scheduled searches, and they did run. Further I checked the command and ran it again - no return: They all start somewho with forwarder_metrics | eval bla bla. Even if I take forwarder_metrics it does not return anything. That's what I meant, since all summary* buckets are empty, there is no return. However, I'll try to setup another Dep. Mon. on another Splunk instance . . .

I'll come back to you again - Cheers Martin

[msettipane]

There is a difference between the deployment server and the Deployment Monitor application. The deployment server must have a serverclass.conf file to work.

[LCM]

It is the Deployment Monitor App, which comes with Splunk 4.2 which isn't working anymore (not dep. server)