Deployment Architecture

Why is my deployment application disappearing?

jamesoconnell
Path Finder

As part of the work to upgrade all of our forwarders in our enterprise, we are also pointing forwarders to a NEW deployment server.

This is done by running a bash script on each forwarder that downloads a new deployment application that contains a different deploymentclient.conf -- which points to the new deployment server URL:PORT.

The version of the forwarder software is also upgraded and the forwarder restarted.

All is well.

However, on some forwarders, the new deployment application that replaced a deployment application of the same name is removed (I can only assume by the synchronization of the deployment server) -- though not necessarily right away; sometimes there is a delay of unknown time.

We have experienced issues before with synchronization in the deployment server and the forwarders. Does this problem sound familiar?

Regards.

0 Karma
1 Solution

sloshburch
Splunk Employee
Splunk Employee

I'm a little unclear on some of the details here but some thoughts:

  • if the new DS doesn't have the new app and a serverclass.conf mapping for it, it's very reasonable it will be removed from the deploymentclient
  • In my environments, I usually keep the deploymentclient.conf definition in the system/local. Then if I want to update it across the board, I use a scripted input shell script to make the change. I do this because I don't want it in etc/apps where another app by the same name or a typo in serverclass could blow away the client and therefore sever my connection (to many clients all at the same time).
  • Use DNS. I always set my DS to a friendly DNS name. That way if I do have to move my DS I can rebuild, test with a handful of hosts and then do a real cutover with a DNS change and never have to touch the forwarders. In general, I'm a fan of DNS entries for any single point of failure so as to provide me more flexibility if I have to replace/rebuild that single point of failure.
  • Remember that each app gets a hash generated to act as a fingerprint. That way if the app is edited and the DS is reloaded (or restarted) the hash is updated. If different than what's on the client then the client will pull the version on the DS. That means that even if you have the same app on both DS I suppose the hash could be different. Furthermore, check the splunkd.log which might have some messages about what went down with the 'jim_rocks' app in the corresponding component's events.

With my luck, none of that will help BUT give it a read and respond back as I'm sure you'll catch any incorrect assumptions or misunderstands I had.

View solution in original post

0 Karma

Lucas_K
Motivator

If you utilise a new deployment server the app checksum's that is provided to the client will differ. As such the client will remove the old app(s) and install the new one(s).

This is why you need to use "sticky sessions" if you load balance deployment servers. Each deployment servers checksums are unique even if they contan that EXACT same (ie. md5sum shows exactly the same) files.

If you replace one deployment server for another this same behaviour will occur also.

The delay in replacing the app will be the phonehome time. This is also based on the initial start time of the instance.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

This sentence really trips me up "the new deployment application that replaced a deployment application of the same name is removed". I'll make up a name to try to digest: the deployment app containing the deploymentclient.conf will be 'jim_rocks', the old DS will be called 'old_ds' and the new one 'new_ds'.

So, are you saying: the forwarders already had a 'jim_rocks' app with a deploymentclient.conf file pointing to 'old_ds:port'. You then have a script run that updates that file to 'new_ds:port'. Then that client connects to 'new_ds' and 'jim_rocks' is totally blown away. Is that the scenario?

In this sequence, when did you upgrade? Before the script run or after?

Does 'jim_rocks' live on the 'new_ds'? If so, did it exist on the 'old_ds'? How were the two apps on the two different DS different?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I'm a little unclear on some of the details here but some thoughts:

  • if the new DS doesn't have the new app and a serverclass.conf mapping for it, it's very reasonable it will be removed from the deploymentclient
  • In my environments, I usually keep the deploymentclient.conf definition in the system/local. Then if I want to update it across the board, I use a scripted input shell script to make the change. I do this because I don't want it in etc/apps where another app by the same name or a typo in serverclass could blow away the client and therefore sever my connection (to many clients all at the same time).
  • Use DNS. I always set my DS to a friendly DNS name. That way if I do have to move my DS I can rebuild, test with a handful of hosts and then do a real cutover with a DNS change and never have to touch the forwarders. In general, I'm a fan of DNS entries for any single point of failure so as to provide me more flexibility if I have to replace/rebuild that single point of failure.
  • Remember that each app gets a hash generated to act as a fingerprint. That way if the app is edited and the DS is reloaded (or restarted) the hash is updated. If different than what's on the client then the client will pull the version on the DS. That means that even if you have the same app on both DS I suppose the hash could be different. Furthermore, check the splunkd.log which might have some messages about what went down with the 'jim_rocks' app in the corresponding component's events.

With my luck, none of that will help BUT give it a read and respond back as I'm sure you'll catch any incorrect assumptions or misunderstands I had.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...