I was reminded of a lesson I learnt the hard way a number of years ago. The importance of using build artifacts in
When I first learnt this lesson, I ended up with multiple production servers failing and a supposedly highly-available,
redundant website offline. One of the major package/dependency repositories had gone offline; after scaling down
overnight the cluster started scaling up for the day; the new servers were passing healthchecks; but the servers were
returning errors. As the problem persisted the last working servers were cycled out of production and terminated. At
the time there was nothing I could do until the dependency repository came back online. Thankfully I no longer
remember how long the site was offline for, but at the time it was extremely stressful.
The solution to this issue turns out to be quite easy, and in compiled languages the issue is far less likely to occur
as developers are more used to using build artifacts. The simple solution is to create a deployment package with no
automated) only needs minor modifications to create build artifacts.
As development teams adopt best practices and stop editing code on production servers, a testing process inevitably gets
implemented. If you’re not yet at this stage of development maturity then I suggest you stop reading this article and
start asking your fellow developers why your current practices need to change; seriously, do it now! For the rest of
you, prior to testing I’m sure you install and/or update any required dependencies. Once testing has been completed,
you likely upload your code to a central location. Then you either deploy it to the servers, or trigger your cluster
to perform a deployment. If you’re relying on your package manager during a deployment it is at this point that
everything can go wrong. When the package repository is offline or experiencing intermittent errors your servers may
fail to deploy.
To avoid this, when the deployment package is created, include all the dependencies; then, when the package is deployed,
don’t run the package manager. Yes, it’s that easy. Keep your deployments completely under your own control, as long
as the package manager is working during the build process, then the upgrades can get deployed; but if it’s down, the
build will fail and won’t be promoted to production, saving you from the headache of your site starting to fail. When
the package manager comes back online, trigger a rebuild and the deployments can resume.