Infrastructure from scratch. Part 9. Continuous Delivery.


Frankly speaking, I wished to put together CI and CD thoughts. But the post became too large. So, I’ve got the idea to separate them and take the easier way for reading. It’s still TL/DR, but not so exhausted.

Continuous Delivery is not a technology

The same thing matters Continuous Integration. They are methodologies, practices, principles. Ok, techniques. But not technologies!

Continuous Delivery practice gets a build artifact and checks the system product quality. My thought that CD is a bunch of blackbox testing to check the functionality, performance, high load acceptability. Our app didn’t become slower, the general functions still work as predicted, we can still handle plenty of concurrent requests and so on.

Continuous Deployment is more complicated, but possible. Continuous Deployment doesn’t include the human factor. It’s going to release on production environment. Surely this process has some triggers like release statistics, current load and so on. But anyway be automated. Not every team comes to this solution.

Check what you have

To get started with Delivery, you have to apply the CI methodology in your process at first. To get started with Deployment, have a Delivery. 3 methodologies looks like a my birthplace symbol. I mean each practice includes another within themselves.

What’s done

Eventually I’ve got the CD pipeline run on Jenkins with specific parameters. The pipeline was written the similar way as the CI i described before.


  1. Terraform appliance

At first we have to launch the staging environment. In our case staging is on-demand environment. It means we have to start from scratch. To launch on-demand instances we use Terraform. This pipeline step runs ”terraform apply” command. It launches the copy of production environment for real case testing. Environment includes asg, launch configuration, load balancer and DNS record attached to balancer. That way we’ll get one EC2 instance to run the service.

If we already have a Staging, Terraform just will make sure it works. Terraform is idempotent tool, i.e. oriented on current state than in action. It just will skip this step.

  1. Payment gateway switch

Here is the Ansible playbook which runs Nginx role on payment gateway service. It creates the staging vhost symlink to ”sites-enabled” and reloads Nginx to apply changes. Won’t dive into details, it’s just our needs as every project have.

  1. CodeDeploy deployment

If we set deploy flag to yes in Jenkins build parameters – the Python script will run. It starts new CodeDeploy revision of last package passed through QA before. It should be succeeded, so that way we run the software on staging environment.

  1. Ansible provisioning

Also we’ll need some maintenance in staging server. Here is the Ruby tool which changes SSH config. The goal – get current machine IP got from AWS API. Finally Ansible connects to set IP and provision the instance. Provisioning includes Filebeat configuration to ship the app logs to Logstash. The feature lies in changed hostname and EC2 instance ID injected into Filebeat fields.

  1. System testing

Here is kind of E2E to run basic queries trying all application functions. The purpose – verify it works as before. It could be discovery page scroll, payment transaction, video upload and so on. The tool is protractor.js – Selenium web-driver analog.

  1. Performance testing

Here is the 3 concurrent rps to 5 the most usable HTTP requests. Each iteration is dynamically time-measured. If 95th percentile takes longer than limit some time, the test will be failed. Otherwise it will print the average HTTP request time. So, that way we’ll check the performance changes: did it become faster or slower.

The testing tool is the Yandex.Tank utility. It has pretty easy configuration and failover workflow. The Yandex.Tank job is wrapped into Bash script. The wrapper adds iteration and average time calculation.

  1. Load testing

Here we check application high load acceptability. We run load test with Yandex.Tank with predicted rps count. During the test we look at the hardware utilization from remote monitoring. The metrics for search are CPU system time and load average, RAM usage and network bandwidth use. Finaly the shell wrapper makes the metrics analysis. If any metric is more than limit – the test will be failed.

  1. Security testing

Here is just the basic check of new possible vulnerabilities. Here is the 2 tools to check the security integrity:

  • ZAProxy checks the HTTP response of web service as a user. It may be called a blackbox testing. It generates a simple HTML report. Report is saved as an artifact with possibility to download and investigate. Report may include security vulnerabilities. There may be bad HTTP headers, non-secured connection and so on and so forth.
  • nsp utility scans Node.js modules list and checks are they reliable or not. It prints all security vulnerabilities from obsolete modules versions if any exists. Returns an error code if any exists.

What about Continuous Deployment?

I’ve got the deployment practice, but not continuous. It’s just a one step – push the revision with AWS CodeDeploy. Amazon provides CodeDeploy as a tool for painless deploys with no extra charges. With CodeDeploy we’ve got file checksums verification technique. Finally it gets the changelist and makes the changed files replacement. Of course, CodeDeploy can run scripts/commands before/after deployment. The concept is easy and running fast on existing environment.

Should we have fully automated deploy? Good question. It’s not the technical problem, but the cultural one. Also that’s would be great to learn how to deploy on different hardware platforms. Mostly I mean self-hosted.


  1. Roll-forward is better than rollback. You can’t get a full rollback. You database may be changed, data will be always changed, some uses found the bug. Instead prepare the new release when problem has been learned and fixed.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s