Infrastructure from scratch. Part 10. Testing


This is the last part of my Junior Ops cycle. Who reads it ever?

The proper testing can reduce troubleshooting stuff and speed up development-commit-deploy lifecycle. There are many kinds of software testing, each of them has a specific area. Here I’d like to concentrate on 2 areas: performance testing and load testing. I’m dealing with them choosing Yandex.Tank.

Why only 2 areas?

Generally I focus on on 4 items: unit testing, E2E, performance and load testing. Unit verifies general application integrity and health, checking all code migrations are working. End-to-end (E2E) works like a blackbox testing emulating customer behaviours. Performance testing allows to track the service speed. Slow service is even worse than the downtime. Load testing shows up the rate of accepted payload on production-like environment. So, we can ensure once more we’re still can handle as much rps as we need.

Ok, won’t keep any longer. If you’re curious – read about these kinds at intended web sources.

Why 2? I think unit tests as E2E mostly supported by software testing engineers. In operations the head concept – load acceptance. We verify the product quality under battle cases where no chance to step back. And here is the solution – Yandex.Tank.

Why Yandex.Tank? What it is?

Yandex.Tank is a performance testing application produced by Yandex software performance department. It’s a Python application which works as CLI utility configured on INI syntax. It has the same purposes as a popular Apache Jmeter. But workflow is faster (because on CLI) and much easier to maintain.

It still supports Jmeter interface just making it easier to maintain. Actually, who can read hundreds lines of XML code on the fly? If we look wider, we may find better solutions. But they’re not free, not open-source, so here is a complicated conundrum.

How does your Yandex.Tank work?

  1. Configure the main ini.file (load.ini).
rps_schedule=const(3, 60s)

Configuration tips:

  • use ”disk_limit” option to reduce the probability of testing failure. Yandex.Tank won’t be run if system resources amount is less than should be. You can configure how should it be.
  • ”rps_schedule” is the crucial parameter. Here you measure how much capacity do you want to push on the testing environment and how does it changed. There are 3 rps (requests per second) strategies:
  1. const (constant load all testing time pointed in second parameter);
  2. line (a,b,dur) makes linear load, where a,b are start/end load, dur – the time for linear load increase from a to b;
  3. step (a,b,step,dur) makes stepped load, where a,b are start/end load values, step – increment value, dur – step duration.
  • ”autostop” setting automatically exits the testing process if result is worse than set value. There is huge selection of autostop strategies, we have the 95th percentile.

The Yandex team prepared a good docs to dive into this utility.

  1. Configure the ammo – subject you will request through Yandex.Tank. Just HTTP request with parameters. Here is ”example” file:
[Accept:application/json, text/plain, */*]
[Accept-Encoding:gzip, deflate, sdch]
[Authorization:Basic dGlja3NhOnQxY2tzQA==]

Here I pasted all necessary HTTP headers and request name at the bottom. Just copy all headers from Chrome Developer Console to the ammo file. It’s the easiest way to customize ammo file.

  1. Run the test. One command.
yandex-tank example

You can see the pretty CLI output during the testing.


Performance testing

With Yandex.Tank I work on 2 major tasks:

  1. General performance testing. We have 5 main HTTP requests during the client session. We know how much time do they usually take for processing. We base the time value on our tests. Approaches for testing – 95th percentile. It means that 95% of all requests should be no less than a set value.

In Continuous Delivery pipeline we run Yandex.Tank under Bash wrapper. Script dynamically changes the ini config and inserts the request-specific time limit. Later one test per each request is performed. Finally script prints the 95th percentile calculated from logs. If percentile takes longer than limit- the process will be exited with error code.

  1. New Node.js versions testing. I don’t only check the Node.js performance. But look how our application works under the specific version. Will requests get the 200 code, how fast do they work compared with the current version and so on.

For the time of post writing (October 2016) we’ve got a choice between Node.js 4.x mainline. I got the necessary benchmarks and started analysis, also including legacy 0.12 branch.

Won’t apply benchmarks (because I sadly missed them), just saying impressions. Under low load our app worked faster on 0.12 version. But the more capacity we put, the faster 4.x become. So, finally we chose 4.4 for production despite of 4.6 at the mainline. Such performance testing can clearly answer on the question “What’s better for you?”. I can’t take responsibility to say that Node 4.6 is slower than its successors. I can only talk about my situation. And this slight variation matters in almost all cases when we need to choose the software.

Load testing

Also you can simply make a high load from one medium-sized node with Yandex.Tank. Tank makes requests working on Phantom load generator by default which is definitely powerful. For example, I tested staging environment on Amazon EC2 to understand Autoscaling group opportunity. How fast will it trigger the new instance? How long does it take to refresh Cloudwatch metrics? It’s not only work, it’s just for fun to research these behaviors.

Fun throw out the toys. In reality this kind of testing can usually fit 2 issues:

  1. Pre-release verification. Thus we check: service still can handle necessary rps amount and won’t lay down on production.
  2. New environment testing. For example, you wish to cut the resources switching to the low-class machine. You can test the real use case on new environment to check you won’t lose anything.
  3. Capacity planning. It’s similar with the second use case. Take a load increasion trend, surplus current load and apply the tests. The benefit of such testing is better understanding of scalability needs.

What about security?

I use 2 utilities in Continuous Delivery pipeline shortly described at the previous article. ZAProxy for HTTP data exchange and nsp for Node modules security vulnerabilities.

Don’t lose the security, please.

How about Chaos Monkey?

Netflix run Chaos Monkey test to be prepared for deep troubleshooting in unbelievable use-cases. I think that’s great if you improved your infrastructure so hard to make this kind of testing. In my case I use simple services switch-off checking how fast the Consul agent gets it up.

Generally Monkey testing may be called as E2E for operations. Do it or not? Well, it depends on infrastructure scale and requirements.


  1. Regularly test infrastructure recovery from scratch. I didn’t describe the testing recovery process. It’s even more boring than my previous posts. But anyway if you want to be prepared – make this test. Also recovery testing should force you to make the infrastructure code more reasonable.
  2. Don’t benchmark the popular software as only software. If you need to see how does your service work with different tool or version – it’s OK. But don’t compare the software itself. It’s a complete waste of time if you’re not researcher. If you’re so curious – search this test on the Web. Probably you’ll find what you want with complete comparison and description.

Alright, now I have a huge desire to finish it all. Steps I did and described may seem pretty enough for small startup IT project. Later on, after infra construction we need to focus on things like maintenance and ongoing improvement.

So, let the new challenges be!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s