Infrastructure from scratch. Part 7. Security

The word we say almost everything and everywhere. The question we have to double check when implementing new feature. It’s a kind of work allows to sleep calmly. Yes, this is security.

It’s such easy to throw out safety questions concentrating on other project work. But this is the misleading feeling. Like backups, security makes our protection thicker and much more reliable.

Oh, too much water I started with. Let’s dig into security on small startup project.

Check what you have

---

To start solving the issue, you have to correctly understand what the issue is. To get started, you have some ways of infrastructure analysis:

1. Personal check. Based on your experience. Look at the things going on with your network, OS management and TCP connections generally. You can see obvious things like “everybody using root” or “port is open for 0.0.0.0/0”;
2. Third-party audit. This is a popular practice to invite expert for independent and honest analysis. When I came into the project, I’ve already got this kind of report. It was difficult to skip any vulnerability in my infrastructure. Human reports are usually thorough and describable;
3. Audit software. There are wide choice of tools. What I was need – OS investigation, network scan and AWS resources testing.

Look at the hacking trends

---

In IT world days goes by much faster than anywhere else. Such thing happens with security issues. Bug fix version of software only released and new vulnerability already exposed and leveraged. So, every new quarter attacks rate could be significantly changed. Subscribe on the security reports or search this kind of issues once in awhile. Such reports, if verified, allows to concentrate on specific attacks prevention mostly happened. Here is example report.

What’s done

---

Working on security, I did some small and big tasks. The scale is not always proportional to importance and general effects. There can be simple solution which will significantly decrease the exposure probability.

Warning! TL/DR, guys!

1. HIDS implementation. HIDS helps to keep a shrewd sight on things that’s going on your servers. You may simply check what file was changed, look at the strange logs and so on. Generally intrusion detections is a rootkit prevention and events register. If you look at the HIDS deeper, you can build bullet-proof system automatically responding on malicious events.
   I use OSSEC for intrusion detection. OSSEC is simple and powerful, has clear architecture and low overhead. With e-mail notifications, I can easily be acknowledged every little event on my servers. That’s great, absolutely! OSSEC also split events on 15 levels – from simple logs to direct hacking.
2. Fail2ban for TCP connections control. There always will be strange HTTP connections to your public web-server with /phpmyadmin referers. Also SSH hacking with thousands of login attempts. Better to block these request then flood your network.
   Fail2ban got it! This utility is just a customized set of filters converting to iptables firewall rules. Fail2Ban parse logs where client IP is described and detects malicious events by pattern. After few occurrences, IP address will be blocked for further connections. Number of occurrences and ban time is configured.
   It’s a good practice to hang Fail2ban filters on SSH port or web-app. But be careful to implement this on production! Setting up wrong filter might block any access to your application. Double test your deployment and verify all filters is reasonable. Personally I use some Nginx filters to prevent PHP script URIs, bots flood etc..
   It also would be handy to configure e-mail notifications about events generated by Fail2Ban.
3. Move SSH from 22 port. Easy, but mighty solution I talk about. Even custom public services face with SSH flood. Even if you have key-only access, you’ll get plenty of possible break-in’s. You’re able to count on SSH protocol reliability, although it has many vulnerabilities exposed.
   The best practice will be just moving from 22 port. Even it’s 222 or 2222 port, it’s going to save you from bots and naive hackers. For easy access from SSH client, set up the config on your workstation. Place the n port as default to the host connection.
4. Backup. That’s for the way if your service component has been willy-nilly compromised. Take regular backups automatically and test their restore. You’ll save a lot of time doing backups and testing restores. The compromised component thus may be easily destroyed.
   I described few days ago how I work with backups.
5. Pre-signed URLs for media content. It’s not a great idea if you’re public media content provider. Would be better to get your content from everywhere. But! If your content is must-paid you’ll have to take care of access.
   Pre-signed URL technique is going to solve it. In our case we must secure S3 and Cloudfront content. Amazon has built-in technique. Implementation depends on programming language used in project. For our Node.js S3 has a built-in with Amazon SDK, Cloudfront is not.
6. Nginx HTTP authentication. You may have web-services must-have to be secured and restricted from the third part. It could be staging version of your product or worldwide shared monitoring system.
   Nginx as a proxy allows easily to make that. Install apache2-utils, create .htpasswd file, set up restricted authentication and you’ve done. Here is the guide.
7. Firewall. To deal with firewall, you should clearly understand how TCP/UDP works. Basically firewall may be used to drop invalid connections. It can be too small TCP MSS, SYN-flood attempt, timeout expiration etc..
   Working with iptables, I realised the best practice lies in prerouting level. The closer connection check to the server, the flooder your bandwidth becomes. Because of this, firewall rules on prerouting label seems much more reliable.
   Iptables is the most popular solution. You can find a lot of good practices to master this utility and learn to write the rules yourself. But, additionally, firewall mastering is not only Iptables rules. It’s a good knowledge about hacking trends and network connections understanding. Learn the theory, if you don’t want to get stuck on practice.
8. Privilege separation: AWS, database, OS. Internal security is important as much as the public one. You can shift an eye on your team members 24/7. You can’t expect anything special from them, but be prepared. They may have a bad mood, have a big anger after they quit or simply be compromised. Privilege separation is the best practice. Everywhere.
   In AWS account developers are only able to launch instances + read-only for some services. IAM provides a lot of opportunities to configure account policy for your needs. Only team leaders could have an access to infrastructure servers. And mostly read-only like read logs. Service restart is allowed for emergency as well.
   In application databases development/management could make only SELECT from their accounts. Always database is driven by special role (in PostgreSQL it’s called role, yeah). Each database should have its own user with isolated privileges to only one DB. Thus it’s going to be impossible to compromise all DBs when only one was leaked.
9. AWS security groups. Must-use feature for AWS administrators and developers. Permit only connections to ports you need. And not 0.0.0.0/0. Of course, public web-products is an exception. But otherwise make sure only necessary sources have port access. It would be better to divide your machines on few subnets and base your security groups on subnet mask.
10. AWS Cloudtrail + Traildash. Cloudtrail is a declarative service for real-time AWS account audit. Dig up what user did the special API call, tried to penetrate into the sensitive components and so on and so forth.
    Sometimes it’s hard to analyze any trend with default CloudTrail declaration. Here will be third-party tool the great solution. Meet the Traildash! Simple Go binary which deploys Kibana dashboard. So, visualization is nice and clear where you can easily track each user in your AWS account.
11. AWS WAF. There is an issue with hotlinking content. Where someone steals your resources URL for his own website. AWS WAF helps to prevent this problem working with Cloudfront. It provides ACL and rules filtering incoming HTTP header.
    We use WAF to prevent content share from third-party referrers i.e. hotlinking.
12. Unprivileged user for service launch. You’re afraid of shell-shock or simple shell commands in application text field. That’s the old story here. If you launch your app from root, you can get answer; rm -rf / on the text field and say goodbye to your server. But it’s not so hard to prevent.
    Just make a new user with little privileges your app needs and launch the app from it. Thus your app won’t be able to execute privileged malicious commands. Also it matters databases. Execute some queries from unprivileged user to prevent SQL injection consequences.
13. Source code credentials encryption. Even you have private Gitlab repository, you’re still face with credentials challenge.  Only operations lead and developers lead should know the DB passwords and API tokens. The issue is more keen, when you host your project on GitHub where everyone can see your credentials.
    I can’t say there is wide choice of Git repo encryption. But they exist. At first we tried a git-crypt, but this one is so complicated to manage. Complete solution was blackbox utility from StackExchange team. It’s such flexible, clear and powerful encryption engine. Anyway learn the GPG basics and how this kind of encryption is working. And, yes, it matters your Infrastructure-as-a-Code too.
14. Final testing. After all possible solutions it’s time to test it again. Some issues still were, some more came up with continuous infrastructure improvement. All we need here – clearly understand requirements, set goals and do some hardens again. It’s not a big problem when you personally observed security holes. The big problem is when you don’t care about them.

Advices

---

1. Where additional validation comes in infrastructure, the performance degradation may happen. Test every change you make in your infrastructure. Ensure your application speed hasn’t decreased. Or decreased, but not significantly and you can allow it for better attacks prevention.
2. Read anywhere the best security practices in work. Like “don’t store your passwords in text file”, “backup your SSH key” and so on. Won’t describe all these practices, cause I still don’t know all of them. But that’s important. The problem lies much deeper you’d think.
3. Teach security everyone. If you’re SRE – ask your developers to perform source code pentesting and encrypt credentials. Also consult your managers and stakeholders how to work with sensitive data.

• audit
• AWS
• Cloudtrail
• fail2ban
• firewall
• infrastructure
• iptables
• ossec