Below is a short recap for anyone who missed it:
On June 17th 2014 Code Spaces, a New Jersey-based maker of development collaboration tools, received a Distributed Denial of Service (DDoS) attack against their site. The attacker left messages for the company to contact them at a Hotmail address, presumably for extortion purposes. In responding to the event, Code Spaces realized the attacker had gained access to their AWS Console, from which they had the ability to terminate the company’s infrastructure. The team at Code Spaces attempted to regain access to their console, not realizing the attacker had set up backdoor accounts. Upon recognizing the attempt to regain access, the attacker proceeded to terminate most of the Code Spaces infrastructure. A few days later the company announced it would be closing its doors due to the inability to recover financially from the event.
This was not my first experience with the potential risks of cloud infrastructure. Several years ago, I had an experience with what I call Cloud Fratricide in which a DevOps engineer fat fingered his way to the termination of a search cluster containing 1B+ documents (leaving me pondering Task-Based Access Control). But something about the termination of a business by a malicious external hacker really shook me.
Below are some basic changes I’d recommend to my Boston colleagues using AWS:
1 - Implement MFA
One of the features we were negligent with at my company is multi-factor authentication (MFA). While securing administrative access to the AWS Console via MFA is a good start, we decided to implement it for all interactive console access. The support for virtual MFA devices makes this a 10 minute investment you will never regret (except at 2 AM when you have to run to get your phone to access to the console - ;) ).
2 - Limit Administrative Access
Every Amazon account has administrative credentials. This account is essential for performing a few actions (e.g. configuring consolidating billing, enabling MFA access) - but also has access to all other privileges across your infrastructure. While this account is essential, it’s access should be highly limited and rarely used.
3 - Rotate Existing Keys
In chatting with a former colleague, he found some of his company’s AWS access keys searchable via Google as a result of a public GitHub Gist by a developer. You can see how easily this could happen (e.g. a developer shares a Gist with a colleague that contains debug-level output inadvertently containing an access/secret key). Take a moment and rotate all your keys, and institute a regular rotation policy consistent with the needs of your business.
4 - Limit Power User
The Amazon policy wizard makes it easy to create a Power User group. While the default power user does not have access to the administrative console features (e.g. account settings, setup consolidated billing), they do have access to pretty much everything else. In short: a power user can shut down all your infrastructure with a few lines of code and a few minutes of time. We decided to separate the power user group into two: a group that has the full access of a power user, and a group that has the full access minus the ability to delete/terminate infrastructure. We have highly restricted the former group.
5 - Ensure Least-Privilege For Services
Cloud-based infrastructure almost always requires API access to other infrastructure (e.g. to push objects to S3, to delete old snapshots, to create/attach EBS volumes). It is essential to ensure this access has the minimum set of privileges required to perform its task. For example, if you have a service that pushes backups to S3, you should ensure this service has only write access to the specific bucket on S3 in which the backups will reside. While this is labor intensive and will require careful testing to rollout, it is essential to minimizing risk of a service-account used in a cloud fratricide / hijacking event. One short-term way to minimize risk is decribed in Denying Destructive Privileges in AWS.
Conclusions
Amazon does not get much credit for its incredibly powerful features around security. Features such as security groups, VPC, role/resource-based access control, multi-factor authentication, encrypted data storage, and more, make it possible for customers to implement highly secured environments. But all this expressiveness is useless if we don’t take advantage of it.
So if you have not taken the time to push security stories to your agile board, the above recommendations might be a good place to start. Let's all ensure Code Spaces didn't die in vain.