This is a story of a friend of mine who has a strange hobby: He loves nodejs and other javascript related frameworks. A few days ago unfortunately I sent him a link about Claudia.js. Which is actually a cool thing by Gojko Adzic.
Deploy Node.js microservices to AWS easily
He was so excited. He never used Amazon Web Services before so he signed up for an AWS Free Tier. He found a few tutorials, and he just followed the steps in the tutorial.
The Email You NEVER Want to Get
Tree days later on a sunny morning, he asked me if there is anything to do with AWS credentials. He got an e-mail from AWS to remove his AWS credentials. He was pretty calm, had no idea that this is something pretty serious.
Hmm, that sounded pretty bad, so I asked him to forward the e-mail to me. Actually it was THE email you never want to get.
Amazon Web Services has opened case xxxxxx on your behalf.
The details of the case are as follows:
Case ID: xxxxxx
Subject: Your AWS account is compromised
Severity: Low
Correspondence: Dear AWS Customer,Your AWS Account is compromised! Please review the following notice and take >immediate action to secure your account.
Your security is important to us. We have become aware that the AWS Access Key AKIAXXXX (belonging to IAM user “xxx”) along with the corresponding Secret Key is publicly available online at github.
Holy shit. It seems he had accidentally committed his AWS keys to GitHub. At this point I made a phone call and asked him to quickly review his AWS account. He logged in and saw 20 running instances instead of his t2.micro. His account was full of g2.8xlarge instance types. That’s not that cheap.
So I asked him to immediately deactivate all AWS API keys and terminate all instances in his current region and asked him to check all regions too. Unfortunately it wasn’t enough. Replacement servers were fired up after a few minutes. I knew that it won’t be that simple, so I felt I have to do it myself.
I logged in with this root AWS credentials and navigated to the Auto Scaling Groups, but that was empty. My next tip was Spot Request instances. Bingo. There were 20 spot requests / region with $3.2 max price.
“Luckily” his instance and spot limit were around 20, so I cancelled all spot bids and terminated all servers too. Double checked all regions.
The worst part
As the account was cleaned up, it was time to check the billing. His account was exposed for about 2-3 days, which can cause a lot of instance hours. I clicked on the Billing & Cost Management tab and saw the following on the dashboard:
Current balance is $6,437
Oh my. I wasn’t quite sure how to tell him. I was shocked. I asked my colleague about this, and he told me that:
AWS will probably credit him back if he gets it closed up quickly
I hope this is the case and he won’t need to pay that.
Takeaway
Can’t emphasize enough how important is to keep your credentials safe. What can you do to prevent this happening from the future?
Use ephemeral tokens if possible. This is provided by AWS Security Token Service. You can define temporary cedentials with it.
The temporary security credentials are valid for the duration that you specified when calling AssumeRole, which can be from 900 seconds (15 minutes) to a maximum of 3600 seconds (1 hour). The default is 1 hour.
Monitor Estimated Charges Using Billing Alerts on a daily basis. There are tools out there already.
Remove keys after using them. If you’re done with experimenting, do not leave the keys there. Just remove them.
Decrease your default limits. If you know you won’t need ~20 instances only 1-2, open a support ticket and ask them to decrease your instance limits.
Use IAM roles and policies. Restrict the accesses of the keys to as minimum as you can. A few examples are here.
Do not store your credentials on github without encrpyting them.