Authenticating S3 access using non-anonymous request URLs

If you run a small data center and have capped bandwidth you don't want to be delivering bulk data to customers. It is better to place the data in the cloud and redirect your customers to get the data there. Amazon's S3 is a good place for that as creating a public URL is trivial. If the data is not public then S3 has a simple mechanism for enabling you to authenticate access. To do this you run your own authentication service; this service prepares a signed, time limited URL that you give to the client to use to download the data from S3. The network interaction is all done within SSL and so you don't need to worry about the URL escaping into the wild and even if it did the loss is time limited.

The AWS S3 service calls this a non-anonymous request URL. For example, if your data is in the "2019-Q4.tsv" item in the "com.andrewgilmartin.bucket1" bucket the URL is

https://s3.amazonaws.com/com.andrewgilmartin.bucket1/2019-Q4.tsv

Your authentication service will (after authenticating the user) redirect the user's HTTP client to the URL

https://s3.amazonaws.com/com.andrewgilmartin.bucket1/2019-Q4.tsv
    ?AWSAccessKeyId=<<AWS_ACCESS_KEY>>
    &Expires=<<EXPIRES>>
    &Signature=<<SIGNATURE>>

This is the non-anonymous request URL. The <<SIGNATURE>> is a base64 encoding of an SHA1 encryption of the HTTP method ("GET"), the path ("/com.andrewgilmartin.bucket1/2019-Q4.tsv"), and the expiration time (<<EXPIRES>>). The <<AWS_ACCESS_KEY>> corresponding secret key is used for the encryption. An example Java implementation is at S3RestAuthenticationUrlFactory.

For any of this to work you will need an AWS access key id and secret key that is associated with an IAM user with a policy to access the S3 bucket. If you have not done this before the video AWS S3 Bucket Security, Restrict Privileges to User using IAM Policy is a good tutorial. If you only want to allow read access then remove the "s3:PutObject" and "s3:DeleteObject" actions from the example policy.

The splendor and complexity of networked service at scale

The splendor and complexity of networked service at scale. Josh Evans' talk Mastering Chaos - A Netflix Guide to Microservices at InfoQ.

Incident Response Slack application series

I am have finished working on the Incident Response Slack application. It was a hobby project and I feel no need to take it any further, although, I would like to switch to RDS from SimpleDB. Here are the postings related to the project

1. Incident Response Slack App 
2. Adding persistence to the Incident Response Slack application 
3. Who wants to perpetuate a flawed design when a proper one is just around the corner? 
4. Incident Response and stating requirements
5. Access AWS Secrets Manager from AWS Elastic Beanstalk

The code is on GitHub at https://github.com/andrewgilmartin/com.andrewgilmartin.incidentresponse

Access AWS Secrets Manager from AWS Elastic Beanstalk

I finally figured out how to connect my Incident Response Java application running in AWS Elastic Beanstalk with its associated secret in AWS Secrets Manager. As always, the solution is obvious once it is known. What is not so obvious to me as I write this, is how many of previous "solutions" were correct but the "system" had not yet become "eventually consistent?" Moreover, the current solution is very specific and that seems wrong. Nevertheless, here are the solution's parts.

Assume you have created an AWS Secrets Manager called S1 and an AWS Elastic Beanstalk called A1. What you now need to do is to

1. Create an IAM Policy called P1 that enables access to S1.
2. Attach policy P1 to the role aws-elasticbeanstalk-service-role.
3. Attach policy P1 to the role aws-elasticbeanstalk-ec2-role.

When creating policy P1 give it all Secrets Manager's List and Read actions. I am sure some can be skipped and so you and I should go back and remove unneeded actions. You will also need to limit the actions to only the S1 resource, aka its ARN.

Once this is done, restart your application A1 and it will now have access the S1 secret and its keys and their values.

Since Incident Response needs to store its data in AWS SimpleDB, I extended the S1 policy to include all access to any AWS SimpleDB domain, but this did not work. I had to restrict access to a specific domain's ARN. What bothers me about this is that I have now specified the specific domain in both P1 and as a secret S1 key "aws.simpledb.domain" value. Perhaps there is a way I can read the domain from the policy; an exploration for another day.

Good Terraform and AWS tutorial

The Learn DevOps: Infrastructure Automation With Terraform tutorial from Udemy is very good. While it is focuses on using Terraform to manage AWS it also gives a good introduction on how to assemble AWS' many moving parts.

The only trouble I had with the examples was that the initial ones assume that you have IP access via the AWS default security group. That might not be the case and so you will need to use the AWS console to allow your machine's public IP address ingress. To do this, go to the AWS Console, then to the EC2 dashboard, then to the Security Groups resource, and in that dialog select the default security group and add an Inbound rule for your public IP address.

Amazon's SimpleDB is free for the vast majority of datasets

You can keep up to 1 gigabyte of data in SimpleDB without paying any storage fees. You can transfer 1 GB of data and use up to 25 Machine Hours to process your queries each month. This should be sufficient to allow you to issue about 2 million PutAttribute or Select calls per month.

More at Don't Forget: You Can Use Amazon SimpleDB For Free!