Search for a command to run...
Hi! I'm Guille Ojeda, Cloud Architecture Consultant To put it simply, I create simple solutions for complex technical problems.
My focus is on designing scalable, secure, and cost-effective cloud solutions that startups, scaleups and SMBs can lean on to grow.
I lean on years of experience in cloud architecture and consulting to deliver solutions that are easy to understand and manage.
No comments yet. Be the first to comment.
What Are Agentic AI Architectures? I won't waste your time with a long, fluffy introduction about how AI is changing the world. Let's get straight to the point: agentic AI architectures are fundamentally different from the prompt-response pattern you...
Building distributed systems requires putting a lot of attention on communication between components. These components often need to exchange information asynchronously, and that's where message queues and pub/sub systems are the go-to solution. AWS ...
Database failures are inevitable. Even with the most reliable hardware and software, something will eventually break. AWS RDS Multi-AZ deployments promise to handle these failures gracefully, automatically failing over to a standby database when prob...
There are two managed relational database services in AWS: Amazon Relational Database Service (RDS) and Amazon Aurora. Both provide the benefits of a fully managed database solution, but they have distinct features and use cases. In this article, we'...
Containers offer a lightweight, portable, and scalable solution for running software consistently across different environments. But as the number of containers grows, managing them becomes increasingly complex. That's where container orchestration c...
You've deployed your app on an EC2 instance, and there's a file in an S3 bucket that you need to access from the app. You created a public S3 bucket and uploaded the file, and it works! But then you read somewhere that keeping your private files in a public S3 bucket is a bad idea, so you set out to fix it.
Here's the initial setup, and you can deploy it here:
This is what it looks like before the solution:
This is what it looks like with the solution:
Open the CloudFormation console
Select the initial state stack
Click the Outputs tab
Copy the value for EC2InstancePublicIp
Paste it in the browser, append :3000 and hit Enter/Return
Go to the VPC console
In the panel on the left, click Endpoints
Click Create Endpoint
Enter a name
In the Services section, enter S3 in the search box, and select the one that says 'com.amazonaws.your_region.s3' (replace 'your_region' with the region where you deployed the initial setup, which is where the S3 bucket is). Then select the one that says Interface in the Type column.
For VPC, select SimpleAWSVPC from the dropdown list
Under Subnets, select us-east-1a and us-east-1b, and for each click the dropdown and select the only available subnet
Under Security groups, select the one called VPCEndpointSecurityGroup
Under Policy, pick Full Access for now (we'll change that in Step 2).
Open Additional settings
Check Enable DNS name
Uncheck Enable private DNS only for inbound endpoint
Click Create endpoint
In the Amazon VPC console, go to Endpoints
Select the Endpoint you just created
Click the Policy tab
Click Edit Policy
Modify the following JSON by replacing the placeholder values REPLACE_BUCKET_NAME and REPLACE_VPC_ID with the name of your S3 bucket and the ID of SimpleAWSVPC. Then paste it into the Edit Policy page, and click Save.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAccessToSpecificBucket",
"Principal": "*",
"Action": "s3:*",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::REPLACE_BUCKET_NAME",
"arn:aws:s3:::REPLACE_BUCKET_NAME/*"
],
"Condition": {
"StringEquals": {
"aws:sourceVpc": "REPLACE_VPC_ID"
}
}
}
]
}
Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.
Open the S3 console
Click on the bucket that you created with the initial setup
Click on the Permissions tab
Scroll down to Bucket Policy and click Edit
Paste the following policy, replacing the placeholders REPLACE_BUCKET_NAME and REPLACE_VPC_ENDPOINT_ID with their values (REPLACE_VPC_ENDPOINT_ID is not the same as REPLACE_VPC_ID from the previous step). Then click Save changes
{
"Version": "2012-10-17",
"Id": "Policy1415115909153",
"Statement": [
{
"Sid": "Access-only-from-SimpleAWSVPC",
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:PutObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::REPLACE_BUCKET_NAME",
"arn:aws:s3:::REPLACE_BUCKET_NAME/*"
],
"Condition": {
"StringNotEquals": {
"aws:SourceVpce": "REPLACE_VPC_ENDPOINT_ID"
}
}
},
{
"Sid": "Access-from-everywhere",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::REPLACE_BUCKET_NAME",
"arn:aws:s3:::REPLACE_BUCKET_NAME/*"
]
}
]
}
Before deleting the CloudFormation stack, you'll need to empty the S3 bucket! The Node.js app puts a file in there.
First of all, you'll notice that a VPC Endpoint is for one specific service, S3 in this case. If you wanted to connect to other services you'd need to create a separate VPC Endpoint for each different service.
The second thing you'll notice is that there are 2 types of endpoints: Interface and Gateway. Gateway endpoints are only for S3 and DynamoDB, while Interface endpoints are for nearly everything. Gateway endpoints are simpler, so use them when you can (except if you're writing a newsletter and want to show a few things about Interface endpoints).
Interface endpoints work by creating an Elastic Network Interface in every subnet where you deploy it, and automatically routing to that ENI the traffic that's addressed to the public endpoint of the service. That way, you don't need to make any changes to the code. This only works if you check Enable DNS name.
The existing policy is a Full Access policy, which is the default policy when a VPC endpoint is created. It allows all actions on the S3 service from anyone.
Instead of that, we're setting up a more restrictive policy, which only allows access to our specific bucket, and denies access to all other buckets.
VPC Endpoint policies are IAM resource policies, and as such, anything that's not explicitly allowed is implicitly denied.
Bucket policies are another type of IAM resource policies. Obviously, this bucket policy will only apply to our S3 bucket. It's important to add it because, while we've restricted what the VPC Endpoint can be used for, the S3 bucket can still be accessed from outside the VPC (e.g. from the public internet). This bucket policy is the one that's going to prevent that, restricting access to only from the VPC Endpoint.
In this case I kept internet access for the VPC and for the EC2 instance itself, just to make it easier to trigger the code with an HTTP request. This solution is a good idea in these cases because traffic to S3 doesn't go over the public internet, but admittedly, the public internet is a viable alternative.
Where this solution matters more is when you don't have access to the internet. Sure, adding it is rather simple, but you're either exposing yourself unnecessarily by giving your instances a public IP address they don't need, or you're paying for a NAT Gateway. In those cases, VPC Endpoints are a much simpler, safer and cheaper solution.
Conceptually, you can think of this as giving the S3 service a private IP address inside your VPC. In reality, what you're doing is creating a private IP address in your VPC that leads to the S3 service, so that conception is pretty accurate! Behind the scenes (and you can see this easily), the VPC service creates an Elastic Network Interface (ENI) in every subnet where you deploy the VPC Endpoint. Those ENIs will forward the traffic to the S3 service endpoints that are private to the AWS network.
Also, behind the scenes there's a Route 53 Private Hosted Zone that you can't see, but which resolves the S3 address to the private IPs of those ENIs, instead of to the public IPs of the public endpoints. That's why you don't need to change the code: Your code depends on the address of the S3 service, and that private hosted zone takes care of resolving it to a different address. You can't see this private hosted zone, it's managed by AWS and hidden from users.
Least Privilege Access to Bucket: This is basically what we did in Step 3: We disabled public access, and implemented a policy that only allows reads from the VPC. Try reading from that S3 bucket from your own computer: aws s3api get-object --bucket 12ewqaewr2qqq --key thankyou.txt thankyou.txt --region us-east-1
Regularly Audit IAM Policies: Regularly review and tighten your IAM policies. Not only for the VPC Endpoint and S3 bucket, but also for the EC2 instance!
Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.
Real scenarios and solutions
The why behind the solutions
Best practices to improve them
If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com