Let's dive deep into how RDS Multi-AZ really works, what happens during failover, and how to design your applications to handle it properly. Understanding these internals will help you build more reliable applications and troubleshoot issues when they occur.

Understanding Amazon RDS Architecture

Before we can understand Multi-AZ, we need to understand how RDS works under the hood. RDS is a complex distributed system that manages databases. When you create an RDS instance, you're actually getting several pieces working together.

At the core is an EC2 instance running your chosen database engine. This instance has EBS volumes attached to it for storage, and it's connected to your VPC through Elastic Network Interfaces. There's also a control plane running in AWS's infrastructure that manages everything from automated backups to failover decisions.

This separation between the control plane and data plane is crucial. The control plane runs in AWS's infrastructure, independently of your database instances. This means it can continue making decisions and taking actions even when your database instances are having problems. That's particularly important during failover scenarios.

The storage layer is equally important. Your data lives on EBS volumes, which operate independently from the EC2 instance running your database. This separation of compute and storage enables some of RDS's coolest features, including the storage-level replication that makes Multi-AZ work.

Availability Zones in AWS

AWS documentation often describes Availability Zones as "physically separated locations with independent power, networking, and cooling." That's true, but the key point is that they're engineered for complete failure isolation from other AZs.

AWS runs dedicated fiber connections between AZs in a region, engineered for consistent low latency. These connections typically maintain sub-millisecond latency between AZs, with multiple redundant paths. This high-bandwidth, low-latency connectivity is what makes synchronous replication practical.

The network between AZs isn't part of the public internet. It's a dedicated network owned and operated by AWS, with quality of service controls that prioritize critical traffic like database replication. This matters because replication performance directly impacts how quickly your database can commit transactions in Multi-AZ deployments.

Multi-AZ Approaches in Amazon RDS

RDS actually offers two different types of Multi-AZ deployments, and the differences matter. Traditional Multi-AZ deployments, which we'll focus on first, use a single primary instance with a standby replica. The newer Multi-AZ DB clusters use a primary instance with two readable standbys. The key difference isn't really the number of standbys, but how replication works.

In traditional Multi-AZ, replication happens at the storage level. When your database writes to disk, that write is synchronously replicated to the standby's EBS volumes before being acknowledged. The standby database instance runs in recovery mode, continuously applying changes it sees in the storage layer.

Multi-AZ DB clusters work differently, using the database engine's native replication. This means the standbys can serve read traffic, and it means replication has different performance characteristics and failure modes. The choice between these approaches depends on your specific needs for read scaling and consistency.

How RDS Multi-AZ Instance Replication Works

When you write data to a Multi-AZ database, several things happen behind the scenes. First, your write operation arrives at the primary instance. The database engine processes it and writes to its local EBS volume. But before acknowledging the write back to your application, that write must be replicated.

The replication process is handled by EBS, not the database engine. EBS synchronously copies each 16KB block that changes to the standby's EBS volumes. When a write occurs, EBS maintains a replication queue for changed blocks. Each block is checksummed and tracked to ensure consistency between volumes. If the queue starts growing too large, RDS will throttle writes to prevent the standby from falling too far behind.

Behind the scenes, EBS also performs continuous consistency checking between volumes. If it detects inconsistent blocks, it will automatically repair them in the background. This process ensures that the standby's storage is truly a consistent copy of the primary, which is crucial for clean failovers.

Only after both the primary and standby volumes have persisted the changes will the write be acknowledged. This ensures zero data loss if a failover occurs, but it also adds latency to every write operation.

The standby instance runs in recovery mode, continuously monitoring its storage for changes and applying them to its internal state. This means it's ready to take over quickly if needed, but it can't serve queries or accept connections while it's in recovery mode.

The replication process adds latency to every write operation. In typical scenarios, you'll see an additional 0.5-1ms for same-AZ writes and 1-2ms for cross-AZ writes. Large writes can take longer, sometimes adding 2-5ms of latency. This might seem small, but it can add up in write-heavy workloads.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Anatomy of an RDS Failover

A failover in RDS isn't a single operation, but a complex sequence of events that happens in several phases. When RDS detects a problem with the primary instance, it doesn't immediately fail over. Instead, it goes through a careful validation process to ensure the failover will succeed.

The detection phase involves multiple health checks. RDS monitors EC2 status checks, EBS volume health, network connectivity, and replication status. It uses a complex decision matrix to determine whether a failure has actually occurred and whether failover is the appropriate response. This process typically takes between 0 and 10 seconds.

Once RDS decides to fail over, it enters the validation phase. It verifies that the standby is healthy, that replication is current, and that all network paths are working. This includes checking storage consistency and ensuring the standby database can actually take over. This typically takes another 5-15 seconds.

The actual failover begins with DNS changes. RDS updates the endpoint's CNAME record to point to the standby instance and adjusts the TTL to 5 seconds to speed up propagation. This process, including propagation time, typically takes 30-60 seconds.

Meanwhile, the promotion phase begins. The standby database stops recovery mode, replays any remaining transactions from its storage, and starts accepting connections. This process typically takes 15-30 seconds, running in parallel with DNS propagation.

Finally, RDS begins provisioning a new standby in the background. This doesn't affect database availability, but it's critical for maintaining high availability for future failures.

Building Applications That Handle RDS Failover

Application design for Multi-AZ isn't just about handling database connection failures. You need to think about transaction retry logic, connection pooling, and how your application behaves during the transition period. Here's a Python example that illustrates some key concepts:

import pymysql
import time
from contextlib import contextmanager

class RDSConnectionManager:
    def __init__(self, host, user, password, database):
        self.db_config = {
            'host': host,
            'user': user,
            'password': password,
            'database': database
        }

    @contextmanager
    def get_connection(self):
        conn = None
        try:
            conn = self._create_connection()
            yield conn
        except pymysql.Error as e:
            if self._should_retry(e):
                time.sleep(2)  # Basic backoff
                conn = self._create_connection()
                yield conn
            else:
                raise
        finally:
            if conn:
                conn.close()

    def _create_connection(self):
        return pymysql.connect(**self.db_config)

    def _should_retry(self, error):
        # Add logic to determine if error is retryable
        return True

This code demonstrates connection handling, but real applications need more sophisticated retry logic and connection pooling. Your application should handle various types of errors. Network timeouts might occur during the DNS switch. Transactions might be rolled back during the promotion phase. Connections might fail with various errors depending on exactly when and how they fail. Each of these scenarios needs appropriate handling.

Monitoring and Troubleshooting

Effective monitoring of Multi-AZ deployments requires watching several CloudWatch metrics. ReplicaLag tells you how far behind the standby is. WriteIOPS and WriteLatency help you understand replication performance. ReadIOPS and ReadLatency on the primary help you understand the workload.

But raw metrics aren't enough. You need to understand how these metrics relate to each other and what patterns indicate problems. High WriteLatency combined with increasing ReplicaLag might indicate replication problems. High CPUUtilization might explain increased ReplicaLag. The relationships between metrics often tell you more than individual metrics alone.

CloudWatch alarms should monitor for both immediate problems and trending issues. A spike in ReplicaLag needs immediate attention, but gradually increasing WriteLatency might indicate growing problems that need addressing before they cause failures.

Advanced Configurations and Edge Cases

Multi-AZ works with various database engines, but the details vary. MySQL and PostgreSQL handle recovery mode differently, which affects failover timing. Oracle has its own nuances around transaction replay. Understanding these engine-specific details helps you design better applications.

Parameter groups also affect Multi-AZ behavior. Settings that control durability and consistency can impact replication performance. Memory settings affect how quickly the standby can catch up after falling behind. Network timeout settings influence how quickly failures are detected.

Edge cases are particularly important to understand. What happens if both AZs have connectivity issues? How does RDS handle simultaneous instance and storage failures? What if DNS propagation is delayed? These scenarios are rare but understanding them helps you build more resilient systems. Note that this doesn't mean you need to ensure your application can handle these scenarios. Not doing anything is a valid response, but only if you understand the risk first.

Through this deep dive into RDS Multi-AZ, we've seen that while AWS handles much of the complexity, understanding the underlying mechanics helps you build better applications. From the basic architecture to complex failure scenarios, each aspect of Multi-AZ deployments has implications for your application's reliability and performance. So, now that you understand how that works in RDS, go build!

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

In this article, we'll explore the key features and capabilities of RDS and Aurora, compare their differences, and provide guidance on choosing the right service for your application.

Understanding Amazon RDS (Relational Database Service)

Let's start by taking a closer look at Amazon RDS, AWS's fully managed relational database service. RDS makes it easy to set up, operate, and scale a relational database in the cloud, supporting a wide range of database engines.

RDS Key Features:

• Fully managed database service

• Supports multiple database engines: MySQL, PostgreSQL, Oracle, SQL Server, MariaDB

• Automatic backups and point-in-time recovery

• Multi-AZ deployments for high availability

• Read replicas for read scalability

• Vertical and horizontal scaling options

RDS Instance Types and Storage:

RDS offers a variety of instance types optimized for different workloads and performance requirements. Instance types range from small burstable instances to large memory-optimized instances.

For storage, RDS provides three options:

• General Purpose (SSD): Balanced performance for a wide range of workloads

• Provisioned IOPS (SSD): High-performance storage for I/O-intensive workloads

• Magnetic: Cost-effective storage for infrequently accessed data

Pricing:

With RDS, you pay for the database instance hours, storage, I/O requests, and data transfer. RDS pricing varies based on the database engine, instance type, storage type, and region.

RDS Backup and Restore

One of the key benefits of using RDS is the automated backup and restore capabilities. RDS provides two types of backups:

• Automated Backups: RDS automatically takes daily snapshots of your database, allowing you to restore to any point in time within the retention period (up to 35 days).

• Manual Snapshots: You can manually create database snapshots at any time, which are stored until you explicitly delete them.

To restore a database from a backup, you simply create a new RDS instance and specify the backup to use. RDS handles the rest, creating a new instance with the restored data.

Point-in-time recovery (PITR) is another powerful feature of RDS. With PITR, you can restore your database to any point in time within the backup retention period, down to the second. This is particularly useful for recovering from accidental data modifications or deletions.

RDS High Availability and Failover

High availability is crucial for many applications, and RDS provides several options to ensure your database remains available in the event of a failure.

Multi-AZ Deployments:

With a Multi-AZ deployment, RDS automatically provisions and maintains a synchronous standby replica in a different Availability Zone (AZ). If the primary instance fails, RDS automatically fails over to the standby, minimizing downtime.

Multi-AZ deployments provide enhanced durability and fault tolerance, with failover typically completing within a minute or two. This is ideal for production workloads that require high availability.

Read Replicas:

Read replicas are separate database instances that are asynchronously replicated from the primary instance. They are used to offload read traffic from the primary instance and improve read scalability.

You can create up to 5 read replicas per primary instance, within the same region or across different regions. Read replicas can be promoted to standalone instances if needed, providing a way to create independent databases for specific use cases.

Understanding Amazon Aurora

Amazon Aurora is a fully managed relational database service that is compatible with MySQL and PostgreSQL. It offers the simplicity and cost-effectiveness of open-source databases with the performance and availability of commercial databases.

Aurora Key Features:

• MySQL and PostgreSQL compatible

• High-performance storage and caching

• Auto-scaling of read replicas

• Serverless option for automatic scaling

• Global database for multi-region deployments

• Continuous backups and point-in-time restore

Aurora Storage and Replication:

Aurora uses a distributed, fault-tolerant, and self-healing storage system that automatically scales up to 128 TB per database instance. It replicates data across multiple AZs, providing high durability and availability.

Aurora's storage is designed for fast, consistent performance. It uses a multi-tier caching architecture that includes an in-memory cache, a buffer pool, and a storage cache, reducing the need for disk I/O and improving performance.

Pricing:

With Aurora, you pay for the database instance hours, storage, I/O requests, and data transfer. Aurora pricing varies based on the database engine (MySQL or PostgreSQL), instance type, and region.

Aurora Performance and Scalability

One of the key advantages of Aurora is its high-performance storage and caching architecture. Aurora can deliver up to 5X the throughput of standard MySQL and 3X the throughput of standard PostgreSQL, without requiring any changes to your application code.

Auto-scaling Read Replicas:

Aurora automatically scales read replicas based on the workload, ensuring your database can handle read-heavy traffic patterns. As the read traffic increases, Aurora seamlessly adds new read replicas to the cluster, distributing the load across multiple instances.

Aurora Serverless:

For applications with unpredictable or intermittent workloads, Aurora Serverless provides a fully managed, auto-scaling configuration for Aurora MySQL and PostgreSQL. With Aurora Serverless, your database automatically starts up, shuts down, and scales capacity based on your application's needs.

This is particularly useful for development and testing environments, or applications with variable traffic patterns, as it eliminates the need to manage database capacity manually.

Aurora Backup and Restore

Like RDS, Aurora provides automated continuous backups and point-in-time restore capabilities. However, Aurora takes it a step further with some additional features.

Continuous Backups:

Aurora automatically takes incremental backups of your database, continuously and transparently, with no impact on performance. These backups are stored in Amazon S3, providing 11 9's of durability.

Backup Retention:

Aurora backups are retained for a default period of 1 day, but you can configure this up to 35 days. Backups are automatically deleted when the retention period expires, or when the DB cluster is deleted.

Point-in-time Restore:

With Aurora, you can restore your database to any point in time within the backup retention period, down to the second. This is similar to RDS PITR, but with the added benefit of Aurora's distributed storage architecture, which enables faster restores.

Database Cloning:

Aurora allows you to create a new database cluster from an existing one, effectively "cloning" the database. This is useful for creating test or development environments, or for performing analytics on a copy of your production data without impacting the live database.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

RDS vs Aurora: Key Differences and Use Cases

Now that we've explored the key features and capabilities of RDS and Aurora, let's compare them side by side.

Use Cases for RDS:

• Applications with moderate performance and scalability requirements

• Workloads that require a specific database engine (e.g., Oracle, SQL Server)

• Migrating existing on-premises databases to the cloud

• Development and testing environments

Use Cases for Aurora:

• Applications with high-performance and high-scalability requirements

• Read-heavy workloads that can benefit from Aurora's caching and auto-scaling capabilities

• Applications with unpredictable or variable traffic patterns (using Aurora Serverless)

• Global applications that require multi-region database deployments

Choosing the Right Relational Database Service

Choosing between RDS and Aurora depends on your specific application requirements and workload characteristics. Here are some key factors to consider:

Performance and Scalability:

If your application demands high performance and scalability, particularly for read-heavy workloads, Aurora is the better choice. Its high-performance storage and caching architecture, along with auto-scaling read replicas, make it well-suited for demanding applications.

Database Engine Compatibility:

If your application requires a specific database engine, such as Oracle or SQL Server, RDS is the way to go. RDS supports a wide range of database engines, making it easier to migrate existing applications to the cloud.

Cost Considerations:

For general-purpose workloads with moderate performance requirements, RDS is more cost-effective than Aurora. However, if your application requires the high performance and scalability of Aurora, the additional cost may be justified.

Existing Skills and Expertise:

If your team is already familiar with MySQL or PostgreSQL, both RDS and Aurora are good choices. However, if you have expertise with a specific database engine supported by RDS, such as Oracle or SQL Server, that may be a deciding factor.

When to Use RDS

• Migrating an existing on-premises database to the cloud

• Applications with moderate performance and scalability requirements

• Workloads that require a specific database engine not supported by Aurora

• Development and testing environments

Example: A web application with a backend database that requires SQL Server compatibility and has moderate traffic and performance requirements.

When to Use Aurora

• Building a new, high-performance application from scratch

• Applications with demanding read-heavy workloads

• Serverless applications with unpredictable traffic patterns

• Global applications that require multi-region database deployments

Example: A large-scale e-commerce platform with millions of daily users, requiring high throughput and low latency for product catalog searches and user profile management.

Best Practices for Running Relational Databases on AWS

Regardless of whether you choose RDS or Aurora, here are some best practices to keep in mind when running relational databases on AWS:

Performance Optimization

• Choose the appropriate instance type and size based on your workload requirements

• Monitor CPU, memory, and I/O utilization to identify bottlenecks and optimize performance

• Use caching solutions like ElastiCache to offload read traffic and improve performance

Security Best Practices

• Use IAM roles and policies to control access to your database instances

• Enable encryption at rest and in transit to protect sensitive data

• Regularly apply security patches and updates to your database engine

• Use VPC security groups to control network access to your database instances

Monitoring and Logging

• Enable and configure Amazon CloudWatch for monitoring database metrics and setting alarms

• Use AWS CloudTrail to log and audit API activity related to your database instances

• Enable database engine-specific logging, such as MySQL slow query logs or PostgreSQL query planner statistics

Scaling and High Availability

• Use read replicas to scale read traffic and improve performance

• Enable Multi-AZ deployments for high availability and automatic failover

• Monitor replication lag and ensure it stays within acceptable limits

• Test failover scenarios regularly to ensure your application can handle database failures gracefully

Conclusion

AWS provides two powerful managed services for running them in the cloud: Amazon RDS and Amazon Aurora. RDS is a fully managed service that supports a wide range of database engines, making it a good choice for general-purpose workloads and migrating existing applications to the cloud. Aurora, on the other hand, is a high-performance, MySQL and PostgreSQL-compatible database service that is well-suited for demanding, read-heavy workloads.

When choosing between RDS and Aurora, it's important to consider your application's specific requirements, including performance, scalability, compatibility, and cost. However, in cases where MySQL or PostgreSQL are suitable, Aurora is generally my preferred choice due to its advanced architecture and auto-scaling capabilities.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

AWS offers two powerful container orchestration services: Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Both services help you run and scale containerized applications, but they differ in their approach, features, and use cases.

In this article, I'll dive deep into the world of containers on AWS. I'll explore the key features and components of ECS and EKS, compare their similarities and differences, and provide guidance on choosing the right service for your needs. By the end, you'll have a solid understanding of how to leverage these services to build and manage containerized applications on AWS effectively.

Understanding Amazon ECS (Elastic Container Service)

Let's start by looking at Amazon ECS, AWS's fully managed container orchestration service. ECS allows you to run and manage Docker containers at scale without worrying about the underlying infrastructure.

ECS Key Features:

• Fully managed container orchestration

• Integration with other AWS services

• Support for both EC2 and Fargate launch types

• Built-in service discovery and load balancing

• IAM integration for security and access control

ECS Components:

• Clusters: Logical grouping of container instances or Fargate capacity

• Task Definitions: Blueprints that describe how to run a container

• Services: Maintain a specified number of task replicas and handle scaling

• Tasks: Instantiation of a Task Definition, representing a running container

ECS Launch Types:

ECS supports two launch types for running containers: EC2 and Fargate.

• EC2: You manage the EC2 instances that make up the ECS cluster. This gives you full control over the infrastructure but requires more management overhead.

• Fargate: AWS manages the underlying infrastructure, and you only pay for the resources your containers consume. Fargate abstracts away the EC2 instances, making it easier to focus on your applications.

Pricing:

With ECS, you pay for the AWS resources you use, such as EC2 instances, EBS volumes, and data transfer. Fargate pricing is based on the vCPU and memory resources consumed by your containers.

ECS Architecture and Components

Let's take a closer look at the key components of ECS and how they work together.

ECS Clusters

An ECS cluster is a logical grouping of container instances or Fargate capacity. It provides the infrastructure to run your containers. You can create clusters using the AWS Management Console, AWS CLI, or CloudFormation templates.

Task Definitions

A Task Definition is a JSON file that describes how to run a container. It specifies the container image, CPU and memory requirements, networking settings, and other configuration details. Task Definitions act as blueprints for creating and running tasks.

Services

An ECS Service maintains a specified number of task replicas and handles scaling. It ensures that the desired number of tasks are running and automatically replaces any failed tasks. Services integrate with ELB for load balancing and service discovery.

Tasks

A Task is an instantiation of a Task Definition, representing a running container. When you create a task, ECS launches the container on a suitable container instance or Fargate capacity, based on the Task Definition and launch type.

Example ECS Cluster and Task Definition:

{
  "cluster": "my-cluster",
  "taskDefinition": "my-task-definition",
  "desiredCount": 2,
  "launchType": "FARGATE",
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": [
        "subnet-12345678",
        "subnet-87654321"
      ],
      "securityGroups": [
        "sg-12345678"
      ],
      "assignPublicIp": "ENABLED"
    }
  }
}

ECS Launch Types: EC2 vs Fargate

One key decision when using ECS is choosing between the EC2 and Fargate launch types.

EC2 Launch Type

With the EC2 launch type, you manage the EC2 instances that make up your ECS cluster. This gives you full control over the infrastructure, including instance types, scaling, and networking. However, it also means more management overhead, as you're responsible for patching, scaling, and securing the instances.

Use cases for EC2 launch type:

• Workloads that require specific instance types or configurations

• Applications that need to access underlying host resources

• Scenarios where you want full control over the infrastructure

Fargate Launch Type

Fargate is a serverless compute engine for containers. It abstracts away the underlying infrastructure, allowing you to focus on your applications. With Fargate, you specify the CPU and memory requirements for your tasks, and ECS manages the rest.

Benefits of Fargate:

• No need to manage EC2 instances or clusters

• Pay only for the resources your containers consume

• Automatic scaling based on task resource requirements

• Simplified infrastructure management

Example of running a containerized application using Fargate:

Resources:
  MyFargateService:
    Type: AWS::ECS::Service
    Properties:
      Cluster: !Ref MyCluster
      TaskDefinition: !Ref MyTaskDefinition
      DesiredCount: 2
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          Subnets:
            - !Ref SubnetA
            - !Ref SubnetB
          SecurityGroups:
            - !Ref MySecurityGroup

Understanding Amazon EKS (Elastic Kubernetes Service)

Now let's shift gears and explore Amazon EKS, a managed Kubernetes service that makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS.

EKS Key Features:

• Fully managed Kubernetes control plane

• Integration with AWS services and Kubernetes community tools

• Automatic provisioning and scaling of worker nodes

• Support for both managed and self-managed node groups

• Built-in security and compliance features

EKS Architecture

EKS consists of two main components:

• EKS Control Plane: The control plane is a managed Kubernetes master that runs in an AWS-managed account. It provides the Kubernetes API server, etcd, and other core components.

• Worker Nodes: Worker nodes are EC2 instances that run your containers and are registered with the EKS cluster. You can create and manage worker nodes using EKS managed node groups or self-managed worker nodes.

Pricing:

With EKS, you pay for the AWS resources you use, such as the EKS control plane, EC2 instances for worker nodes, EBS volumes, and data transfer. You also pay a hourly rate for the EKS control plane based on the number of Kubernetes API requests.

EKS Architecture and Components

Let's dive deeper into the EKS architecture and its key components.

EKS Control Plane

The EKS control plane is a managed Kubernetes master that runs in an AWS-managed account. It provides the following components:

• Kubernetes API Server: The primary interface for interacting with the Kubernetes cluster

• etcd: The distributed key-value store used by Kubernetes to store cluster state

• Scheduler: Responsible for scheduling pods onto worker nodes based on resource requirements and constraints

• Controller Manager: Manages the core control loops in Kubernetes, such as replica sets and deployments

Worker Nodes

Worker nodes are EC2 instances that run your containers and are registered with the EKS cluster. Each worker node runs the following components:

• Kubelet: The primary node agent that communicates with the Kubernetes API server and manages container runtime

• Container Runtime: The runtime environment for running containers, such as Docker or containerd

• Kube-proxy: Maintains network rules and performs connection forwarding for Kubernetes services

Example EKS Cluster Configuration:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: my-eks-cluster
  region: us-west-2

managedNodeGroups:
  - name: my-node-group
    instanceType: t3.medium
    minSize: 1
    maxSize: 3
    desiredCapacity: 2

EKS Managed vs Self-Managed Node Groups

EKS provides two options for managing worker nodes: managed node groups and self-managed worker nodes.

EKS Managed Node Groups

EKS managed node groups automate the provisioning and lifecycle management of worker nodes. Key features include:

• Automatic provisioning and scaling of worker nodes

• Integration with AWS services like VPC and IAM

• Managed updates and patching for worker nodes

• Simplified cluster autoscaler configuration

Self-Managed Worker Nodes

With self-managed worker nodes, you have full control over the provisioning and management of worker nodes. This allows for more customization but also requires more effort to set up and maintain.

Example of creating an EKS managed node group:

eksctl create nodegroup --cluster my-eks-cluster --name my-node-group --node-type t3.medium --nodes 2 --nodes-min 1 --nodes-max 3

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

ECS vs EKS: Key Differences and Use Cases

Now that we've explored the key features and components of ECS and EKS, let's compare them side by side.

Use cases for ECS:

• Simpler containerized applications

• Workloads that heavily utilize AWS services

• Teams more familiar with AWS ecosystem

• Serverless applications using Fargate

Use cases for EKS:

• Complex, large-scale containerized applications

• Workloads that require Kubernetes-specific features

• Teams with Kubernetes expertise

• Applications that need to be portable across cloud providers

Choosing the Right Container Orchestration Service

Choosing between ECS and EKS depends on various factors specific to your application and organizational needs.

Factors to consider:

• Application complexity and scalability

• Team's skills and familiarity with AWS and Kubernetes

• Integration with existing tools and workflows

• Long-term container strategy and portability requirements

When to use ECS

• Simpler applications with a limited number of microservices

• Workloads that primarily use AWS services

• Teams more comfortable with AWS tools and concepts

• Serverless applications that can benefit from Fargate

Example: A web application consisting of a frontend service, backend API, and database, all running on ECS with Fargate.

When to use EKS

• Complex applications with a large number of microservices

• Workloads that require Kubernetes-specific features like Custom Resource Definitions (CRDs)

• Teams with extensive Kubernetes experience

• Applications that need to be portable across cloud providers

Example: A large-scale machine learning platform running on EKS, leveraging Kubeflow and other Kubernetes-native tools.

Best Practices for Container Orchestration on AWS

Regardless of whether you choose ECS or EKS, here are some best practices to keep in mind:

• Use infrastructure as code (IaC) tools like CloudFormation or Terraform to manage your container orchestration resources

• Implement a robust CI/CD pipeline to automate container builds, testing, and deployment

• Leverage AWS services like ECR for container image registry and ELB for load balancing

• Use IAM roles and policies to enforce least privilege access to AWS resources

• Monitor your containerized applications using tools like CloudWatch, Prometheus, or Grafana

• Optimize costs by right-sizing your instances, using Spot Instances when appropriate, and leveraging reserved capacity

Conclusion

AWS provides two powerful services for container orchestration: ECS and EKS.

ECS is a fully managed service that offers simplicity and deep integration with the AWS ecosystem. It's well-suited for simpler containerized applications and teams more familiar with AWS tools and concepts.

On the other hand, EKS is a managed Kubernetes service that provides the full power and flexibility of Kubernetes. It's ideal for complex, large-scale applications and teams with Kubernetes expertise.

Ultimately, the choice between ECS and EKS depends on your application requirements, team skills, and long-term container strategy. By understanding the key features, differences, and use cases of each service, you can make an informed decision and build scalable, resilient containerized applications on AWS.

Still, I prefer ECS =)

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

In this article, we'll dive deep into IAM best practices and advanced techniques to help you lock down your AWS environment like a pro. We'll start with the fundamentals, then move on to more advanced topics like granular access control, cross-account access, and automating IAM with Infrastructure as Code. By the end, you'll have a solid understanding of how to use IAM to secure your AWS resources and protect your sensitive data.

Understanding IAM Fundamentals

Before we jump into the best practices and advanced techniques, let's make sure we're all on the same page with the IAM basics. Understanding these foundational concepts is crucial for designing and implementing an effective IAM strategy.

IAM Users, groups, and roles

At the core of IAM are three main identity types: users, groups, and roles. IAM users represent individual people or applications that need access to your AWS resources. IAM groups are collections of IAM users, making it easier to manage permissions for multiple users at once. IAM roles are a bit different: they're not associated with a specific user, but rather are used by AWS services or external identities that need temporary access to your resources.

IAM policies and permissions

IAM policies are JSON documents that define permissions for IAM identities. They specify what actions an identity can perform on which AWS resources. Policies can be attached to IAM users, groups, or roles, or even directly to AWS resources (more on that later).

Resource-based policies vs. identity-based policies

There are two main types of IAM policies: identity-based policies and resource-based policies. Identity-based policies are attached to IAM identities (users, groups, or roles) and define what actions those identities can perform on which resources. Resource-based policies, on the other hand, are attached directly to AWS resources (like S3 buckets or KMS keys) and define who can access those resources and what actions they can perform.

How IAM interacts with other AWS services

IAM is deeply integrated with other AWS services. It's used to control access to virtually every AWS resource, from EC2 instances to S3 buckets to Lambda functions. Many AWS services also have their own resource-based policies that work in conjunction with IAM policies to provide fine-grained access control.

IAM Best Practices

Now that we've got the fundamentals down, let's dive into some IAM best practices that every AWS user should follow.

Principle of least privilege

The principle of least privilege is the golden rule of IAM. It means only granting users the permissions they need to perform their job duties; no more, no less. This helps minimize the blast radius if a user's credentials are compromised, and makes it easier to audit and manage permissions over time.

Proper IAM user and role management

Managing IAM users and roles can get complex, especially in large organizations. Some key best practices include:

• Create individual IAM users for each person who needs access to AWS, rather than sharing credentials

• Use IAM roles for applications and services that need access to AWS resources

• Regularly review and remove unused IAM users and roles

Using IAM groups for better organization

IAM groups make it easier to manage permissions for multiple users at once. By creating groups for different job functions or teams, you can assign permissions at the group level rather than individually. This makes it easier to onboard new users and ensure consistent permissions across your organization.

Password policies and MFA enforcement

Strong password policies and multi-factor authentication (MFA) are critical for protecting your IAM users. AWS allows you to set password policies that enforce minimum length, complexity, and rotation requirements. You should also require MFA for all IAM users, especially those with administrative privileges.

Regularly reviewing and rotating IAM credentials

Over time, IAM users can accumulate unnecessary permissions, and credentials can become stale or compromised. That's why it's important to regularly review IAM users and their permissions, and rotate access keys and passwords on a regular basis. AWS recommends rotating access keys every 90 days, and immediately revoking credentials for users who leave your organization.

Avoiding use of root user account

The root user account has unrestricted access to all AWS resources in your account, making it a prime target for attackers. Best practice is to avoid using the root user account for day-to-day tasks, and instead create individual IAM users with specific permissions. You should also enable MFA on the root user account and use it only for tasks that absolutely require root privileges.

Implementing Granular Access Control

One of the most powerful features of IAM is the ability to create fine-grained policies that precisely control access to your AWS resources. Here are some techniques for implementing granular access control:

Creating fine-grained IAM policies

When creating IAM policies, it's important to be as specific as possible. Instead of granting broad permissions like s3:*, grant only the specific actions needed, like s3:GetObject or s3:PutObject. You can also restrict access to specific resources using ARNs (Amazon Resource Names), and limit permissions to specific IP ranges or VPC endpoints.

Using policy conditions for more precise control

IAM policy conditions allow you to further refine permissions based on specific criteria. For example, you can use conditions to allow access only during certain time windows, from specific IP ranges, or for requests that include certain headers or parameters.

Leveraging IAM policy variables

IAM policy variables allow you to create dynamic policies that adapt to your environment. For example, you can use the aws:username variable to grant users access to their own home directory in an S3 bucket, or the aws:SourceIp variable to restrict access based on the requester's IP address.

Combining multiple policies for complex permissions

In some cases, you may need to combine multiple policies to achieve the desired level of access control. For example, you might use an identity-based policy to grant broad permissions to a group of users, then use a resource-based policy to further restrict access to specific resources.

Real-world examples of granular access control in AWS

Let's look at a couple real-world examples of granular access control in action:

Granting read-only access to an S3 bucket for a specific IAM user

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOnlyAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ]
    }
  ]
}

Allowing an EC2 instance to access S3, but only from a specific VPC endpoint

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AccessFromVPCEndpoint",
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:sourceVpce": "vpce-1a2b3c4d"
        }
      }
    }
  ]
}

Cross-Account Access and IAM Roles

In many organizations, you'll need to grant access to AWS resources across multiple accounts. That's where IAM roles and cross-account access come in.

Understanding cross-account access

Cross-account access allows IAM users or roles in one AWS account to access resources in another account. This is useful for scenarios like granting developers access to a production account, or allowing a central security team to monitor multiple accounts.

Using IAM roles for secure access delegation

IAM roles are the preferred way to grant cross-account access. Instead of sharing access keys or passwords, you create an IAM role in the target account and grant permissions to the trusted entity (user or role) in the source account. The trusted entity can then assume the role and access resources in the target account.

Assuming roles vs. using access keys

When accessing resources across accounts, it's best to assume an IAM role rather than using access keys. Access keys are long-term credentials that can be easily leaked or compromised, while IAM roles provide temporary, short-lived credentials that automatically expire.

Best practices for managing cross-account access

Some best practices for managing cross-account access include:

• Use IAM roles for cross-account access instead of sharing long-term access keys

• Limit the permissions granted to cross-account roles to the minimum necessary

• Regularly review and audit cross-account access

• Use external ID's to prevent the confused deputy problem

Securing Access to AWS Resources

In addition to identity-based policies, AWS also supports resource-based policies that allow you to control access to specific resources like S3 buckets, KMS keys, and Lambda functions.

Using resource-based policies (e.g., S3 bucket policies)

Resource-based policies are attached directly to an AWS resource and define who can access that resource and what actions they can perform. For example, an S3 bucket policy can allow read access to objects from a specific IP range, or deny all public access to the bucket.

Combining resource-based and identity-based policies

Resource-based policies work in conjunction with identity-based policies to provide comprehensive access control. When an IAM user or role tries to access a resource, AWS evaluates both the identity-based policies attached to the user/role and the resource-based policy attached to the resource. Access is granted only if both policies allow it.

VPC endpoints and IAM policies

VPC endpoints allow you to securely access AWS services from within your VPC, without traversing the public internet. You can use IAM policies to control access to VPC endpoints, ensuring that only authorized users or roles can access the services behind the endpoint.

Securing access to API Gateway and Lambda

API Gateway and Lambda are powerful tools for building serverless applications, but they also introduce new security challenges. Best practices for securing access to these services include:

• Use IAM roles to grant Lambda functions access to other AWS services

• Implement OAuth or JWT authentication for APIs

• Use API keys and usage plans to control access to APIs

• Enable AWS WAF to protect against common web exploits

Protecting sensitive data with KMS and IAM

AWS Key Management Service (KMS) allows you to encrypt your sensitive data using centrally managed keys. IAM policies can be used to control access to KMS keys, ensuring that only authorized users or roles can encrypt or decrypt data.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Centralized IAM Management with AWS Organizations

For organizations with multiple AWS accounts, managing IAM across all those accounts can be a challenge. That's where AWS Organizations comes in.

Benefits of using AWS Organizations

AWS Organizations allows you to centrally manage access across multiple accounts. You can create an organization, invite accounts to join, and then use Service Control Policies (SCPs) to enforce IAM policies across all accounts in the organization.

Setting up an organization and creating member accounts

To get started with AWS Organizations, you create an organization and invite existing accounts to join, or create new accounts directly within the organization. You can organize accounts into Organizational Units (OUs) to apply policies hierarchically.

Implementing Service Control Policies (SCPs)

Service Control Policies are a powerful feature of AWS Organizations that allow you to centrally control what actions can be performed by IAM users and roles across all accounts in your organization. SCPs are similar to IAM policies, but they apply at the account level and can be used to enforce security best practices and compliance requirements.

Delegating access across accounts with IAM roles

In addition to SCPs, AWS Organizations also simplifies cross-account access using IAM roles. You can create a role in a central account and grant access to users or roles in other accounts within the organization. This allows you to centrally manage permissions while still enabling teams to access the resources they need.

Best practices for AWS Organizations

Some best practices for using AWS Organizations include:

• Use SCPs to enforce security best practices and compliance requirements

• Implement a least privilege model, granting only the permissions necessary for each account

• Use AWS CloudTrail to monitor IAM activity across all accounts

• Regularly review and audit IAM policies and roles

• Use automation tools like AWS CloudFormation to manage IAM resources consistently across accounts

Monitoring and Auditing IAM Activity with AWS CloudTrail

Monitoring and auditing IAM activity is critical for detecting and responding to security incidents. AWS CloudTrail is a powerful tool for tracking IAM activity across your AWS accounts.

Importance of monitoring IAM events

By monitoring IAM events, you can detect suspicious activity like unauthorized access attempts, changes to IAM policies, or creation of new IAM users or roles. This allows you to quickly investigate and respond to potential security breaches.

Using AWS CloudTrail to track IAM actions

AWS CloudTrail logs all API calls made to IAM, including who made the call, what actions were performed, and what resources were affected. You can use CloudTrail to create a complete audit trail of IAM activity in your account.

Monitoring IAM events with Amazon CloudWatch

In addition to CloudTrail, you can use Amazon CloudWatch to monitor IAM events in real-time. CloudWatch allows you to create alarms based on specific IAM events, like failed login attempts or changes to sensitive policies.

Detecting and alerting on suspicious IAM activity

By combining CloudTrail and CloudWatch, you can create a comprehensive monitoring and alerting system for IAM. Some best practices include:

• Create alarms for high-risk events like IAM policy changes or root account usage

• Use CloudTrail Insights to detect unusual activity patterns

• Integrate with SIEM tools like Splunk or AWS Security Hub for centralized monitoring

Conducting regular IAM audits and compliance checks

In addition to real-time monitoring, it's important to conduct regular IAM audits to ensure your policies and permissions are configured correctly and comply with your security and compliance requirements. Tools like AWS IAM Access Analyzer and AWS Config can help automate this process.

Advanced IAM Security Features

These are some more advanced features of AWS IAM, or some related services that will help you secure your AWS accounts and workloads.

IAM Access Analyzer

AWS IAM Access Analyzer is a powerful tool for identifying unintended access to your AWS resources. It analyzes your IAM policies and resource-based policies to determine who has access to your resources and whether that access is intended.

IAM Access Analyzer can help you identify scenarios like:

• Public access to S3 buckets or other resources

• Access granted to external AWS accounts

• Overly permissive IAM policies

By identifying these issues early, you can take corrective action before they lead to a security breach.

IAM Permission Boundaries

IAM Permission Boundaries are a way to limit the maximum permissions that can be granted to an IAM user or role. They're useful for scenarios like allowing developers to create their own IAM policies, but ensuring they can't grant themselves excessive permissions.

To implement a permission boundary, you create an IAM policy that defines the maximum permissions allowed, then attach that policy as a permission boundary to an IAM user or role. Any policies attached to the user or role are evaluated within the constraints of the permission boundary.

IAM Policy Conditions

IAM Policy Conditions allow you to create more fine-grained access control policies based on specific attributes of a request, like the source IP address, time of day, or presence of multi-factor authentication.

Some examples of using IAM policy conditions include:

• Allowing access only during business hours

• Requiring multi-factor authentication for sensitive actions

• Restricting access to specific IP ranges or VPC endpoints

IAM Identity Center for AWS SSO

IAM Identity Center (formerly AWS Single Sign-On) is a centralized access management service that allows users to sign in once and access multiple AWS accounts and cloud applications.

With IAM Identity Center, you can create and manage user identities in a central directory, then assign permissions to those users across multiple AWS accounts. Users sign in once to the IAM Identity Center portal, then access their assigned accounts and applications without needing to manage separate credentials.

Integrating IAM Identity Center with third-party identity providers

IAM Identity Center also allows you to integrate with third-party identity providers like Azure AD, Okta, or Ping Identity. This allows you to use your existing identity management system to control access to AWS, without needing to recreate user identities in IAM.

Automating IAM with Infrastructure as Code Tools

As your AWS environment grows, managing IAM policies and roles manually becomes increasingly difficult. That's where Infrastructure as Code (IaC) tools like AWS CloudFormation, Terraform, and the AWS CDK come in.

Benefits of using Infrastructure as Code (IaC) for IAM

By defining your IAM resources as code, you can:

• Version control your IAM policies and roles

• Automate the creation and updates of IAM resources

• Ensure consistency across multiple AWS accounts and regions

• Easily roll back changes if needed

Using AWS CloudFormation to manage IAM resources

AWS CloudFormation is a native AWS service that allows you to define your infrastructure as code using JSON or YAML templates. You can use CloudFormation to create and manage IAM users, groups, roles, and policies across multiple accounts and regions.

Terraform and AWS CDK for IAM automation

Terraform and the AWS Cloud Development Kit (CDK) are popular third-party IaC tools that support IAM resource management. Terraform uses a declarative language called HCL (HashiCorp Configuration Language) to define infrastructure resources, while the AWS CDK allows you to define infrastructure using familiar programming languages like JavaScript, TypeScript, Python, or Java.

Best practices for IAM automation and version control

When automating IAM with IaC tools, it's important to follow best practices like:

• Storing your IaC templates in a version control system like Git

• Using separate AWS accounts for development, staging, and production environments

• Implementing a code review process for IAM changes

• Using tools like AWS CloudTrail and AWS Config to monitor and audit IAM changes

By treating your IAM resources as code and following these best practices, you can ensure consistency, maintainability, and auditability of your IAM configuration.

Conclusion

IAM is a critical component of securing your AWS environment, but it can be really complex and challenging to manage at scale. By following best practices like the principle of least privilege, using IAM roles for cross-account access, and implementing strong password policies and MFA, you can lay a solid foundation for your IAM strategy.

But to truly secure your accounts and environments, you need to go beyond the basics. Techniques like granular access control with policy conditions, resource-based policies, and permission boundaries allow you to implement fine-grained security policies that precisely control access to your resources. Centralized management with AWS Organizations and monitoring with CloudTrail and CloudWatch provide visibility and actionable data across your entire AWS environment.

As your AWS usage grows, automating IAM with Infrastructure as Code tools like CloudFormation, Terraform, and the AWS CDK becomes increasingly important. By defining your IAM resources as code and following best practices for version control and testing, you can ensure consistency and maintainability of your IAM configuration.

Securing your AWS environment is an ongoing process, not a one-time task. As you adopt new AWS services and your application requirements evolve, it's important to continually review and update your IAM policies to ensure they align with your security goals. Regular audits and compliance checks, along with automated monitoring and alerting, can help you stay on top of your IAM configuration and quickly detect and respond to potential issues.

By following the best practices and techniques outlined in this article, you can build a robust and secure IAM strategy that helps you protect your critical AWS resources and data. But don't stop here! Continue to explore and adopt new security services and features like AWS GuardDuty, AWS Security Hub, and AWS Secrets Manager to further strengthen your security posture.

Remember, security is a shared responsibility between AWS and you, the customer. By taking a proactive and layered approach to IAM and security, you can ensure that your AWS environment is protected against evolving threats and ready to support your business needs for years to come.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

Amazon Simple Storage Service (S3) is an object storage service by AWS that can store any kind of information. S3 is known for its durability, availability, and scalability, and the fact that all of these features come out of the box makes S3 a go-to solution for a wide range of data storage needs.

In S3 users create 'buckets' – containers for data stored in the AWS cloud. Storing data in buckets serves various use cases, from website hosting to backup and recovery, data archiving, and big data analytics.

S3 Storage Classes

Amazon S3 offers several storage classes designed for different use cases:

1. S3 Standard: For frequently accessed data. You're billed per storage and per request.

2. S3 Standard-IA (Infrequent Access): For data that is accessed less frequently but requires rapid access when needed. Lower fee per GB stored than Standard, but a higher fee per request.

3. S3 One Zone-IA: Similar to Standard-IA, but data is stored in a single Availability Zone only, and it's also cheaper.

4. S3 Express One Zone: High-performance storage for your most frequently accessed data.

5. S3 Intelligent-Tiering: Automatically moves data between the Standard and Standard-IA tiers based on continuously evaluating your access patterns. Ideal for data with unknown or changing access patterns.

6. S3 Glacier: For long-term archival. Very low storage cost, but retrieving data can take several hours and is even more expensive than Standard-IA.

7. S3 Glacier Deep Archive: Amazon S3's lowest-cost storage class for long-term archiving where data retrieval times of 12 hours or more are acceptable.

S3 Pricing Explained

As mentioned above, the different storage classes have different prices. Here are the prices for each S3 storage class:

Pricing for S3 Standard

• Storage: $0.023 per GB for the first 50 TB, $0.022 per GB for the next 450 TB, $0.021 per GB for storage over 500 TB.

• Access: $0.005 per 1000 PUT, COPY, POST, LIST requests. $0.0004 per 1000 GET, SELECT requests.

• Data Retrieval: $0.00 per GB

• Other charges: None

Pricing for S3 Standard-IA (Infrequent Access)

• Storage: $0.0125 per GB

• Access: $0.01 per 1000 PUT, COPY, POST, LIST requests. $0.001 per 1000 GET, SELECT requests.

• Data Retrieval: $0.01 per GB

• Other charges: $0.01 per Lifecycle Transition request

Pricing for S3 One Zone-IA

• Storage: $0.01 per GB

• Access: $0.01 per 1000 PUT, COPY, POST, LIST requests. $0.001 per 1000 GET, SELECT requests.

• Data Retrieval: $0.01 per GB

• Other charges: $0.01 per Lifecycle Transition request

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Pricing for S3 Express One Zone

• Storage: $0.16 per GB

• Access: $0.0025 per 1000 PUT, COPY, POST, LIST requests. $0.0002 per 1000 GET, SELECT requests.

• Data Retrieval: $0.00 per GB

• Other charges: None

Pricing for S3 Intelligent-Tiering

• Storage:
  Frequent Access tier: $0.023 per GB for the first 50 TB, $0.022 per GB for the next 450 TB, $0.021 per GB for storage over 500 TB.
  Infrequent Access tier: $0.0125 per GB.
  Archive Instant Access tier: $0.004 per GB.

• Access: $0.005 per 1000 PUT, COPY, POST, LIST requests. $0.0004 per 1000 GET, SELECT requests.

• Data Retrieval: $0.00 per GB

• Other charges: $0.0025 per 1,000 objects

Pricing for S3 Glacier

• Storage:
  Instant Retrieval: $0.004 per GB
  Flexible Retrieval: $0.0036 per GB

• Access:
  Instant Retrieval: $0.02 per 1000 PUT, COPY, POST, LIST requests. $0.01 per 1000 GET, SELECT requests.
  Flexible Retrieval: $0.03 per 1000 PUT, COPY, POST, LIST requests. $0.0004 per 1000 GET, SELECT requests.

• Data Retrieval:
  Instant Retrieval: $0.03 per GB
  Flexible Retrieval: $0.03 per GB for Expedited, $0.01 per GB for Standard

• Other charges:
  Instant Retrieval: $0.02 per Lifecycle Transition request
  Flexible Retrieval: $0.03 per Lifecycle Transition request

Pricing for S3 Glacier Deep Archive

• Storage: $0.00099 per GB

• Access: $0.05 per 1000 PUT, COPY, POST, LIST requests. $0.0004 per 1000 GET, SELECT requests.

• Data Retrieval: $0.02 per GB for Standard, $0.0025 per GB for Bulk

• Other charges: $0.05 per Lifecycle Transition request

Pricing Examples for S3 Storage Classes

To give you a clearer picture of how S3 pricing works, let's see some examples. For each example, assume the following:

• Storage: 100 GB

• Access: 100,000 GET requests, 10,000 PUT requests

• Data Retrieval: 100 GB

Example 1: S3 Standard Storage

• Storage Cost: $2.30

• Access Cost: $90

• Data Retrieval Cost: $0

• Total Cost: $92.30

Example 2: S3 Express One Zone

• Storage Cost: $16

• Access Cost: $200

• Data Retrieval Cost: $0

• Total Cost: $216

Example 3: S3 Standard-IA

• Storage Cost: $1.25

• Access Cost: $200

• Data Retrieval Cost: $10

• Total Cost: $211.25

AWS S3 Free Tier

AWS offers a free tier for S3, which includes:

• 5 GB of Standard Storage

• 20,000 GET Requests

• 2,000 PUT, COPY, POST, or LIST Requests

This free tier is a great way to start experimenting with S3 without incurring immediate costs. Also, for really small uses like MVPs you end up paying $0 initially, and your costs only grow as you acquire more users.

Tips for Optimizing AWS S3 Costs

1. Understand Your Data Usage: Analyze your data access patterns to choose the most cost-effective storage class.

2. Monitor Your S3 Billing: Regularly check your AWS billing dashboard to track your S3 usage and costs.

3. Leverage S3 Lifecycle Policies: Automatically move or archive data to lower-cost storage classes.

4. Use S3 Analytics: Monitor and analyze storage access patterns for cost optimization.

The goal of this guide was to help you understand AWS S3 pricing. Now you're able to use the best storage classes for your use cases, minimizing cost while maintaining durability and availability.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

What is AWS Lambda?

AWS Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources for you. This service is capable of executing code in various languages and is commonly used for applications such as web application backends, data processing, and real-time file processing.

How AWS Lambda Works

• Event-Driven Execution: AWS Lambda is designed to run code in response to triggers such as changes in data within AWS services (like S3 or DynamoDB), requests to an API Gateway, or direct invocations via SDKs.

• Automatic Scaling: The service scales automatically, executing code in parallel and handling each trigger individually.

• Flexible Resource Allocation: Compute power is allocated based on the memory configured for your function, ensuring efficient resource utilization.

Key Components of AWS Lambda

• Lambda Functions: The core unit where your code resides, along with associated configuration information such as the function name, memory, and timeout settings.

• Event Sources: These are AWS services or custom sources that trigger your Lambda function.

• Logs and Monitoring: Integration with AWS CloudWatch ensures detailed monitoring and logging of your Lambda functions.

• Runtime Environments: Supports multiple programming languages and runtimes.

Understanding AWS Lambda Pricing

AWS Lambda's pricing is primarily based on two components: the number of requests your functions process and the compute time they consume. Understanding these components in detail, including their cost, is crucial for effectively managing your AWS Lambda expenses. Here's an expanded breakdown:

1. Requests:

   • Cost: AWS Lambda charges $0.20 per 1 million requests.

   • What It Means: Every time your function is triggered and executed, it counts as a request.

2. Compute Time:

   • Cost: Compute time is charged at $0.00001667 for every GB-second used.

   • Calculation: The cost is based on the amount of memory allocated to your function and the time it takes to execute.

   • GB-Second: A GB-second is a measure that combines memory usage and execution time. If your function uses 512MB of memory and runs for 3 seconds, it consumes 1.5 GB-seconds (0.5 GB * 3 seconds).

AWS Lambda Free Tier

AWS offers a generous free tier for Lambda:

• 1 million free requests per month.

• 400,000 GB-seconds of compute time per month.

Pricing Examples for AWS Lambda

To illustrate how Lambda pricing works, let's consider a few examples:

• Example 1: Low Frequency, Simple Function

  • Requests: 100,000 in a month

  • Duration: Each request runs for 500ms with 128MB memory allocation.

  • Total Cost: $0.02 for invocations + $0.1042 for execution time = $0.1242 / month.

• Example 2: High Frequency, Complex Function

  • Requests: 10 million in a month

  • Duration: Each request runs for 800ms with 256MB memory allocation.

  • Total Cost: $2.00 for invocations + $33.34 for execution time = $35.34 / month.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Tips for Optimizing AWS Lambda Costs

• Monitor Function Invocations: Regularly review your Lambda function metrics through AWS CloudWatch to understand your usage patterns.

• Adjust Memory Allocation: Optimize the memory allocation for your functions to balance performance and cost.

• Reduce Execution Time: Optimize your code to run faster, which directly reduces the compute time cost.

• Regularly Review Your Architecture: As your application evolves, continually reassess whether your use of Lambda aligns with your operational requirements and cost objectives.

• Leverage Free Tier: Make the most out of the AWS Lambda Free Tier, especially for development and testing purposes.

Conclusion

AWS Lambda offers a flexible, cost-effective solution for running code in response to events. By understanding its pricing model and effectively managing your usage, you can leverage Lambda to build scalable, efficient applications without worrying about infrastructure management.

The goal of this guide is to help you gain a better understanding of AWS Lambda's pricing structure, enabling you to use this fantastic service more efficiently while keeping your AWS costs manageable.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

Disaster Recovery and Business Continuity

Disasters are events that cause critical damage to our ability to operate as a business. Consider an earthquake near your datacenter (or the ones you're using in AWS), or a flood in that city (this happened to GCP in Paris in the second half of 2023). It follows that Business Continuity is the ability to continue operating (or recovering really fast) in the event of a Disaster. The big question is: How do we do that?

First, let's understand what recovering looks like, and how much data and time can we lose (yes, we lose both) in the process. There are two objectives that we need to set:

Recovery Point Objective (RPO)

The RPO is the maximum time that passes between when the data is written to the primary storage and when it's written to the backup. For periodic backups, RPO is equal to the time between backups. For example, if you take a snapshot of your database every 12 hours, your RPO is 12 hours. For continuous replication, the RPO is equal to the replication delay. For example, if you continuously replicate data from the primary storage to a secondary one, the RPO is the delay in that replication.

Data that hasn't yet been written to the backup won't be available in the event of a disaster, so you want your RPO to be as small as possible. However, minimizing it may require adopting new technologies, which means effort and money. Sometimes it's worth it, sometimes it isn't.

Different data may require different RPOs. Since the easiness of achieving a low RPO mostly depends on what technologies you use, the decision of what the RPO is for a specific set of data should be considered when selecting where to store it.

Recovery Time Objective (RTO)

The RTO is the maximum time that can pass from when a failure occurs to when you're operational again. The thing that will have the most impact on RTO is your disaster recovery strategy, which we'll see a bit further down this article. Different technologies will let you reduce the RTO within the same DR strategy, and a technology change may be a good way to reduce RTO without significantly increasing costs.

Stages of a Disaster Recovery Process

These are the three stages that a disaster recovery process goes through, always in this order.

Detect

Detection is the phase between when the failure actually occurs and when you start doing something about it. The absolute worst way to learn about a failure is from a customer, so detection should be the first thing you automate. The easiest way to do so is through a health check, which is a sample request sent periodically (e.g. every 30 seconds) to your servers. For example, Application Load Balancer implements this to detect whether targets in a target group are healthy, and can raise a CloudWatch Alarm if it has no healthy targets. You can connect that alarm to SNS to receive an email when that happens, and you'd have automated detection.

Escalate and Declare

This is the phase from when the first person is notified about an event 🔥 and when the alarm 🚨 sounds and everyone is called to battle stations 🚒. It may involve manually verifying something, or it may be entirely automated. In many cases it happens after a few corrective actions have been attempted, such as rolling back a deployment.

Restore

These are the steps necessary to get a system back online. It may be the old system that we're repairing, or it may be a new copy that we're preparing. It usually involves one or several automated steps, and in some events manual intervention is needed. It ends when the system is capable of serving production traffic.

Fail over

Once we have a live system capable of serving production traffic, we need to send traffic to it. It sounds trivial, but there are several factors that make it worth being a stage on its own:

• You usually want to do it gradually, to avoid crashing the new system

• It may not happen instantly (for example, DNS propagation)

• Sometimes this stage is triggered manually

• You need to verify that it happened

• You continue monitoring afterward

Disaster Recovery Strategies on AWS

The two obvious solutions to disaster recovery are:

• Backing up data to another region and re-creating the entire system

• Continuously running the system in two regions

Both work, but they're not the only ones. They're actually the two extremes of a spectrum:

Backup and Restore

This is the simplest strategy, and the playbook is:

• Before an event (and continuously):

  • Back up all your data to a separate AWS region, which we call the DR region

• When an event happens:

  • Restore the data stores from the backups

  • Re-create the infrastructure from scratch

  • Fail over to the new infrastructure

It's by far the cheapest, all you need to pay for are the backups and any other regional resources that you need to operate (e.g. KMS keys used to encrypt data). When a disaster happens, you restore from the backups and re-create everything.

I'm being purposefully broad when I say "re-create everything". I bet your infrastructure took you a long time to create. How fast can you re-create it? Can you even do it in hours or a few days, if you can't look at how you did it the first time? (Remember the original region is down).

The answer, of course, is Infrastructure as Code. It will let you launch a new stack of your infrastructure with little effort and little margin for error. That's why we (and by we I mean anyone who knows what they're doing with cloud infrastructure) insist so much on IaC.

As you're setting up your infrastructure as code, don't forget about supporting resources. For example, if your CI/CD Pipeline runs in a single AWS Region (e.g. you're using CodePipeline), you'll need to be ready to deploy it to the new region along with your production infrastructure. Other common supporting resources are values stored in Secrets Manager or SSM Parameter Store, KMS keys, VPC Endpoints, and CloudWatch Alarms configurations.

You can define all your infrastructure as code, but creating the new copy from your templates usually requires some manual actions. You need to document everything, so you're clear on what's the correct order for the different actions, what parameters to use, common errors and how to avoid or fix them, etc. If you have all of your infrastructure defined as code, this documentation won't be really large. However, it's still super important.

Finally, test everything. Don't just assume that it'll work, or you'll find out that it doesn't right in the middle of a disaster. Run periodic tests for your Disaster Recovery plan, keep the code and the documentation up to date, and keep yourself and your teams sharp.

Pilot Light

With Backup and Restore you need to create a lot of things from scratch, which takes time. Even if you cut down all the manual processes, you might spend several hours staring at your terminal or the CloudFormation console waiting for everything to create.

What's more, most of these resources aren't even that expensive! Things like an Auto Scaling Group are free (without counting the EC2 instances), an Elastic Load Balancer costs only $23/month, and VPC and subnets are free. The largest portion of your costs come from the actual capacity that you use: a large number of EC2 instances, DynamoDB tables with a high capacity, etc. But since most of them are scalable, you could keep all the scaffolding set up with capacity scaled to 0, and scale up in the event of a disaster, right?

That's the idea behind Pilot Light, and this is the basic playbook:

• Before an event (and continuously):

  • Continuously replicate all your data to a separate AWS region, which we call the DR region

  • Set up your infrastructure in the DR region, with capacity at 0

• When an event happens:

  • Scale up the infrastructure in the DR region

  • Fail over to the DR region

One of the things that takes the longest time to create is data stores from snapshots. For that reason, the prescriptive advice (though not a strict requirement) for Pilot Light is to keep data stores functioning, instead of just keeping the backups and restoring from them in a disaster. It is more expensive though.

Since scaling can be done automatically, the Restore stage is very easy to automate entirely when using Pilot Light. Also, since the scaling times are much shorter than creating everything from scratch, the impact of automating all manual operations will be much higher, and the resulting RTO much lower than with Backup and Restore.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Warm Standby

The problem with Pilot Light is that, before it scales, it cannot serve any traffic at all. It works just like the pilot light in a home heater: a small flame that doesn't produce any noticeable heat, but is used to light up the main burner much faster. It's a great strategy, and your users will appreciate that the service interruption is brief, in the order of minutes. But what if you need to serve at least those users nearly immediately?

Warm Standby uses the same idea as Pilot Light, but instead of remaining at 0 capacity, it keeps some capacity available. That way, if there is a disaster you can fail over immediately and start serving a subset of users, while the rest of them wait until your infrastructure in the DR region scales up to meet the entire production demand.

Here's the playbook:

• Before an event (and continuously):

  • Continuously replicate all your data to a separate AWS region, which we call the DR region

  • Set up your infrastructure in the DR region, with capacity at a percentage greater than 0

• When an event happens:

  • Reroute a portion of the traffic to the DR region

  • Scale up the infrastructure

  • Reroute the rest of the traffic

What portion of the traffic you reroute depends on how much capacity you maintain "hot" (i.e. available). This lets you do some interesting things, like setting up priorities where traffic for some critical services is rerouted and served immediately, or even for some premium users.

It also presents a challenge: How much infrastructure do you keep hot in your DR region? It could be a fixed number like 2 EC2 instances, or you could dynamically adjust this to 20% of the capacity of the primary region (just don't accidentally set it to 0 when the primary region fails!).

You'd think dynamically communicating to the DR region the current capacity or load of the primary region would be too problematic to bother with. But you should be doing it anyway! When a disaster occurs and you begin scaling up your Pilot Light or Warm Standby infrastructure, you don't want to go through all the hoops of scaling slowly from 0 or low capacity to medium, to high, to maximum. You'd rather go from wherever you are directly to 100% of the capacity you need, be it 30 EC2 instances, 4000 DynamoDB WCUs, or whatever service you're using. To do that, you need to know how much is 100%, or in other words, how much capacity the primary region was running on before it went down. Remember that once it's down you can't go check! To solve that, back up the capacity metrics to the DR region. And once you have them, it's trivial to dynamically adjust your warm standby's capacity.

You can pick any number or percentage that you want, and it's really a business decision, not a technical one. Just keep in mind that if you pick 0 you're actually using a Pilot Light strategy, and if you pick 100% it's a variation of Warm Standby called Hot Standby, where you don't need to wait until infrastructure scales before rerouting all the traffic.

An important aspect that Warm Standby introduces is the fact that all three strategies that we've seen so far are active/passive, meaning that one region (the active one) serves traffic, while the other region (the DR one, which is passive) doesn't receive any traffic. With Backup and Restore and with Pilot Light that should be obvious, since they're not able to serve any traffic. Warm Standby is able to serve some traffic, and Hot Standby is able to serve the entirety of the traffic. But even then, they don't get any traffic, and the DR region is passive.

The reason for this is that, if you allow your DR region to write data while you're using the primary region (i.e. while it isn't down), then you need to deal with distributed databases with multiple writers, which is much harder than a single writer and multiple readers. Some managed services handle this very well, but even then there are implications that might affect your application. For example, DynamoDB Global Tables handle writes in any region where the global table is set up, but they resolve conflicts with a last-writer-wins reconciliation strategy, where if two regions receive write operations for the same item at the same time (i.e. within the replication delay window), the one that was written last is the one that sticks. Not a bad solution, but you don't want to overcomplicate things if you don't have to.

Multi-site Active/Active

In active/passive configurations, only one region serves traffic. Active/active spreads the traffic across both regions in normal operation conditions (i.e. when there's no disaster). As mentioned in the previous paragraph, this introduces a few problems.

The main problem is the read/write pattern that you'll use. Distributed data stores with multiple write nodes can experience "contention", a term that means everything is slowed down because multiple nodes are trying to access the same data, and they need to wait for the others so they don't cause inconsistencies. Contention is one of the reasons why databases are hard.

Another problem is that you're effectively managing two identical but separate infrastructures. Suddenly it's not just a group of instances plus one of everything else (Load Balancer, VPC, etc), but two of everything.

You also need to duplicate any configuration resources, such as Lambda functions that perform automations, SSM documents, SNS topics that generate alerts, etc.

Finally, instead of using the same value for "region" in all your code and configurations, you need to use two values, and use the correct one in every case. That's more complexity, more work, more cognitive load, and more chances of mistakes or slip ups.

Overall, Multi-Site Active/Active is much harder to manage than Warm Standby, but the advantage is that losing a region feels like losing an AZ when you're running a Highly Available workload: You just lose a bit of capacity, maybe fail over a couple of things, but overall everything keeps running smoothly.

Tips for Effective Disaster Recovery on AWS

Decide on a Disaster Recovery Strategy

You can choose freely between any of the four strategies outlined on this article, or you can even choose not to do anything in the event of a disaster. There are no wrong answers, there's only tradeoffs.

To pick the best strategy for you:

• Calculate how much money you'd lose per minute of downtime

• If there are hits to your brand image, factor them in as well

• Estimate how often these outages are likely to occur

• Calculate how much each DR strategy would cost

• Determine your RTO for each DR strategy

• Plug everything into your calculator

• Make an informed decision

"I'd rather be offline for 24 hours once every year and lose $2.000 than increase my anual AWS expenses by $10.000 to reduce that downtime" is a perfectly valid and reasonable decision, but only if you've actually run the numbers and made it consciously.

Improve Your Detection

The longer you wait to declare an outage, the longer your users have to wait until the service is restored. On the other hand, a false positive (where you declare an outage when there isn't one) will cause you to route traffic away from a region that's working, and your users will suffer from an outage that isn't there.

Improving the granularity of your metrics will let you detect anomalies faster. Cross-referencing multiple metrics will reduce your false positives without increasing your detection time. Additionally, consider partial outages, how to differentiate them from total outages, and what the response should be.

Practice, Practice, Practice

As with any complex procedure, there's a high probability that something goes wrong. When would you rather find out about it, on regular business hours when you're relaxed and awake, or at 3 am with your boss on the phone yelling about production being down and the backups not working?

Disaster Recovery involves software and procedures, and as with any software or procedures, you need to test them both. Run periodic disaster recovery drills, just like fire drills but for the prod environment. As the Google SRE book says: "If you haven’t gamed out your response to potential incidents in advance, principled incident management can go out the window in real-life situations."

Recommended Tools and Resources for Disaster Recovery

One of the best things you can read on Disaster Recovery is the AWS whitepaper about Disaster Recovery. In fact, it's where I took all the images from.

Another fantastic read is the chapter about Managing incidents from the Site Reliability Engineering book (by Google). If you haven't read the whole book, you might want to do so, but chapters stand independently so you can read just this one.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

• Customers are stored with a Customer ID starting with c# (for example c#123) as the PK and SK.

• Products are stored with a Product ID starting with p# (for example p#123) as the PK and SK, and with an attribute of type number called 'stock', which contains the available stock.

• Orders are stored with an Order ID starting with o# (for example o#123) for the PK and the Product ID as the SK.

• When an item is purchased, we need to check that the Product is in stock, decrease the stock by 1 and create a new Order.

• Payment, shipping and any other concerns are magically handled by the power of "that's out of scope for this issue" and "it's left as an exercise for the reader".

There are more attributes in all entities, but let's ignore them.

We're going to use the following AWS services:

• DynamoDB: A NoSQL database that supports ACID transactions, just like any SQL-based database.

Before Implementing DynamoDB Transactions

We need to read the value of stock and update it atomically. Atomicity is a property of a set of operations, where that set of operations can't be divided: it's either applied in full, or not at all. If we just ran the GetItem and PutItem actions separately, we could have a case where two customers are buying the last item in stock for that product, our scalable backend processes both requests simultaneously, and the events go down like this:

1. Customer123 clicks Buy

2. Customer456 clicks Buy

3. Instance1 receives request from Customer123

4. Instance1 executes GetItem for Product111, receives a stock value of 1, continues with the purchase

5. Instance2 receives request from Customer456

6. Instance2 executes GetItem for Product111, receives a stock value of 1, continues with the purchase

7. Instance1 executes PutItem for Product111, sets stock to 0

8. Instance2 executes PutItem for Product111, sets stock to 0

9. Instance1 executes PutItem for Order0046

10. Instance1 receives a success, returns a success to the frontend.

11. Instance2 executes PutItem for Order0047

12. Instance2 receives a success, returns a success to the frontend.

The data doesn't look corrupted, right? Stock for Product111 is 0 (it could end up being -1, depends on how you write the code), both orders are created, you received the money for both orders (out of scope for this issue), and both customers are happily awaiting their product. You go to the warehouse to dispatch both products, and find that you only have one in stock. Where did things go wrong?

Steps to Implement DynamoDB Transactions

The problem is that steps 4 and 7 were executed separately, and Instance2 got to read the stock of Product111 (step 6) in between them, and made the decision to continue with the purchase based on a value that hadn't been updated yet, but should have. Steps 4 and 7 need to happen atomically, in a transaction.

Install the AWS SDK

First, install the packages from the AWS SDK V3 for JavaScript:

npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb

Update the Code to Use Transactions

This is the code in Node.js to run the steps as a transaction (you should add this to the code imaginary you already has for the service):

const { DynamoDBClient } = require('@aws-sdk/client-dynamodb');
const { DynamoDBDocumentClient } = require('@aws-sdk/lib-dynamodb');

const dynamoDBClient = new DynamoDBClient({ region: 'us-east-1' });
const dynamodb = DynamoDBDocumentClient.from(dynamoDBClient);

//The code imaginary you already has

//This is just some filler code to make this example valid. Imaginary you should already have this solved
const newOrderId = 'o#123' //Must be unique
const productId = 'p#111' //Comes in the request
const customerId = 'c#123' //Comes in the request

const transactItems = {
  TransactItems: [
    {
      ConditionCheck: {
        TableName: 'SimpleAwsEcommerce',
        Key: { id: productId },
        ConditionExpression: 'stock > :zero',
        ExpressionAttributeValues: {
          ':zero': 0
        }
      }
    },
    {
      Update: {
        TableName: 'SimpleAwsEcommerce',
        Key: { id: productId },
        UpdateExpression: 'SET stock = stock - :one',
        ExpressionAttributeValues: {
          ':one': 1
        }
      }
    },
    {
      Put: {
        TableName: 'SimpleAwsEcommerce',
        Item: {
          id: newOrderId,
          customerId: customerId,
          productId: productId
        }
      }
    }
  ]
};

const executeTransaction = async () => {
  try {
    const data = await dynamodb.transactWrite(transactItems);
    console.log('Transaction succeeded:', JSON.stringify(data, null, 2));
  } catch (error) {
    console.error('Transaction failed:', JSON.stringify(error, null, 2));
  }
};

executeTransaction();

//Rest of the code imaginary you already has

After Implementing DynamoDB Transactions

Here's how things may happen with these changes, if both customers click Buy at the same time:

1. Customer123 clicks Buy

2. Customer456 clicks Buy

3. Instance1 receives request from Customer123

4. Instance2 receives request from Customer456

5. Instance1 executes a transaction:

   1. ConditionCheck for Product111, stock is greater than 0 (actual value is 1)

   2. PutItem for Product111, set stock to 0

   3. PutItem for Order0046

   4. Transaction succeeds, it's committed.

6. Instance1 receives a success, returns a success to the frontend.

7. Instance2 executes a transaction:

   1. ConditionCheck for Product111, stock is not greater than 0 (actual value is 0)

   2. Transaction fails, it's aborted.

8. Instance2 receives an error, returns an error to the frontend.

Overview of DynamoDB

DynamoDB is so scalable because it's actually a distributed database, where you're presented with a single resource called Table, but behind the scenes there's multiple nodes that store the data and process queries. Data is partitioned using the Partition Key, which is part of the Primary Key (the other part is the Sort Key).

DynamoDB is highly available (meaning it can continue working if an Availability Zone goes down) because each partition is stored in 3 nodes, each in a separate Availability Zone. This is the "secret" behind DynamoDB's availability and durability. You don't need to know this to use DynamoDB effectively, but now that you do, you see that transactions in DynamoDB are actually distributed transactions.

How Transactions Work in DynamoDB

Two-Phase Commit

DynamoDB implements distributed transactions using Two-Phase Commit (2PC). This strategy is pretty simple: All nodes are requested to evaluate the transaction to determine whether they're capable of executing it, and only after all nodes report that they're able to successfully execute their part, the central controller sends the order to commit the transaction, and each node does the actual writing, affecting the actual data. For this reason, all operations done in a DynamoDB transaction consume twice as much capacity.

Itempotency

DynamoDB transactions are idempotent. They're identified by an attribute called ClientRequestToken, which the DynamoDB SDK includes automatically on any transactions. If you use the TransactReadItems API or TransactWriteItems API without the SDK, you'll need to include it to achieve transaction idempotency.

Isolation

Transaction isolation (the I in ACID) is achieved through optimistic concurrency control. This means that multiple DynamoDB transactions can be executed concurrently, but if DynamoDB detects a conflict, one of the transactions will be rolled back and the caller will need to retry the transaction.

Transactions on Multiple Tables

DynamoDB Transactions can span multiple tables, but they can't be performed on indexes. Also, propagation of the data to Global Secondary Indexes and DynamoDB Streams always happens after the transaction, and isn't part of it.

Pricing for DynamoDB Transactions

There is no direct cost for using transactions. However, all operations performed on DynamoDB as part of a transactions will consume double the amount of capacity units as they regularly would. Write and delete operations consume write capacity, and any condition expression consumes read capacity. This extra capacity is only consumed for the operations on the table, the read and write capacity consumed for updating secondary indexes and for DynamoDB Streams isn't affected. When working with DynamoDB On-Demand Mode, Request Units are doubled, just like Capacity Units.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

DynamoDB vs SQL databases

The whole point of this article and the others I've written about DynamoDB is that SQL databases shouldn't be your default. I've shown you that DynamoDB can handle an e-commerce store just fine, including ACID-compliant transactions. That's because for an e-commerce, and in fact for 95% of the applications we write, we can predict data access patterns. When we can do that, we can optimize the structure and relations of a NoSQL database like DynamoDB and have it perform much better than a relational database for those known and predicted access patterns.

The use case for SQL databases is unknown access patterns! And those come from either giving the user a lot of freedom (which might be a mistake, or might be a core feature of your application), or from doing analytics and ad-hoc queries. In those cases, definitely go for relational databases. Otherwise, see if you can solve it with a NoSQL database like DynamoDB. It'll be much cheaper, and it will scale much better. I'll make one concession though: If all your dev team knows is SQL databases, just go with that unless you have a really strong reason not to.

Using SQL in DynamoDB

This is gonna blow your mind: You can actually query DynamoDB using SQL! Or more specifically, a SQL-compatible language called PartiQL. Amazon developed PartiQL as an internal tool, and it was made generally available by AWS. It can be used on SQL databases, semi-structured data, or NoSQL databases, so long as the engine supports it.

With PartiQL you could theoretically change your Postgres database for a DynamoDB database without rewriting any queries. In reality, you need to consider all of these points:

• Why are you even changing? It's not going to be easy.

• How are you going to migrate all the data?

• You need to make sure no queries are triggering a Scan in DynamoDB, because we know those are slow and very expensive. You can use an IAM policy to deny full-table Scans.

• Again, why are you even changing?

I'm not saying there isn't a good reason to change, but I'm going to assume it's not worth the effort, and you'll have to prove me otherwise. Remember that replicating the data somewhere else for a different access pattern is a perfectly valid strategy (in fact, that's exactly how DynamoDB GSIs work). We'll discuss it further in a future issue.

Are there any limitations to using transactions in DynamoDB?

Yes, there are some limitations to using transactions in DynamoDB. Transactions are limited to a maximum of 100 unique items and the total data size within a transaction cannot exceed 4 MB. Additionally, transactions cannot operate on tables with global secondary indexes that have projected attributes.

Best Practices

Operational Excellence

• Monitor transaction latencies: Monitor latencies of your DynamoDB transactions to identify performance bottlenecks and address them. Use CloudWatch metrics and AWS X-Ray to collect and analyze performance data.

• Error handling and retries: Handle errors and implement exponential backoff with jitter for retries in case of transaction conflicts.

Security

• Fine-grained access control: Assign an IAM Role to your backend with an IAM Policy that only allows the specific actions that it needs to perform, only on the specific tables that it needs to access. You can even do this per record and per attribute. This is least privilege.

Reliability

• Consider a Global Table: You can make your DynamoDB table multi-region using a Global Table. Making the rest of your app multi-region is more complicated than that, but at least the DynamoDB part is easy.

Performance Efficiency

• Optimize provisioned throughput: If you're using Provisioned Mode, you'll need to set your Read and Write Capacity Units appropriately. You can also set them to auto-scale, but it's not instantaneous. Remember the article on using SQS to throttle writes.

Cost Optimization

• Optimize transaction sizes: Minimize the number of items and attributes involved in a transaction to reduce consumed read and write capacity units. Remember that transactions consume twice as much capacity, so optimizing the operations in a transaction is doubly important.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

Imagine this scenario: You have some data that's absolutely critical to your business. If you lose it, it's a disaster! How do you recover?

Data Loss Scenarios

First, we need to define what we mean when we say "lose it". How do you lose data? Let's consider some scenarios, and what we can do to in each case.

Data Loss Because of Hardware Failure

As I'm sure you know, computer hardware is sensitive equipment, which will inevitably fail at some point. When working with AWS we don't really see or manage the hardware, but we're still vulnerable to hardware failing. That's why AWS services publicly advertise their durability: EBS has 99.9% or 99.999% depending on volume type, while S3 has 99.999999999% (referred to as 11 nines, or 11 9s). 99.999999999% (11 nines) durability means that if you store 10 million objects, then you expect to lose an object of your data every 10,000 years.

How to prevent it

S3 should be more than enough to protect from bit rot or simultaneous hardware failures. If you're not storing your critical data in S3, start at least backing it up there. You can create Snapshots from EBS volumes or RDS instances, which are stored in S3.

Data Loss Because of Human Error

This includes scenarios where you or anyone on your team (with legitimate access and good intentions) accidentally deletes or overwrites data. In S3, it can be deleting an object or an entire bucket. In EBS, EFS or anything mounted in the file system, it can be a typo when running a command like rm -rf. In a database, it's more often than not a query ran with the wrong parameters, such as a SQL UPDATE with no WHERE clause.

Automated processes are also included in this scenario, since the reason they might delete or overwrite data that they shouldn't have touched is always due to human error when programming or configuring them.

How to prevent it

The first step is to understand that anyone can make a mistake, no matter how skilled or careful. Training and clearly defined procedures will reduce the probability of mistakes, but they'll never take it to 0. Limiting accesses and implementing guardrails and additional confirmations will further reduce the probability.

Overall, the best way to protect from this is to have backups of the data that can't be overwritten or deleted through the same means. For example, you can copy database snapshots to another AWS account.

Data Loss Because of Hacks or Ransomware

In this case, you're dealing with a malicious actor intentionally trying to delete the data, or make it inaccessible to you. The most common scenarios are Ransomware attacks, where an attacker either steals or encrypts the data, and asks you for money in exchange for granting you access to it.

Attackers gain the ability to affect your data in AWS through credentials. This can be your own username and password stolen, the IAM role of an EC2 instance that the attacker gained access to, or any other way that they can gain AWS credentials to your account.

How to prevent it

Basic security measures such as security best practices for AWS accounts, minimum privileges, and application security go a long way. Requiring Multi-Factor Authentication for certain AWS operations, such as deleting objects in S3, is another good measure.

What can happen is that an attacker gains some form of access, often not enough to compromise the entire AWS account or access your data, and then performs lateral movements and privilege escalations to gradually gain more access. A really simple example would be an EC2 instance with no access to S3 but an IAM Policy that grants permissions iam:*. An attacker with access to that instance can't immediately encrypt an S3 bucket, but they can use that instance's credentials to create a new IAM User for themselves, which has access to S3.

A way to protect against that is to store backups of data where they can't be tampered with from the same place where the data is. A good example, which I'll show you how to configure, is to set up a separate AWS account (let's call it Account B) and replicate there all objects in an S3 bucket. So long as there's no way to delete those objects in Account B from Account A, there's no path for an attacker with access to Account A to delete or encrypt the data in Account B. This doesn't completely eliminate the risk! But it makes it much less likely to occur, since an attacker would need to succeed at two separate attacks, one to gain access to Account A and one to Account B. This, coupled with ways to detect failed sign in attempts to AWS, significantly improves your security.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Disaster Recovery Metrics: RPO and RTO

Before we move on to the solution, there's two things I want to discuss briefly, which will determine how often you perform your backups and what backup strategies and/or technologies you can use. They're two of the most common Disaster Recovery metrics:

Recovery Point Objective (RPO)

This is a measure of how much time can pass from when data is written to when it's backed up. It's measured in time units, usually in hours or minutes. Any data that was written less than "RPO" ago isn't guaranteed to be backed up, and in the event of a disaster it wouldn't be found in the backups, and would be lost.

For example, an RPO of 12 hours means any data that was written less than 12 hours ago isn't guaranteed to be backed up. A typical way to implement an RPO of 12 hours is to create backups twice a day, for example at 00:00 and 12:00. Data isn't guaranteed to not be backed up either, for example a disaster occurs at 13:00 the only data lost would be from the previous hour, which is the time since the last backup.

The reason RPO isn't exactly equivalent to time between backups is because backups can also be implemented in a continuous or nearly continuous way. For our backup every 12 hours example we're assuming the backup is instantaneous, which isn't exactly true but it's a good approximation. If we run backups every 1 minute, then the 30 seconds that the backup process might last isn't a number we can ignore, and our RPO would be 1 minute and 30 seconds.

The duration of the backup process is sometimes called replication delay. For example, Aurora has a replication delay of 1 minute between the primary instance and its replicas. That gives a Disaster Recovery strategy of using an Aurora replica a 1-minute RPO, since the replication process is started nearly immediately when data is changed.

Different data can have different RPOs. For example, data stored in S3 can have an RPO of 1 hour, and data on RDS an RPO of 6 hours. That's perfectly normal, and you should consider how bad it would be to lose all new data from the last X time to decide whether you're good with your numbers or need to improve them. RPO can be heavily constrained by the technology used to store the data and the technologies and techniques used to back it up. For example, an RPO of 1 hour is normal for S3 because the easiest backup method to set up for S3 is Cross-Region Replication, a feature already built into S3 with an RPO of 1 hour (15 minutes if you enable Replication Time Control).

Recovery Time Objective (RTO)

This is the target time between when you detect that a failure is happening and when you have the backup live and serving traffic at the same quality of service as if there was no failure. It's measured in time units, usually minutes or hours.

For example, if you're backing up your RDS database with RDS Snapshots, your RTO is going to be more or less the time it takes you to create a new RDS instance from the snapshot (usually between 30 minutes and 2 hours, depending on the size of the snapshot).

More accurately, your RTO would be the time between when you detect a failure in the original RDS instance and when the new RDS instance created from the snapshot is serving traffic. If you've automated this process, 99% of the recovery time is going to be creating the RDS instance. If you haven't, you need to take into account the time it takes you to:

1. Receive an alert

2. View that alert

3. Log in to the system

4. Find the correct snapshot

5. Figure out the correct configurations for the new RDS instance

6. Launch the creation of a new RDS instance

7. Switch over traffic to the new RDS instance

Every part of that which you can automate will reduce your RTO, and also reduce the chance that you make a mistake and, for example, restore the incorrect snapshot, or create the new RDS instance in the wrong VPC. Automate whatever you can (you can automate all of that). Start with the longest sentences, I wrote them like that on purpose.

Disaster Recovery in AWS

In AWS jargon, Disaster Recovery means being able to get the entire system back online in the case of an AWS Region failing. For that, you'll need to have the data available in that other region, as well as any additional resources required to access the data, such as the KMS key used to encrypt it (remember that KMS keys are regional by default, and you can create multi-region keys).

Getting the entire system back online also requires you to stand up compute capacity (be it EC2 instances, an ECS on Fargate cluster, Lambda functions, etc), make the data accessible (e.g. launch an RDS instance from the copied RDS snapshot), and switch over traffic. It's a complex process, there are different strategies, and there are multiple things to take into account depending on your RTO and RPO.

The next post is going to be about Disaster Recovery strategies, and being prepared to deploy the entire system in another AWS region. The first step towards that is to have the data accessible, so let's focus on that.

How to Configure S3 Replication Across Different AWS Accounts

Let's view a solution to back up data in S3 to another S3 bucket in a different AWS account. To better protect from different disaster scenarios, you should make sure access to this other AWS account is very restricted.

Step 0: Preparation

1. Log in to an AWS account, let's call it Account A.

2. Open the S3 console

3. Click Create bucket

4. Enter a name for the source bucket (must be unique across all AWS)

5. Scroll down to Bucket Versioning and select Enable

6. Click Create

7. Copy or write down the Account ID of Account A (you'll need it later)

8. Log in to a different AWS account, let's call it Account B

9. Open the S3 console

10. Click Create bucket

11. Enter a name for the destination bucket (must be unique across all AWS). Write down the name.

12. Scroll down to Bucket Versioning and select Enable

13. Click Create

14. Copy or write down the Account ID of Account B (you'll need it later)

Step 1: Enable Replication in the Source Bucket

1. Log back in to Account A and go to S3

2. Click on the source bucket (the one you created on Step 0)

3. Click the Management tab

4. Scroll down to Replication rules and click Create replication rule

5. Under Replication rule name enter a name for the rule, such as cross-account replication

6. Under Choose a rule scope, select Apply to all objects in the bucket

7. In the Destination section, under Destination select Specify a bucket in another account

8. Under Account ID enter the Account ID of Account B (where the destination bucket is)

9. Under Bucket name, enter the name of the destination bucket

10. Check Change object ownership to destination bucket owner

11. In the IAM role section, under IAM role open the dropdown and select Create new role

12. Click Save

13. Click Submit (we don't have any existing objects, so what we choose here doesn't really matter)

14. In the Replication configuration settings section, under IAM role, copy the name of the IAM role that was created automatically

Step 2: Update the Policy on the Destination Bucket

1. Log in to Account B and go to S3

2. Click on the destination bucket (the one you created on Step 0)

3. Click the Permissions tab

4. Next to Bucket policy, click Edit

5. In the following policy, replace ID_OF_ACCOUNT_A with the ID of the account where the source bucket is, NAME_OF_THE_IAM_ROLE with the last value you copied in Step 1, and NAME_OF_THE_DESTINATION_BUCKET with the name of the destination bucket. Then click Save.

{
   "Version":"2012-10-17",
   "Id":"",
   "Statement":[
      {
         "Sid":"Set-permissions-for-objects",
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::ID_OF_ACCOUNT_A:role/service-role/NAME_OF_THE_IAM_ROLE"
         },
         "Action":["s3:ReplicateObject", "s3:ReplicateDelete"],
         "Resource":"arn:aws:s3:::NAME_OF_THE_DESTINATION_BUCKET/*"
      },
      {
         "Sid":"Set permissions on bucket",
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::ID_OF_ACCOUNT_A:role/service-role/NAME_OF_THE_IAM_ROLE"
         },
         "Action":["s3:List*", "s3:GetBucketVersioning", "s3:PutBucketVersioning"],
         "Resource":"arn:aws:s3:::NAME_OF_THE_DESTINATION_BUCKET"
      }
   ]
}

Step 3: Upload an Object to the Source Bucket

1. Log back in to Account A and go to S3

2. Click on the source bucket

3. Click Upload

4. Click Add files, select one or more files, and click Open

5. Click Upload

6. Verify that the file is uploaded to the Source bucket

Step 4: Upload an Object to the Source Bucket

1. Log back in to Account B and go to S3

2. Click on the destination bucket

3. Verify that the same file you uploaded to the source bucket is present on the destination bucket

Recommended Tools and Resources about Data Replication

When data is replicated across several disks, data loss happens when one of those disks fails, and in the time while a new copy is being created to replace the one that just failed, another disk also fails. The probability of losing data clearly depends on how often a disk fails (called Mean Time Between Failures, or MTBF) and how long it takes to recreate that copy (called Mean Time To Recovery or MTTR). But if I gave you those numbers, would you know how to calculate the probability of data loss? I didn't, until I read this article!

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

Imagine this scenario: You're careful with security, and you set up Multi-Factor Authentication for your AWS IAM or IAM Identity Center user. At one point, a malicious agent of evilness figures out your password, either via phishing, keylogging, or any other technique. The only thing standing between them and $50k in bitcoin mined on your AWS account (and the $500k AWS bill paid with your credit card) is your MFA device. They don't have access it (yet!), so you're safe (for now!).

The correct response is obvious: Change your password, so they'll be back to square one. The problem with this is that most of the times you don't know that a password has become known until someone uses it. This is the reason why we should rotate passwords regularly!

Let me propose an extra layer of security: Get notified every time a login attempt fails because of a failed MFA check.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Understanding CloudTrail event logs

AWS CloudTrail is a service that logs all requests performed against AWS in your account. This includes all actions and requests (even unauthorized ones) done through the Console, CLI, AWS SDKs, and APIs.

By default (already enabled when you create your account) and for free, CloudTrail offers you a viewable, searchable, downloadable, and immutable record of all events that happened in the past 90 days, called Event History.

You can also create Trails, which let you export to S3 or CloudWatch Logs all or a subset of CloudTrail events. This way, you can use other AWS services like Athena and OpenSearch, or external tools like ElasticSearch, to analyze CloudTrail events. Event history is per region, while Trails can be created for all regions in an account, and even for all accounts in an Organization.

Events are JSON objects, which contain all the information of what was attempted, when, by whom, some other details on the request, what was the result, and sometimes some other details on the response, such as the reason.

For example, this is what a ConsoleLogin event that fails looks like (I hid a few details replacing them with HIDDEN):

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "HIDDEN",
        "accountId": "HIDDEN",
        "accessKeyId": "HIDDEN",
        "userName": "HIDDEN"
    },
    "eventTime": "2023-08-24T21:07:08Z",
    "eventSource": "signin.amazonaws.com",
    "eventName": "ConsoleLogin",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "HIDDEN",
    "userAgent": "HIDDEN",
    "errorMessage": "Failed authentication",
    "requestParameters": null,
    "responseElements": {
        "ConsoleLogin": "Failure"
    },
    "additionalEventData": {
        "LoginTo": "https://console.aws.amazon.com/console/home?HIDDEN",
        "MobileVersion": "No",
        "MFAUsed": "Yes"
    },
    "eventID": "HIDDEN",
    "readOnly": false,
    "eventType": "AwsConsoleSignIn",
    "managementEvent": true,
    "recipientAccountId": "HIDDEN",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "signin.aws.amazon.com"
    }
}

How are IAM and IAM Identity Center events logged in CloudTrail?

Naturally, CloudTrail logs events when an IAM or IAM IC user logs in to AWS. However, it's not just one event. Let me show you what happens behind the scenes with IAM IC and IAM.

CloudTrail Events for IAM Identity Center users logging in

These are the events that CloudTrail logs when an IAM or IAM IC user logs in:

• CredentialChallenge: AWS requested some form of credential, such as password or MFA device. Each of these is followed by UserAuthentication and one CredentialVerification event, and this sequence of three is repeated until either all necessary credentials are provided, or CredentialVerification fails.

• UserAuthentication: AWS receives the requested credentials.

• CredentialVerification: AWS checks whether the received credentials are valid. If they are, this event contains: "serviceEventDetails":{ "CredentialChallenge":"Success" }, and the process continues either by requesting the next credential or by authenticating. If the credentials received are invalid, this event contains "serviceEventDetails":{ "CredentialChallenge":"Failure" } and the process stops. The type of credential requested can be found in "additionalEventData":{ "CredentialType":"PASSWORD" }, the value of which can be PASSWORD for the regular password, TOTP for MFA devices that produce temporary codes, WEBAUTHN for web apps using WebAuthn, EXTERNAL_IDP for external identity providers, or RESYNC_TOTP to re-synchronize TOTP devices.

• Authenticate: Once CredentialVerification succeeds and no more credentials are required, this event is logged. This means the user successfully authenticated to IAM Identity Center. If you're authenticating with an external Identity Provider such as Google Workspaces, Microsoft AD or Okta, this is the only event you'll see.

• Federate: This event means the IAM IC user assumed an IAM role in an AWS account.

• ConsoleLogin: This event means the IAM IC user logged in to the AWS Console using the assumed IAM role.

The first four events will be logged on the AWS account and region where IAM IC is configured. The last two are logged in the AWS account where the user signs in, in the default region for that user (i.e. the one that's selected when the user signs in).

CloudTrail Events for IAM users logging in

IAM is much simpler. The only event is ConsoleLogin, and this is how you can identify what happened:

• If the user signed in successfully, the event contains "responseElements": { "ConsoleLogin": "Success" }

• If sign in failed, the event contains "responseElements": { "ConsoleLogin": "Failure" } and can contain "errorMessage": "Failed authentication"

• If the user used MFA (regardless of success or failure), the event contains "additionalEventData": { "MFAUsed": "Yes" }

• If the user is the root user, the event contains "userIdentity": { "type": "Root" }

All of these events are always logged in the us-east-1 region.

Identifying when signing in fails due to MFA

The last section was a bit detailed, but it contains all the information we need to identify when a login attempt fails the MFA check!

For IAM IC, we need to find CredentialVerification events which contain either "additionalEventData":{ "CredentialType":"TOTP" } or "additionalEventData":{ "CredentialType":"WEBAUTHN" }, and contain "serviceEventDetails":{ "CredentialChallenge":"Failure" }.

For IAM, we need to find ConsoleLogin events which contain "responseElements": { "ConsoleLogin": "Failure" } and "additionalEventData": { "MFAUsed": "Yes" }.

You can do this manually in the CloudTrail event history, but if we want to automate it and get notifications, we need to send it to CloudWatch Logs.

Sending CloudTrail events to CloudWatch Logs

1. Sign in to the AWS Console. For IAM IC, sign in to the root account of the Organization

2. Go to CloudTrail

3. Click Create trail

4. For Trail name, enter management-events

5. If you're configuring this for an organization, check Enable for all accounts in my organization

6. Select Create new S3 bucket and under Trail log bucket and folder enter a name, or leave it as default

7. Under AWS KMS alias, enter a name for a KMS key to encrypt the logs

8. Under CloudWatch Logs, check Enabled

9. Under Log group name, enter a name for the CloudWatch Logs log group

10. Under Role name, enter a name for the IAM Role that'll let CloudTrail put logs to CloudWatch Logs

11. Click Next

12. Leave these options as default and click Next again

13. Click Create trail

14. Wait until the Status column changes to Logging (green)

Configuring Alerts and Notifications for CloudWatch Logs

First, let's create a filter to view the failed logins in CloudWatch Logs. Follow the steps, and on Step 6 choose the right filter depending on whether you're using IAM or IAM IC.

Filtering failed login attempts due to MFA

1. Open the CloudWatch console

2. In the panel on the left, under Logs, click Log groups.

3. Click on the name of the log group that you created for the trail

4. Click the Metrics filters tab, and click the button Create metric filter

5. Under Filter pattern, enter the pattern that corresponds to IAM or IAM IC, depending on which you're using:

   1. For IAM: { ($.eventName = ConsoleLogin) && ($.additionalEventData.MFAUsed = "Yes") && ($.responseElements.ConsoleLogin = "Failure") }

   2. For IAM IC: { ($.eventName = CredentialVerification) && (($.additionalEventData.CredentialType = "TOTP") || ($.additionalEventData.CredentialType = "WEBAUTHN")) && ($.serviceEventDetails.CredentialChallenge = "Failure")}

6. Click Next

7. For Filter name, enter SignInFailedMFA

8. Under Metric namespace, enter CloudTrailMetrics

9. For Metric name, enter SigninMFAFailureCount

10. For Metric value, enter 1.

11. Click Next

12. Click Create metric filter

Configuring alerts for failed login attempts due to MFA

1. On the Metric filters tab, find the metric filter you just created, select it and click Create alarm

2. Under Whenever SigninMFAFailureCount is..., select Greater/Equal

3. Under than…, enter 1

4. Click Next

5. Under Send a notification to the following SNS topic, select Create new topic

6. Under Create a new topic…, enter SignInFailedMFAAlarm

7. Under Email endpoints that will receive the notification…, enter your email address

8. Click the Create topic button that's right below where you just entered your email address

9. Click Next

10. Under Alarm name, enter SignInFailedMFAAlarm

11. Click Next

12. Click Create alarm

13. Open your email inbox, open the email titled AWS Notification - Subscription Confirmation, and click Confirm subscription

Testing Alerts When Signing in to AWS Fails Due to Incorrect MFA Code

To test it, you'll need to attempt to sign in, and enter the incorrect MFA code. There's going to be a delay of a few minutes between when the event actually occurs and when you receive the notification. CloudTrail usually takes an average of 5 minutes to export the event logs to CloudWatch Logs, and the alarm could take a couple more minutes to fire.

You can also monitor the alarm in CloudWatch Alarms:

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

Back in the Paleolithic, which for software means 30 years ago, we had HTML, CSS and JavaScript, and we wrote all the structure of the website in HTML. When we wanted to create dynamic content for our website, we added PHP or Java code in the middle of that HTML, and we would run that code on the server and it would output the HTML that we wanted to put there. That process of producing the final HTML for the website by running code is called rendering, and back then it happened Server-Side.

Then came web components and frameworks like Angular, React and Vue, and they changed the paradigm a bit. Now we didn't write plain HTML with snippets of PHP or Java, but instead we wrote JavaScript code that would then output the entire HTML code. Since it was JavaScript, we could run it on the browser, at the user's computer. That's called Client-Side Rendering.

A huge benefit of Client-Side Rendering was that we were using the user's CPU. That was awesome for our pockets, we didn't have to pay for that compute capacity. However, it meant the user had to wait however long their computer took to render that website. Mind you, it wasn't considered a big problem back then. Remember that back in those days users' expectations were pretty different: it was considered normal to wait 3 or 5 seconds for a website to load. Nowadays, more than 2 seconds seems like an eternity.

As compute capacity got cheaper, someone had the idea of bringing the rendering back to the server. The idea was to use our significantly more powerful servers to render the website much faster than what the user computer could achieve, reducing load time significantly. Sure, we were paying extra, but overall the improved user experience was worth the extra money. Cloud computing played a big part there as well: Not only was infrastructure cheaper per CPU-hour, but it also required less engineering hours to create and maintain.

We didn't bring back the old languages though! Instead, we started running that same React code in our servers, as if it were the user's browser. A big reason for this is that, while all of this back and forth was happening, frontends got increasingly more complex. That led to developers being classified as frontend or backend, and while there are full stack devs, most are just strong on one side and really weak on the other one (I'm one of those cases!). So, the folks who knew how to code UIs didn't know Java or PHP (not even backend devs like those nowadays), and the folks who did know those languages didn't know how to code UIs. The solution? Let's create a framework like Next.js that runs that same React code in our servers. That way, we get the best of both worlds: Frontend frameworks with Server-Side Rendering!

That's how we went from Server-Side Rendering, to Client-Side Rendering, and back to Server-Side Rendering. I've said it often, technology is cyclic.

How does Amplify help

AWS Amplify is a set of tools that help frontend devs use AWS infrastructure and even build backends, without knowing a lot about infrastructure or backend. There's two parts to it: Amplify Hosting is a managed service that provides hosting and CI/CD for serverless apps. Amplify Studio is a visual development environment that lets you build a UI and a backend as with a no-code tool, with reusable components. We're going to use Amplify Hosting, but you might be interested in checking out Amplify Studio.

Why not an EC2 instance?

We know that AWS Amplify is going to be using EC2 instances behind the scenes, right? So, since we really know our way around AWS (I mean, you're reading a newsletter about AWS!), why shouldn't we just use EC2 instances for this instead of relying on a managed service? You could! It's a bit of our classic buy vs build decision, right? And you know by now that I always recommend you default to managed services and only build if there's a good reason to do so. I'm tempted to recommend the same here, but Amplify takes the managed in managed service and cranks it up to 11, and the price reflects that.

Amplify works as a self-service platform that people with little to no knowledge of cloud infrastructure can use to develop and deploy their applications. This is like an extremely managed big service: You're no longer solving just one part of the problem, but the entire problem of hosting an app in the cloud. It does get pretty expensive, so you probably want to consider EC2, or an ECS cluster on Fargate (more expensive than plain EC2, but easier). Let's run some numbers, so you can see how much it can really cost you.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

AWS Amplify Pricing

Here's how Amplify charges you:

• Build and deploy: $0.01 per minute

• Data storage: $0.023 per GB per month

• Data transfer out: $0.15 per GB served

• Requests for SSR: $0.30 per 1 million requests + $0.20 per GB-hour

For example, say you're a startup with the following assumptions:

• 10,000 daily active users

• 5 devs, each doing 2 commits a day

• Average build time is 3 minutes

• The team works Monday to Friday (20 days/month) <-- this is the least realistic assumption for startups

• The app is 25 MB

• Average page size is 1.5 MB

Here are our calculations:

Total build time per month = devs * commits/day * days/month * avg. build time = 5 * 2 * 20 * 3 = 600 build minutes per month. 600 * $0.01 = $6

Monthly GB served = daily active users * average page size * days/month = 10,000 * (1.5/1024) * 30 = 439.45 GB. 439.45 * $0.15 = $65.92

Monthly GB storage = app size * builds/month = (25/1024)(5*2*20) = 4.88 GB. 4.88 * $0.023 = $0.11

Total charges = $6 + $65.92 + $0.11 = $72.03/month

Ok, that wasn't so bad, right? Well, it's dominated by Monthly GB served, so let's run those numbers with CloudFront:

• For the most expensive regions: 439.45 * $0.12 = $52.73

• For the least expensive regions: 439.45 * $0.085 = $37,35

Amplify is 25% to 75% more expensive! Is that a lot? It depends. 75% more on $0,10 is less than the electricity I spent typing this sentence (shame on me! especially for adding this parenthesis to make the sentence long enough for that claim to be true). 75% more on $1.000 is $750, I'll set up CloudFront for you for that money!

AWS Amplify for the backend

Amplify also lets you host a backend, which it runs in Lambda functions. You don't have a lot of control over it, but it works well for its intended audience: People who wouldn't know what to do if they had a lot of control over their Lambda functions. Amplify also lets you consume other AWS services easily, through declarative and easy-to-use libraries. That way, you can consume Cognito or S3 from the frontend without knowing a lot about Cognito or S3. Here's the complete list of libraries for Amplify, and you can check the Readme of the JavaScript one as an example of its features.

Scenario

You work 8 hours a day as an engineer, and you want to launch a startup in your free time. You know you want good practices, but you don't have the time to set everything up manually. You want to start with the website, built with React and Next.js. You want Server-Side Rendering, and obviously a CI/CD Pipeline. You want everything to be serverless so you pay as little as possible while you get the hang of running a startup, getting users and all of that (which is actually the most difficult part).

Solution

Host the application in AWS Amplify, which handles hosting and CI/CD. As you build the backend, either use Amplify to create your Lambda functions, or go with something more traditional like Serverless or SAM, depending on your expertise.

Step-by-step Instructions

Step 0: Setup

1. Download and install Node.js from the official website. Alternatively, install nvm and your favorite version of Node.js

2. Install npm if it didn't come with Node.js.

3. Install yarn: npm install --global yarn

4. Install git

Step 1: Create a Next.js app

1. Open a terminal

2. Run the following command: yarn create next-app

3. Follow the prompts:

   What is your project named? simpleaws-app

   Would you like to use TypeScript? No

   Would you like to use ESLint? Yes

   Would you like to use Tailwind CSS? No

   Would you like to use src/ directory? Yes

   Would you like to use App Router? (recommended) Yes

   Would you like to customize the default import alias? No

4. Change directories to the app's directory: cd simpleaws-app

5. Start the app locally: yarn dev

6. Open your browser, go to http://localhost:3000 and check that the page loads correctly.

Step 2: Create a git repo for the app

1. Go to https://github.com/new and create a new repository

2. Init the repo locally: git init

3. Add the GitHub repository as an origin (replace YOUR_USERNAME and PROJECT_NAME with your values): git remote add origin git@github.com\:username/project-name.git

4. Add the files, commit and push:
   git add .

   git commit -m 'initial commit'

   git push origin main

Step 3: Create an Amplify project

1. Go to the Amplify console

2. Scroll down to the Get started section, and under Amplify Hosting, click Get started

3. Select GitHub and click Continue

4. Click Authorize AWS Amplify (us-east-1) (the green button)

5. Select your user or organization and click Continue

6. Select Only select repositories, click the dropdown Select repositories and click on your repo. Click Install & Authorize

7. Authenticate to GitHub with your security key, or click Use your password (it's in a really small font) and enter your password.

8. Click the dropdown under Recently updated repositories and select your repo. Leave branch as main, and click Next

9. Verify the Build and test settings (they should be fine, they were created by Next.js when you initialized the project)

10. Check Allow AWS Amplify to automatically deploy all files hosted in your project root directory

11. Click Next

12. Click Save and deploy

13. Wait until Provision, Build and Deploy are done

Step 4: Test the app

1. Click on the link under the window icon with the Amazon arrow. Verify that the website loads correctly.

Step 5: Delete the app

1. On the top right corner click Actions

2. Click Delete app

3. Enter "delete"

4. Click Delete

Explanation

Step 0: Setup

Just installing some tools and dependencies.

Step 1: Create a Next.js app

Next.js has this awesome project initializer, that creates all the basic scaffolding you need. It is a bit opinionated, but I've never met anyone who doesn't like it (if you don't like it, let me know, you'll be the first!). Besides, it fits the scenario: You just want to get things done.

Step 2: Create a git repo for the app

We need the app in a git repo so Amplify can track changes to branches and pull the code from there. You can use GitHub, GitLab, Bitbucket or CodeCommit. You can also not set up a git repo and upload the code manually (or reference an S3 bucket), but in that case Amplify can't do the CI/CD for you.

Step 3: Create an Amplify project

The only odd thing here would be the build steps. Amplify auto-detects them from your package.json file, and creates the configuration file that you saw in that step. Of course, you can edit it. I'd recommend you keep it in line with your package.json file though.

Step 4: Test the app

After everything is deployed, Amplify will be serving your app in a URL that looks something like branch-name.d1m7bkiki6tdw1.amplifyapp.com. You can set up a custom domain through the Amplify console. Don't just go to Route 53 and point a domain to your Amplify URL, that'll fail because of the SSL certificates.

Step 5: Delete the app

Let's not pretend like we always remember to delete the things we deploy. This is a friendly reminder to delete the app!

Best Practices

Operational Excellence

• Set up a branch for each environment: Create a dev env where Amplify deploys from the dev branch, and a prod env where Amplify deploys from the main branch. Only commit to those branches through pull requests. That way, merging a PR means a release in that environment.

• Monitor Performance: Amplify has a monitoring service that allows you to view logs, build events, and other metrics.

Security

• Use Amplify's Built-in Authentication: Amplify integrates with Cognito for user authentication. This lets you use a Cognito user pool really easily.

Reliability

• Data Backup and Versioning: If you're using Amplify DataStore, regularly backup your data.

Performance Efficiency

• API Caching: If you're using GraphQL with AppSync, enable caching to improve API response times and reduce the load on your backend.

Cost Optimization

• Consider not using Amplify: I don't want to position myself for or against Amplify, because the decision depends on your scenario and your problems (remember there is no single best solution!). Just consider the costs, the tradeoffs, and analyze what's the best solution for you. It may be Amplify, it may be doing things manually! Overall, remember that the cost of maintaining software is much higher than the cost of building the initial version.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

As you probably know, DynamoDB is a NoSQL database. It's a managed, serverless service, meaning you just create a Table (that's the equivalent of a Database in Postgres), and AWS manages the underlying nodes. It's highly available, meaning nodes are distributed across 3 AZs, so loss of an AZ doesn't bring down the service. The nodes aren't like an RDS Failover Replica though, instead data is partitioned (that's why Dynamo has a Partition Key!) and split across nodes, plus replicated on other nodes for availability and resilience. That means DynamoDB can scale horizontally!

There are two modes for DynamoDB, which affect how it scales and how you're billed:

DynamoDB Provisioned Mode

You define some capacity, and DynamoDB provisions that capacity for you. This is pretty similar to provisioning an Auto Scaling Group of EC2 instances, but imagine the size of the instance is fixed, and it's one group for reads and another one for writes. Here's how that capacity translates into actual read and write operations.

Capacity in Provisioned Mode

Capacity is provisioned separately for reads and writes, and it's measured in Capacity Units.

1 Read Capacity Unit (RCU) is equivalent to 1 strongly consistent read of up to 4 KB, per second. Eventually consistent reads consume half that capacity. Reads over 4 KB consume 1 RCU (1/2 for eventually consistent) per 4 KB, rounded up. That means if you have 5 RCUs, you can perform 10 eventually consistent reads every second, or 2 strongly consistent reads for 7 KB of data each (remember it's rounded up) plus 1 strongly consistent read for 1 KB of data (again, it's rounded up).

Write Capacity Units (WCU) work the same, but for writes. 1 WCU = 1 write per second, of up to 1 KB. So, with 5 WCUs, you can perform 1 write operation per second of 4.5 KB, or 5 writes of less than 1 KB.

Remember that all operations inside a transaction consume twice the capacity, because DynamoDB uses two-phase commit for transactions. Every node has to simulate the operation and then actually perform it, so it's twice the work.

Also remember that local secondary indexes (LSIs) make each write consume additional capacity: 1 extra write operation for puts or deletes, 2 extra operations per update, with actual WCUs depending on how much data is written on the index (not the base table). Reads on an LSI that query for attributes that aren't projected on the LSI also consume additional read capacity: 1 additional read operation (RCUs for it depend on consistency and size of the data read) for each item that is read from the base table.

If you exceed the capacity (e.g. you have 5 RCUs and in one second you try to do 6 strongly consistent reads), you receive a ProvisionedThroughputExceededException. Your code should catch this and retry. DynamoDB doesn't overload from this, it'll just keep accepting operations up to your capacity and reject the rest (this is called load shedding btw). The AWS SDK already implements retries with exponential backoff, and you can tune the parameters.

Tokens, Burst Capacity and Adaptive Capacity

Under the hood, DynamoDB splits the data across several partitions, and capacity is split evenly across those partitions. So, if you set 30 RCUs for a table and it has 3 partitions, each partition gets 10 RCUs. Each partition has a "token bucket", which refills at a rate of 1 token per second per RCU (so 10 tokens per second in this case). Each read (strongly consistent, up to 4 KB) consumes 1 token, and if there are no more tokens, you get a ProvisionedThroughputExceededException.

There's two separate buckets, one for reads and one for writes. They both work exactly the same, the only difference is the operations that consume those tokens, and the size of the data (4 KB for reads, 1 KB for writes). I'll talk about RCUs, but the same is true for WCUs.

The tokens bucket has a maximum capacity of 300 * RCUs. For our example of 10 RCUs per partition (remember that each partition has its own bucket), it has a maximum capacity of 3000 tokens, refilling at 10 tokens per second. That means, with no operations going on, it takes 5 minutes to fill up to capacity.

If there's a sudden spike in traffic, these extra tokens that have been piling up will be used to execute those operations, effectively increasing the partition's capacity temporarily. For example, if every user performs 1 strongly consistent read per second on this partition, your RCUs of 10 per partition would serve 10 users. Suppose you don't have users for 5 minutes, the bucket fills up. Then, 20 users come in all of a sudden, making 20 reads per second on this partition. Thanks to those stored tokens, the partition can sustain those 20 reads per second for 5 minutes, even though the partition's RCUs are 10. AWS doesn't handle huge spikes instantaneously (e.g. it won't serve 3000 reads in a second, even if you do have 3000 tokens), but it scales this over a few seconds. This is called Burst Capacity, and it's completely separate from Auto Scaling (and will happen even with Auto Scaling disabled).

Another thing that happens with uneven load on partitions is Adaptive Capacity. RCUs are split evenly across partitions, so each of our 3 partitions will have 10 RCUs. If partition 1 is the one getting these 20 users, and the others are getting 0, then AWS can assign part of those 20 spare RCUs you have (remember that we set 30 RCUs on the table) to the partition that's handling that load. The maximum RCUs a partition can get is 1.5x the RCUs it normally gets, so in this case it would get 15 RCUs. That means 5 of our 20 spare RCUs are assigned to that partition, and the other 15 are unused capacity. That would let our partition handle those 20 users for 10 minutes instead of 5 (assuming a full token bucket). This also happens separately from Auto Scaling.

Burst Capacity doesn't effectively change RCUs, but it can make our table temporarily behave as if it had more RCUs than it really does, thanks to those stored tokens (very similar to how CPU credits work for EC2 burstable instances). It's great for performance, but I wouldn't count it as scaling (in case you forgot, we're talking about scaling DynamoDB).

Adaptive Capacity can actually increase RCUs beyond what's set for the table. If all partitions are getting requests throttled, Adaptive Capacity will increase their RCUs up to the 1.5 multiplier, even if this puts the total RCUs of the table above the value you set. This will only last for a few seconds, after which it goes back to the normal RCUs, and to throttling requests. I guess that technically counts as scaling the table's RCUs? Yeah, I'll count that as a win for me. Let's get to the real scaling though.

Scaling in Provisioned Mode

This is the real scaling. DynamoDB tables continuously send metrics to CloudWatch, CloudWatch triggers alarms when those metrics cross a certain threshold, DynamoDB gets notified about that and modifies Capacity Units accordingly.

On DynamoDB you enable Auto Scaling, set a minimum and maximum capacity units, and set a target utilization (%). You can enable scaling separately for Reads and Writes.

In the table metrics (handled by CloudWatch) you can view provisioned and consumed capacity, and throttled request count.

Here's the problem though: Auto Scaling is based on CloudWatch Alarms that trigger when the metric is above/below the threshold in at least 3 data points for 5 minutes. So, not only Auto Scaling doesn't respond fast enough for sudden spikes, it doesn't respond at all if the spikes last less than 5 minutes. That's why the default threshold is 70%, and allowed values are between 20% and 90%: You need to leave some margin for traffic to continue growing while Auto Scaling takes it sweet time to figure out it should scale.

Luckily, we have Bust Capacity and Adaptive Capacity to deal with those infrequent spikes, and retries can help you eventually serve the requests that were initially throttled. You probably can't retry your way into Auto Scaling (imagine waiting 5 minutes for a request…), but retries can give Burst Capacity the few seconds it needs to kick in. Adaptive Capacity adjusts slower, and it's intended to fix uneven traffic across partitions, so don't count on it.

Now, this is all looking a lot like EC2 instances in an Auto Scaling Group, right? And we're seeing the same problems: We need to keep some extra capacity provisioned, as a buffer for traffic spikes. Even then, if a spike is big and fast enough, we can't respond to it! (except for some bursting). Why do we have these problems, if DynamoDB is supposed to be serverless? Well, it is serverless: you don't manage servers, but they're still there. What did you expect, magic? Well, sufficiently advanced science is indistinguishable from magic. Let's see if DynamoDB's other mode is close enough to serverless magic.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

DynamoDB On-Demand Mode

Welcome to the real serverless mode of DynamoDB! With On-Demand mode, you don't need to worry about scaling your DynamoDB table, it happens automatically. Wait, you really believed that? Of course you need to worry! But it's much simpler to understand and manage, and it results in less throttles.

Capacity in On-Demand Mode

The cost of reads and writes stays the same: a read operation consumes 1 Read Request Unit (RRU) for every 4 KB read (half if it's eventually consistent), and a write operation consumes 1 WRU (Write Request Unit) for every 1 KB written. Twice for transactions, LSIs increase it, yadda yadda. Same as for Provisioned, we just changed Capacity Units for Request Units.

Here's the difference: There is no capacity you can set. You're billed for every actual operation, and DynamoDB manages capacity automatically and transparently. However, it does have a set capacity, it does scale, and understanding how it does is important.

Scaling in On-Demand Mode

Every newly-created table in On-Demand mode starts with 4.000 WCUs and 12.000 RCUs (yeah, that's a lot). You're not billed for those capacity units though, you'll only be billed for actual operations.

Every time your peak usage goes over 50% of the current assigned capacity, DynamoDB increases the capacity of that table to double your peak. So, suppose you used 5.000 WRUs, now your table's WCUs are 10.000. This growth has a cooldown period of 30 minutes, meaning it won't happen again until 30 minutes after the last increase.

This isn't documented anywhere, and I haven't managed to get official confirmation, but apparently capacity for On-Demand tables doesn't decrease, ever. This seems consistent with how DynamoDB works under the hood: Partitions are split in two and assigned to new nodes, with each node having a certain maximum capacity of 3000 RCUs and 1000 WCUs. Apparently partitions are never re-combined, so there's no reason to think capacity for On-Demand tables would decrease. Again, this isn't published anywhere, just a common assumption.

Switching from Provisioned Mode to On-Demand Mode

You can switch modes in either direction, but you can only do so once every 24 hours. If you switch from Provisioned Mode to On-Demand mode, the table's initial RCUs are the maximum of 12.000, your current RCUs, or double the units of the highest peak. Same for WCUs, the maximum between 4.000, your current WCUs, or double the units of the highest peak.

If you switch from On-Demand mode to Provisioned Mode, you need to set up your capacity or auto scaling manually.

In either case the switch takes up to 30 minutes, during which the table continues to function like before the switch.

Provisioned vs On-Demand - Pricing Comparison

In Provisioned Mode, like with anything provisioned, you're billed per provisioned capacity, regardless of how much you actually consume. The price is $0.00065/hour per WCU, and $0.00013/hour per RCU.

In On-Demand Mode you're only billed for Request Units (which is basically Capacity Units that were actually consumed). The price is $1.25 per million WRUs and $0.25 per million RRUs.

Let's consider some scenarios. Assume all reads are strongly consistent and read 4 KB of data, and all writes are outside transactions and for 1 KB of data. Also, there are no secondary indexes. Suppose you have the following traffic pattern:

• Between 2000 and 3000 (average 2500) reads per second during 8 hours of the day (business hours).

• Between 400 and 600 (average 500) reads per second during 16 hours of the day (off hours).

• Between 200 and 300 (average 250) writes per second during 8 hours of the day (business hours).

• Between 50 and 150 (average 100) writes per second during 16 hours of the day (off hours).

With Provisioned Mode, no Auto Scaling, we'll need to set 3000 RCUs and 300 WCUs. The price would be $0,39 per hour for reads and $0,195 per hour for writes, for a total of $280,80 + $140,40 = $421,20 per month.

With Provisioned Mode, Auto Scaling set for a minimum 400 and a maximum 3000 RCUs and minimum 50 and maximum 300 WCUs, we'll get the following:
For business hours: We'll use our average 2500 reads, so we get $0,325 per hour for reads, and $78/month for reads for business hours. For writes, using our average of 250, we get $0,1625/hour and $39/month.
For off hours: Using the average values, $0,065/hour and $31,20/month for reads, and $0,065/hour and $31,20/month for writes,
In total, we get $179,40 per month.

With On-Demand Mode, we'll just use the averages. With 2500 reads per second we have 9.000.000 reads per hour on business hours, which costs us $2,25/hour, or $540/month. We have an average 250 writes per second, so 900.000 writes per hour, which costs $1,125/hour or $270/month.
On off hours we have 1.800.000 reads per hour, for $0,45/hour and $216/month. Writes are 360.000/hour, at $0,45/hour and $216/month.
Our grand total is $540 + $270 + $216 + $216 = $1.242/month.

Note: These prices are only for reads and writes. Storage is priced separately, and so are other features like backups.

Best Practices for Scaling DynamoDB

Operational Excellence

• Monitor Throttling Metrics: Keep an eye on the ReadThrottleEvents and WriteThrottleEvents metrics in CloudWatch. Compare them with your app's latency metrics, to determine how much this is impacting your app.

• Audit Tables Regularly: Review your DynamoDB tables to make sure that they're performing and scaling well. This includes reviewing capacity settings, and reviewing indices and keys.

Security

• Enable Point-In-Time Recovery: Data corruption doesn't happen, until it happens. Enabling Point-In-Time Recovery allows you to restore a table to a specific state if needed.

Reliability

• Pre-Warm On-Demand Tables: When expecting a big increase in traffic (like a product launch), if you're using an On-Demand table, make sure to pre-warm it. You can do this by switching it to Provisioned Mode, setting its capacity to a large number and keeping it there for a few minutes, and then switching it back to On-Demand so the On-Demand capacity matches the capacity the table had in Provisioned Mode. Remember that you can only switch once every 24 hours.

Performance Efficiency

• Use Auto-Scaling: This one's quite obvious, I hope. But the point is that there's no reason not to use this. Sure, sometimes it isn't fast enough, but in those cases Provisioned Mode without Auto Scaling won't work well either.

• Choose the Right Partition Key: Remember what I said about capacity units being split across partitions? Well, if you pick a PK that doesn't distribute traffic uniformly (or as uniformly as possible), you're going to have a problem called hot partition. This is part of DynamoDB Database Design, but as you saw, it affects performance and scaling.

Cost Optimization

• Pick The Right Mode: You saw the numbers in the example. On-Demand will very rarely result in throttling, but it is expensive. Only use it for traffic that spikes in less than the 5 minutes that Provisioned Mode with Auto Scaling takes to scale.

• Monitor and Adjust Provisioned Capacity: Regularly review your capacity settings and adjust them. Traffic patterns change over time!

• Use Reserved Capacity: If you have a consistent and predictable workload (like in the pricing example), consider purchasing reserved capacity for DynamoDB. It works similar to Reserved Instances: You reserve it and commit to a year or 3, for a lower price.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

Set up a restrictive bucket policy and add a VPC endpoint with an Endpoint Policy

Here's the initial setup, and you can deploy it here:

Deploy initial setup

This is what it looks like before the solution:

This is what it looks like with the solution:

Step by step instructions to secure the connection from EC2 to S3

Step 0: Test that the connection is working

• Open the CloudFormation console

• Select the initial state stack

• Click the Outputs tab

• Copy the value for EC2InstancePublicIp

• Paste it in the browser, append :3000 and hit Enter/Return

Step 1: Create a VPC Endpoint

• Go to the VPC console

• In the panel on the left, click Endpoints

• Click Create Endpoint

• Enter a name

• In the Services section, enter S3 in the search box, and select the one that says 'com.amazonaws.your_region.s3' (replace 'your_region' with the region where you deployed the initial setup, which is where the S3 bucket is). Then select the one that says Interface in the Type column.

• For VPC, select SimpleAWSVPC from the dropdown list

• Under Subnets, select us-east-1a and us-east-1b, and for each click the dropdown and select the only available subnet

• Under Security groups, select the one called VPCEndpointSecurityGroup

• Under Policy, pick Full Access for now (we'll change that in Step 2).

• Open Additional settings

• Check Enable DNS name

• Uncheck Enable private DNS only for inbound endpoint

• Click Create endpoint

Step 2: Configure the VPC Endpoint Policy

• In the Amazon VPC console, go to Endpoints

• Select the Endpoint you just created

• Click the Policy tab

• Click Edit Policy

• Modify the following JSON by replacing the placeholder values REPLACE_BUCKET_NAME and REPLACE_VPC_ID with the name of your S3 bucket and the ID of SimpleAWSVPC. Then paste it into the Edit Policy page, and click Save.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAccessToSpecificBucket",
            "Principal": "*",
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::REPLACE_BUCKET_NAME",
                "arn:aws:s3:::REPLACE_BUCKET_NAME/*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:sourceVpc": "REPLACE_VPC_ID"
                }
            }
        }
    ]
}

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Step 3: Set up a more restrictive bucket policy

• Open the S3 console

• Click on the bucket that you created with the initial setup

• Click on the Permissions tab

• Scroll down to Bucket Policy and click Edit

• Paste the following policy, replacing the placeholders REPLACE_BUCKET_NAME and REPLACE_VPC_ENDPOINT_ID with their values (REPLACE_VPC_ENDPOINT_ID is not the same as REPLACE_VPC_ID from the previous step). Then click Save changes

{
    "Version": "2012-10-17",
    "Id": "Policy1415115909153",
    "Statement": [
        {
            "Sid": "Access-only-from-SimpleAWSVPC",
            "Effect": "Deny",
            "Principal": "*",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE_BUCKET_NAME",
                "arn:aws:s3:::REPLACE_BUCKET_NAME/*"
            ],
            "Condition": {
                "StringNotEquals": {
                    "aws:SourceVpce": "REPLACE_VPC_ENDPOINT_ID"
                }
            }
        },
        {
            "Sid": "Access-from-everywhere",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::REPLACE_BUCKET_NAME",
                "arn:aws:s3:::REPLACE_BUCKET_NAME/*"
            ]
        }
    ]
}

Step 4: Test that the connection is still working

• Go back to the browser tab where you pasted the public IP address of the instance and refresh the page

Step 5: Empty the S3 bucket

Before deleting the CloudFormation stack, you'll need to empty the S3 bucket! The Node.js app puts a file in there.

How does this solution make the connection from EC2 to S3 more secure?

VPC Endpoints

First of all, you'll notice that a VPC Endpoint is for one specific service, S3 in this case. If you wanted to connect to other services you'd need to create a separate VPC Endpoint for each different service.

The second thing you'll notice is that there are 2 types of endpoints: Interface and Gateway. Gateway endpoints are only for S3 and DynamoDB, while Interface endpoints are for nearly everything. Gateway endpoints are simpler, so use them when you can (except if you're writing a newsletter and want to show a few things about Interface endpoints).

Interface endpoints work by creating an Elastic Network Interface in every subnet where you deploy it, and automatically routing to that ENI the traffic that's addressed to the public endpoint of the service. That way, you don't need to make any changes to the code. This only works if you check Enable DNS name.

VPC Endpoint Policies

The existing policy is a Full Access policy, which is the default policy when a VPC endpoint is created. It allows all actions on the S3 service from anyone.

Instead of that, we're setting up a more restrictive policy, which only allows access to our specific bucket, and denies access to all other buckets.

VPC Endpoint policies are IAM resource policies, and as such, anything that's not explicitly allowed is implicitly denied.

Restrictive S3 bucket policies

Bucket policies are another type of IAM resource policies. Obviously, this bucket policy will only apply to our S3 bucket. It's important to add it because, while we've restricted what the VPC Endpoint can be used for, the S3 bucket can still be accessed from outside the VPC (e.g. from the public internet). This bucket policy is the one that's going to prevent that, restricting access to only from the VPC Endpoint.

Discussing Connection Security to S3

In this case I kept internet access for the VPC and for the EC2 instance itself, just to make it easier to trigger the code with an HTTP request. This solution is a good idea in these cases because traffic to S3 doesn't go over the public internet, but admittedly, the public internet is a viable alternative.

Where this solution matters more is when you don't have access to the internet. Sure, adding it is rather simple, but you're either exposing yourself unnecessarily by giving your instances a public IP address they don't need, or you're paying for a NAT Gateway. In those cases, VPC Endpoints are a much simpler, safer and cheaper solution.

Conceptually, you can think of this as giving the S3 service a private IP address inside your VPC. In reality, what you're doing is creating a private IP address in your VPC that leads to the S3 service, so that conception is pretty accurate! Behind the scenes (and you can see this easily), the VPC service creates an Elastic Network Interface (ENI) in every subnet where you deploy the VPC Endpoint. Those ENIs will forward the traffic to the S3 service endpoints that are private to the AWS network.

Also, behind the scenes there's a Route 53 Private Hosted Zone that you can't see, but which resolves the S3 address to the private IPs of those ENIs, instead of to the public IPs of the public endpoints. That's why you don't need to change the code: Your code depends on the address of the S3 service, and that private hosted zone takes care of resolving it to a different address. You can't see this private hosted zone, it's managed by AWS and hidden from users.

Best Practices for S3 Security

Operational Excellence

• Monitor and Alert Endpoint Health: Monitor the health of your VPC endpoints using CloudWatch metrics. Any unusual activity or degradation in performance should trigger alerts. This could also help you detect a security incident!

Security

• Least Privilege Access to Bucket: This is basically what we did in Step 3: We disabled public access, and implemented a policy that only allows reads from the VPC. Try reading from that S3 bucket from your own computer: aws s3api get-object --bucket 12ewqaewr2qqq --key thankyou.txt thankyou.txt --region us-east-1

• Regularly Audit IAM Policies: Regularly review and tighten your IAM policies. Not only for the VPC Endpoint and S3 bucket, but also for the EC2 instance!

Reliability

• Use Multiple Subnets in Different AZs: Each subnet gets one ENI, so if you distribute your subnets in several AZs, your VPC Endpoint is highly available within the region (i.e., it can continue functioning if an Availability Zone fails).

Performance Efficiency

• Choose the Right VPC Endpoint Type: Choose the right type of VPC Endpoint based on your workload. For S3, a Gateway Endpoint works best. I'll leave it to you to figure out how to create it (=.

Cost Optimization

• Delete Unused VPC Endpoints: Regularly delete any unused VPC endpoints to avoid paying for stuff you don't use.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

Suddenly, someone's product goes viral, and a lot of people rush in to buy it at the same time. Our cache and CDN don't even blink at the traffic, our well-designed Lambdas scale amazingly fast, but our DynamoDB table is suddenly bombarded with writes and the auto scaling can't keep up. Our order processing Lambda receives ProvisionedThroughputExceededException, and when it retries it just makes everything worse. Things crash. Sales are lost. We eventually recover, but those customers are gone. How do we make sure it doesn't happen again?

Option 1 is to change the DynamoDB table to On-demand, which can keep up with Lambda when scaling, but it's over 5x more expensive. Option 2 is to make sure the table's write capacity isn't exceeded. Let's explore option 2.

AWS Services involved:

• DynamoDB: Our database. All you need to know for this post is how DynamoDB scales.

• SQS: A fully managed message queuing service that enables you to decouple components. Producers like our order processing microservice post to the queue, the queue stores these messages until they're read, consumers read from the queue in their own time.

• SES: An email platform, more similar to services like MailChimp than to an AWS service. If you're already on AWS and you just need to send emails programmatically, it's easy to set up. If you're not on AWS, need more control, or need to send so many emails that price is a factor, you'll need to do some research. For this post, SES is good enough.

What is Amazon SQS

SQS is a fully managed message queuing service. A messaging queue is a data structure where items can be read in the same order as they were written: First-In, First-Out (FIFO).

Queues allow us to decouple components by making the consumer (that's the component that reads from the queue) unaware of who wrote the item (the writer is called producer). Additionally, in software architecture we usually focus on another characteristic of queues: The reading of the item can happen some time after the writing. This lets us decouple producers and consumers in time: Consumers don't need to be available when Producers write to the queue. The queue stores the messages for a certain amount of time, and when Consumers are ready, they poll the queue for messages, and receive the oldest message.

For our solution, we're going to use a queue so that our order processing microservice can send a message with the order, the queue stores the message, and a consumer can read it at its own rhythm (i.e. at our DynamoDB table's rhythm).

Types of SQS Queues

There's two types of queues in SQS:

• Standard queues are the default type of queue. They're cheaper than FIFO queues and nearly-infinitely scalable. The tradeoff is that they only guarantee at-least-once delivery (meaning you might get duplicates), and order of the messages is mostly respected but not guaranteed.

• FIFO queues are more expensive than Standard queues, and they don't scale infinitely, but they guarantee ordered, exactly-once delivery. You need to set the MessageGroupId property in the message, since FIFO queues only deliver the next message in a MessageGroup after the previous message has been successfully processed. For example, if you set the value of MessageGroupId to the customer ID and a customer makes two orders at the same time, the second one to come in won't be processed until the first one is finished processing. It's also important to set MessageDeduplicationId, to ensure that if the message gets duplicated upstream, it will be deduplicated at the queue. A FIFO queue will only keep one message per unique value of MessageDeduplicationId.

Most people who think of queues are thinking guaranteed FIFO order and exactly-once delivery. The only way to actually get those guarantees is with FIFO queues.

How to Implement an SQS Queue for DynamoDB

Follow these step by step instructions to implement an SQS Queue to throttle writes to a DynamoDB table. Replace YOUR_ACCOUNT_ID and YOUR_REGION with the appropriate values for your account and region.

Create the Orders Queue

1. Go to the SQS console.

2. Click "Create queue"

3. Choose the "FIFO" queue type (not the default Standard)

4. In the "Queue name" field enter "OrdersQueue"

5. Leave the rest as default

6. Click on "Create queue"

Update the Orders service to write to the SQS queue

We need to update the code of the Orders service so that it sends the new Order to the Orders Queue, instead of writing to the Orders table. This is what the code looks like:

const AWS = require('aws-sdk');
const sqs = new AWS.SQS();
const queueUrl = 'https://sqs.YOUR_REGION.amazonaws.com/YOUR_ACCOUNT_ID/OrdersQueue';

async function processOrder(order) {
  const params = {
    MessageBody: JSON.stringify(order),
    QueueUrl: queueUrl,
    MessageGroupId: order.customerId,
    MessageDeduplicationId: order.id
  };

  try {
    const result = await sqs.sendMessage(params).promise();
    console.log('Order sent to SQS:', result.MessageId);
  } catch (error) {
    console.error('Error sending order to SQS:', error);
  }
}

Also, add this policy to the IAM Role of the function, so it can access SQS. Don't forget to delete the permissions to access DynamoDB!

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:YOUR_REGION:YOUR_ACCOUNT_ID:OrdersQueue"
    }
  ]
}

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Set up SES to notify the customer via email

1. Open the SES console

2. Click on "Domains" in the left navigation pane

3. Click "Verify a new domain"

4. Follow the on-screen instructions to add the required DNS records for your domain.

5. Alternatively, click on "Email Addresses" and then click the "Verify a new email address" button. Enter the email address you want to verify and click "Verify This Email Address". Check your inbox and click the link.

Set up the Order Processing service

Go to the Lambda console and create a new Lambda function. Add the following code:

const AWS = require('aws-sdk');
const dynamoDB = new AWS.DynamoDB.DocumentClient();
const ses = new AWS.SES();

exports.handler = async (event) => {
    for (const record of event.Records) {
        const order = JSON.parse(record.body);
        await saveOrderToDynamoDB(order);
        await sendEmailNotification(order);
    }
};

async function saveOrderToDynamoDB(order) {
    const params = {
        TableName: 'orders',
        Item: order
    };

    try {
        await dynamoDB.put(params).promise();
        console.log(`Order saved: ${order.orderId}`);
    } catch (error) {
        console.error(`Error saving order: ${order.orderId}`, error);
    }
}

async function sendEmailNotification(order) {
    const emailParams = {
        Source: 'you@simpleaws.dev',
        Destination: {
            ToAddresses: [order.customerEmail]
        },
        Message: {
            Subject: {
                Data: 'Your order is ready'
            },
            Body: {
                Text: {
                    Data: `Thank you for your order, ${order.customerName}! Your order #${order.orderId} is now ready.`
                }
            }
        }
    };

    try {
        await ses.sendEmail(emailParams).promise();
        console.log(`Email sent: ${order.orderId}`);
    } catch (error) {
        console.error(`Error sending email for order: ${order.orderId}`, error);
    }
}

Also, add the following IAM Policy to the IAM Role of the function, so it can be triggered by SQS and access DynamoDB and SES:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:YOUR_REGION:YOUR_ACCOUNT_ID:OrdersQueue"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:YOUR_REGION:YOUR_ACCOUNT_ID:table/Orders"
    },
    {
      "Effect": "Allow",
      "Action": "ses:SendEmail",
      "Resource": "*"
    }
  ]
}

Make the Orders Queue trigger the Order Processing service

1. In the Lambda console, go to the Order Processing lambda

2. In the "Function overview" section, click "Add trigger"

3. Click "Select a trigger" and choose "SQS"

4. Select the Orders Queue

5. Set Batch size to 1

6. Make sure that the "Enable trigger" checkbox is checked

7. Click "Add"

Limit concurrent executions of the Order Processing Lambda

1. In the Lambda console, go to the Order Processing lambda

2. Scroll down to the "Concurrency" section

3. Click "Edit"

4. In the "Provisioned Concurrency" section, set "Reserved Concurrency" to 10

5. Click "Save"

Synchronous and Asynchronous Workflows with SQS

Architecture-wise, there's one big change in our solution: We've made our workflow async! Let me bring the diagram here.

Before, our Orders service would return the result of the order. From the user's perspective, they wait until the order is processed, and they see the result on the website. From the system's perspective, we're constrained to either succeed or fail processing the order in the timeout limit of API Gateway (29 seconds). In more practical terms, we're limited by what the user is expecting: we can't just show a "loading" icon for 29 seconds!

After the change, the website just shows something like "We're processing your order, we'll email you when it's ready". That sets a different expectation to the user. That's important for the system, because now we could actually have our Lambda function take 15 minutes, without hitting the 29 seconds limit of API Gateway, or without the user getting angry. It's not just that though, if the Order Processing lambda crashes mid-execution, the SQS queue will make the order available again as a message after the visibility timeout expires, and the Lambda service will invoke our function again with the same order. When the maxReceiveCount limit is reached, the order can be sent to another queue called Dead Letters Queue (DLQ), where we can store failed orders for future reference. We didn't set up a DLQ here, but it's easy enough, and for small and medium-sized systems you can easily set up SNS to send you an email and resolve the issue manually, since the volume shouldn't be particularly large.

Once the order went through all the steps, failed some, retried, succeeded, etc, then we notify the user that their order is "ready". This can look different for different systems, some are just a "we got the money", some ship physical products, some onboard the user to a complex SaaS. For this solution I chose to do it via email because it's easy and common enough, but you could use a webhook and still keep the process async.

Best Practices for SQS and DynamoDB

Operational Excellence

• Monitor and set alarms: You know how to monitor Lambdas. You can monitor SQS queues as well! An interesting alarm to set here would be number of orders in the queue, so our customers don't wait too long for their orders to be processed.

• Handle errors and retries: Be ready for anything to fail, and architect accordingly. Set up a DLQ, set up notifications (to you and to the user) for when things fail, and above all don't lose/corrupt data.

• Set up tracing: We're complicating things a bit (hopefully for a good reason). We can gain better visibility into that complexity by setting up X-Ray.

Security

• Check "Enable server-side encryption": That's all you need to do for an SQS queue to be encrypted at rest: check that box, and pick a KMS key. SQS communicates over HTTPS, so you already have encryption in transit.

• Tighten permissions: The IAM policies in this issue are pretty restrictive. But there's always a nut to tighten, so keep your eyes open.

Reliability

• Set up maxReceiveCount and a DQL: With a FIFO queue, the next message won't be available for processing until the previous one is either processed successfully or dropped (to the DLQ if you set one) after maxReceiveCount attempts. If you don't set these, one corrupted order will block your whole system.

• Set visibility timeout: This is the time that SQS waits without receiving the "success" response, before assuming the message wasn't processed successfully and making it available again for the next consumer. Set a reasonable value, and set the same value as a timeout for your consumer (Order Processing lambda in this case).

Performance Efficiency

• Optimize Lambda function memory: More memory means more money. But it also means faster processing. Going from 30 to 25 seconds won't matter much for a successfully processed order, but if orders are retried 5 times, now it's 25 seconds we're gaining instead of 5. Could be worth it, depending on your customers' expectations.

• Use Batch processing: As discussed earlier, you should consider processing messages in batches.

• Remember the 20 advanced tips for Lambda.

Cost Optimization

• Provisioned vs. On-demand for DynamoDB: Remember that this could be fixed by using our DynamoDB table in On-demand mode. It's 5x more expensive though. Same goes for relational databases (if we use Aurora, then Aurora Serverless is an option).

• Consider something other than Lambda: In this case, we're trying to get all orders processed relatively fast. If the processing can wait a bit more, an auto scaling group that scales based on the number of messages in the SQS queue can work wonders, for a lot less money.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

EBS Basic Concepts

What is an EBS Volume

An EBS volume is a virtual block-level storage device that can be used by EC2 instances to store persistent data. An EBS volume acts like a HDD or SSD, but behind the scenes they're actually an array of physical discs in a RAID configuration, in the same datacenter but physically distanced from each other to minimize the probability of simultaneous failures (e.g. due to fires).

When you create an EBS volume you can define the size, the performance (only for some volume types), and the volume type, which is explained in the next section. Size and performance can be changed later, but not volume type.

EBS volumes exist separate from EC2 instances, and can be detached to one instance and attached to another one. When you create an EC2 instance, an EBS volume is created and attached as a root volume, and the default behavior is to delete it when the instance is terminated (this can be changed). But it's important to understand that they are actually a separate service from EC2.

Types of EBS Volumes

As is often the case in AWS, there's different types of resources to serve different use cases and needs. These are the types of volumes you can create in EBS. Remember that you set this on creation and cannot change it later.

EBS GP3 Volumes

This is the general-purpose volume type, which you should use for most stuff, and default to when in doubt. Size and performance (IOPS) can be configured separately (unlike the previous generation, GP2). Here are some details:

• Volume Size: 1 GB to 16 TB

• Durability: 99.8% to 99.9%

• Max IOPS/Volume: 16,000 (operations of 16K)

• Max Throughput/Volume: 1000 MB/s

• Latency: single digit milliseconds

• Price:

  • $0.08/GB-month

  • $0.005/provisioned IOPS-month over 3,000 (the first 3,000 are free)

EBS IO2 Volumes

GP3 is great for most use cases, but if you need more performance, you go with IO2. If you're running a database on an EC2 instance, this is the one to pick. Size and performance (IOPS) can be configured separately, and the limits are higher than for GP3. Details:

• Volume Size: 4 GB to 16 TB

• Durability: 99.999%

• Max IOPS/Volume: 64,000 (operations of 16K). 256,000 IOPS with io2 Block Express.

• Max Throughput/Volume: 1,000 MB/s. 4,000 MB/s with io2 Block Express.

• Latency: single digit millisecond

• Price:

  • $0.125/GB-month

  • $0.065/provisioned IOPS-month up to 32,000 IOPS (no free IOPS)

  • $0.046/provisioned IOPS-month from 32,001 to 64,000 IOPS

EBS ST1 Volumes

This one is actually an HDD (virtual, but backed by real HDDs). Spinning disks! The use case is sequential access to data that sits contiguously in the physical disks. In general, that means data that is written as a long stream, and then read as a long stream, instead of having different parts accessed at random. It offers pretty good performance for that compared to GP3, at almost half the price. Here are the specs:

• Volume Size: 125 GB to 16 TB

• Durability: 99.8% to 99.9% durability

• Max IOPS/Volume: 500 (operations of 1 MB, not 16K)

• Max Throughput/Volume: 500 MB/s

• Price: $0.045/GB-month

Maximum performance varies per size, at 40 MB/s per TB. Additionally, ST1 Volumes use a burst credits system to accumulate credits while usage is below peak throughput, and consume them to achieve for a period of time a higher throughput than the soft limit. In short, a 12.5 TB volume always performs at max 500 MB/s, and any volume below that has a lower soft limit, but can reach 500 MB/s for a short period of time. This is better explained here.

Certification Exam tip: ST1 volumes can't be root volumes.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

EBS SC1 Volumes

SC1 volumes are also HDDs, aimed at offering the lowest price per GB of all block storage options. They're recommended for infrequently accessed data that needs to be accessed from block storage (the C in SC1 stand for Cold). If you don't strictly need block storage access, S3 Infrequent Access or Glacier are also viable options.

• Volume Size: 125 GB to 16 TB

• Durability: 99.8% to 99.9% durability ← Much less than S3!

• Max IOPS/Volume: 250 (operations of 1 MB, not 16K)

• Max Throughput/Volume: 250 MB/s

• Price: $0.015/GB-month ← Slightly higher than S3 Infrequent Access, but with EBS SC1 you're not billed for reads and writes

SC1 volumes use the same burst credit system as ST1, though their performance is lower.

Certification Exam tip: SC1 volumes can't be root volumes.

Older Volume Types

IO1 (predecessor of IO2) and GP2 (predecessor of GP3) were the norm a few years ago, and you'll most likely find some still in production. IO1 works just like IO2, but with less durability, and a higher price at high IOPS. GP2 has the same use cases than GP3, but performance was tied to volume size and it used a burst credits system, like ST1. It's also 25% more expensive than GP3.

Cost-savings tip: Migrate GP2 volumes to GP3.

Characteristics of EBS Volumes

EC2 Root Volume

An EC2 instance comes with an EBS volume associated with it, called the root volume. This volume contains the OS, some libraries and programs, and some configurations. This is where the EC2 instance boots from when starting up.

Multiple EBS Volumes

You can attach up to 128 EBS volumes to an instance (depending on instance type), so long as they're in the same Availability Zone as the EC2 instance. Once a volume is attached, you'll see it in the OS just like if you were physically attaching a disk, and you can mount it on the OS or file system.

You can detach an EBS volume from an instance and attach it to another instance as many times as you want, so long as both instances are in the same Availability Zone as the EC2 instance.

Volumes of any type other than IO1 or IO2 can only be attached to once instance at a time. IO1 and IO2 volumes can be attached to multiple instances at the same time, in read-write mode.

Availability of EBS Volumes

EBS volumes are zonal resources. They exist in a single Availability Zone, so they are not highly available. This is also true for IO2 volumes, which offer durability of 99.999%.

EBS volumes are redundant within that Availability Zone, so data loss is significantly less likely than with a single disk. They're backed by an array of physical disks in a RAID configuration.

Lifecycle of an EBS Volume

It's important to understand that the lifecycle of an EBS volume is separate from that of the EC2 instance. You can create them, attach them, detach them and delete them on their own. You can also configure them to be deleted when the EC2 instance they're attached to is terminated, which is the default for the root volume.

Encryption of EBS Volumes

EBS volumes can be encrypted using KMS, in a way that's entirely transparent to you. You don't need to manage or use any encryption keys, the EBS service automatically fetches them and decrypts the data when you initiate a read operation.

You can enable Encryption by Default on your AWS account, which means every new EBS volume you create will be encrypted unless you explicitly configure it not to. This is a highly recommended best practice for AWS security.

Backing up EBS Volumes: EBS Snapshots

EBS snapshots are point-in-time copies of EBS volumes, used to back up and restore data. They're incremental, which means they only capture the data that has changed since the last snapshot. This makes EBS snapshots more efficient and cost-effective than full-volume backups. The size of an EBS snapshot is calculated based on the amount of data stored in the volume at the time the snapshot was taken.

Snapshots are regional resources, meaning you can use them to hold a copy of an EBS volume and, if the volume's Availability Zone fails, restore it from the snapshot in a different Availability Zone. They can also be shared across regions and AWS accounts. Here's where you can read more about automating EBS snapshots for Disaster Recovery.

EBS Best Practices

Default to EBS GP3 Volumes

You should default to GP3 volumes, and only use the other volume types if you have a specific use case, or you know you need more performance. Here's a guide to benchmark EBS volumes.

Use EBS-optimized instances for higher performance

EC2 Instance families have a limit on performance with EBS volumes, which is independent of the EBS volume itself. If you need high performance, it may not be enough to just use a better EBS volume such as IO2. You'll also need to look into whether your EC2 instances support that level of performance, and possibly use EC2 instances that are Storage Optimized.

EBS performance is also limited by instance size. You can use the EBSIOBalance% and EBSByteBalance% metrics in CloudWatch to help you determine whether your instances are sized correctly. Instances with a consistently low balance percentage should be increased in size, and instances where the balance percentage never drops below 100% should be reduced in size.

Use EC2 Instance Store for extreme performance

If you need extreme performance, you'll need to use EC2 Instance Store. It's ephemeral (that means non-permanent) block storage with a much higher performance than EBS. The main disadvantages are the pricing (you need an EC2 instance of a special family, which isn't cheap) and the fact that data is lost if the instance is stopped or terminated.

Encrypt your EBS Volumes

This comes at no cost and no performance hit to you, so it should be a no brainer. First, you should enable Encryption by Default, so all future EBS volumes are created with encryption. Then you should encrypt existing EBS volumes by creating a snapshot of them, encrypting that snapshot and creating a new volume from the encrypted snapshot.

Migrate GP2 volumes to GP3

GP3 volumes can do anything that GP2 volumes can, and they're 20% cheaper. Here's a guide to migrate your existing GP2 volumes to GP3.

Back up important EBS volumes

Remember that EBS volumes are zonal resources, meaning that if an Availability Zone goes offline you won't be able to access them. Furthermore, most EBS volume types offer 99.8% durability, making data loss or corruption entirely possible. To guard against that, create Snapshots of your EBS volumes.

Snapshots are regional, so you're good if an AZ goes down. But if you want to do cross-region disaster recovery, you'll need to export the snapshots to another AWS region. If you're exporting encrypted snapshots, use a multi-region KMS key.

You can use Data Lifecycle Manager to automate creating and exporting EBS snapshots.

Accessing a block for the first time on an EBS volume created from a snapshot has huge latency, because the data is lazy loaded from S3. To avoid this, you can initialize (pre-warm) the volume before putting the volume in production, by accessing each block once. You can do this on Linux by attaching the volume to an EC2 instance, installing the fio utility, and running the following program (example for a volume called xvdf)

sudo fio --filename=/dev/xvdf --rw=read --bs=1M --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize

Another option is to enable EBS fast snapshot restore on the snapshot. This will ensure AWS does the initialization for you, and is much faster. However, it costs $540/month per snapshot (regardless of size), so I prefer the manual option.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com

At some point, microservices became this really cool thing that all the cool kids were doing for street cred. "Netflix does it, so if I do it I'll be as cool as Netflix!" Folks, it doesn't work like that.

What Are Microservices?

A microservice is a service in a software application which encapsulates a bounded context (including the data), can be built, deployed and scaled independently, and exposes functionality through a clearly defined API.

Let's expand a bit on each characteristic:

• Bounded context: The concept stems from Domain-Driven Design's bounded contexts, and essentially advocates for dividing the entire domain into several smaller domains. Microservices takes it a step further: each microservice is part of only one bounded context, and owns that context, including all domain entities, all data, and all operations and functionality. Anything that needs to access that bounded context needs to do so through the microservice's API. Conceptually, it's similar to encapsulation in Object-Oriented Programming, but on a higher level.

• Built, deployed and scaled independently: Each microservice is independent in every sense. It can be built by a separate team using different technologies, it has its own deployment pipeline and process, and it can be scaled independently of the rest of the system. This provides a clear separation between what the microservice does and how it does it, and gives you a lot of flexibility on that how.

• Clearly defined API: One microservice can't solve everything, or it would be just a monolith. You need several, and you need to combine them to realize the entire system's functionality. That means, you need a clear and unambiguous way to communicate with a microservice. The API is the interface of the microservice, and it's the only thing that other components of the system can access. This minimizes dependencies between microservices, letting them be implemented and evolved separately, so long as they adhere to their API. Keep in mind that, since each microservice owns its data, the only way to access the data of a microservice is through that microservice's API. No reading directly from another microservice's database!

Why Use Microservices?

Microservices exist to solve a specific problem: problems in complex domains require complex solutions, which become unmanageable due to the size and complexity of the domain itself. Microservices (when done right) split that complex domain into simpler domains, encapsulating the complexity and reducing the scope of changes.

Microservices also add complexity to the solution, because now you need to figure out where to draw the boundaries of the domains, and how the microservices interact with each other, both at the domain level (complex actions that span several microservices) and at the technical level (service discovery, networking, permissions).

So, when do you need microservices? When the reduction in complexity of the domain outweighs the increase in complexity of the solution.

When do you not need microservices? When the domain is not that complex. In that case, use regular services, where the only split is in the behavior (i.e. backend code). Or stick with a monolith, Facebook does that and it works pretty well, at a size we can only dream of.

Types of Microservices

There's two ways in which you can split your application into microservices:

• Vertical slices: Each microservice solves a particular use case or a set of tightly-related use cases. You add services as you add features, and each user interaction goes through the minimum possible number of services (ideally only 1). This means features are an aspect of decomposition. Code reuse is achieved through shared libraries, and cross-service responsibilities are implemented on support microservices. This results in architectures very similar to SOA.

• Functional services: Each service handles one particular step, integration, state, or thing. System behavior is an emergent property, resulting from combining different services in different ways. Each user interaction invokes multiple services. New features don't need entirely new services, just new combinations of services. Features are an aspect of integration, not decomposition. Code reuse is often achieved through invoking another service. This is often much harder to do, both because of the difficulty in translating use cases into reusable steps, and because you need a lot of complex distributed transactions.

Overall, vertical slices is easier to understand, and easier to implement for smaller systems. The drawback is that if your system does 200 different things, you'll need 200 services, plus support services and libraries. Functional services are harder to conceptualize, and it's not uncommon to end up with a ton of microservices that have 50 lines of code and don't own any data. If that's your case, you're doing it wrong. Remember that the split should be at the domain level, not at the code level. It's perfectly ok for a microservice to be implemented with several services!

Don't combine these two types of microservices! If you're doing vertical slices, support microservices should be only for non-business behavior, such as logging. If you're doing functional microservices, don't create a service that just orchestrates calls between other microservices; either use an orchestrator for all transactions, or choreograph them. And don't even think about migrating from one type of microservices to the other one. It's much, much easier to just drop the whole system and start from scratch.

Splitting a Monolith into Microservices

Let's see microservices in a real example. Picture the following scenario: We have an online learning platform built as a monolithic application, which enables users to browse and enroll in a variety of courses, access course materials such as videos, quizzes, and assignments, and track their progress throughout the courses. The application is deployed on Amazon ECS as a single service that's scalable and highly available.

As the app grew, we've noticed that content delivery becomes a bottleneck during normal operations. Additionally, changes in the course directory resulted in some bugs in progress tracking. To deal with these issues, we decided to split the app into three microservices: Course Catalog, Content Delivery, and Progress Tracking.

Out of scope (so we don't lose focus):

• Authentication/authorization: When I say “users” I mean authenticated users. We could use Cognito to secure access to microservices, but let's focus on designing the microservices first.

• User registration and management: Same as above.

• Payments: Since our courses are so awesome, we should charge for them. We could use a separate microservice that integrates with a payment processor such as Stripe.

• Caching and CDN: We should use CloudFront to cache the content, to reduce latency and costs. We'll do that in a future issue, let's focus on the microservices right now.

• Frontend: Obviously, we need a frontend for our app. Let's keep the focus on the microservices, but if you're interested in frontend you might want to check out AWS Amplify.

• Database design: Let's assume our database is properly designed. If you're interested in this topic, you should read DynamoDB Database Design.

• Admin: Someone has to create the courses, upload the content, course metadata, etc. The operations to do that fall under the scope of our microservices, but I feared it would grow too complex, so I cut those features out.

AWS Services involved:

• ECS: Our app is already deployed in ECS as a single ECS Service, we're going to split it into 3 microservices and deploy each as an ECS Service. We won't dive deep into ECS, but if you're interested you can learn about how to deploy a Node.js application on ECS.

• DynamoDB: Our database for this example.

• API Gateway: Used to expose each microservice.

• Elastic Load Balancer: To balance traffic across all the tasks.

• S3: Storage for the content (video files) of the courses.

• ECR: A Docker registry managed by AWS.

Final design of the app split into microservices

How to Split a Monolith Into Microservices

Step 0: Make the Monolith Modular

The first step should always be to make sure your monolith is already separated into modules with clearly defined responsibilities. Modules should be well scoped, both in terms of functionality and in the code that implements that functionality. They should be cohesive, and lowly coupled to other modules. The level of granularity doesn't matter much, though ideally you'd be splitting modules according to the concept of domains from Doman-Driven Design (you don't need to apply the entirety of Domain-Driven Design). However, you can refine the scope and granularity when you start with microservices. For now, what's important is that you have clearly defined modules with clearly defined responsibilities, instead of a bowl of spaghetti code.

For this example we're going to assume this is already the case, but if you're dealing with a monolith that's not well modularized, that should be the first thing you do. If you commit all the way to microservices, you won't really use the modular monolith. However, I still recommend you first work on separating it into modules, to make the overall process easier by tackling one thing at a time.

Step 1: Identify the Microservices

Start by analyzing the monolithic application, focusing on the course catalog, content delivery, and progress tracking functionalities. Based on these functionalities, outline the responsibilities for each microservice:

• Course Catalog: manage courses and their metadata.

• Content Delivery: handle storage and distribution of course content.

• Progress Tracking: manage user progress through courses.

Step 2: Define the APIs for each microservice

Once you understand what each microservice needs to do, you need to design the API endpoints for each microservice:

• Course Catalog:

  • GET /courses → list all courses

  • GET /courses/:id → get a specific course

• Content Delivery:

  • GET /content/:id → get a pre-signed URL for a specific course content

• Progress Tracking:

  • GET /progress/:userId → get a user's progress

  • PUT /progress/:userId/:courseId → update a user's progress for a specific course

API endpoints are how microservices define and expose their functionality to external components. Essentially, the API is what a microservice can do for the user or for other microservices. We already knew the responsibilities of each microservice from Step 1, with this step we're expressing them in technical terms that other components can understand. We're also documenting them in a clear and unambiguous way.

If you're starting from a well-designed modular monolith, these APIs already exist as the APIs for services and interfaces for components, and you're just re-expressing them in a different, unified way. If the starting monolith isn't well modularized, you may find some of these APIs as functions, and you may need to add a few. In those cases it's easier to first modularize the monolith, then split it into microservices.

API design is really important, and hard to do. We're not just splitting the entire app's responsibilities into groups that we call microservices. We're actually creating several apps, that we're then going to interconnect to produce the expected system behavior. We need to not only define those apps' responsibilities well, but also design them in a maintainable way. Check out Fowler's post on consumer-driven contracts for some deeper insights.

Step 3: Configure API Gateway for each microservice

Create an API Gateway resource for each microservice (Course Catalog, Content Delivery, and Progress Tracking). Point the different routes to your monolith's APIs for now, since we don't have any microservices yet. Update any frontend code or DNS routes to resolve to the API Gateways.

This isn't a strict requirement, but I added it as part of the solution because it makes the switchover much easier: All we need to do is update the API Gateway of each microservice to point to the newly deployed microservice. Since everything else already depends on that API Gateway for that functionality, we're just changing who's resolving those requests. This way, we've effectively decoupled the API from its implementation. API Gateway also makes other things much easier, such as managing authentication for microservices.

Step 4: Create separate repositories and projects for each microservice

Set up individual repositories and Node.js projects for Course Catalog, Content Delivery, and Progress Tracking microservices. Structure the projects using best practices, with separate folders for routes, controllers, and database access code. You know the drill.

This is just the scaffolding, moving the actual code comes in the next step. The key takeaway is that you treat each microservice as a separate project. You could also use a monorepo, where the whole codebase is in a single git repository, each service has its own folder, and it's still deployed separately. This works well when you have a lot of shared dependencies, but in my experience it's harder to pull off.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

Step 5: Separate the code

Refactor the monolithic application code, moving the relevant functionality for each microservice into its respective project:

• Move the code related to managing courses and their metadata into the Course Catalog microservice project.

• Move the code related to handling storage and distribution of course content into the Content Delivery microservice project.

• Move the code related to managing user progress through courses into the Progress Tracking microservice project.

The code in the monolith may not be as clearly separated as you might want. In that case, first refactor as needed until you can copy-paste the implementation code from your monolith to your services (but don't copy it just yet). Then test the refactor. Finally, do the copy-pasting.

Step 6: Separate the data

First, create separate Amazon DynamoDB tables for each microservice:

• CourseCatalog: stores course metadata, such as title, description, and content ID.

• Content: stores content metadata, including content ID, content type, and S3 object key.

• Progress: stores user progress, with fields for user ID, course ID, and progress details.

Then update the database access code and configurations in each microservice, so each one interacts with its own table.

Remember that the difference between a service and a microservice is the bounded context. Each microservice owns its domain model, including the data, and the only way to access that model (and the database that stores it) is through that microservice's API.

We could implement this separation of data at the conceptual level, without enforcing it through separate tables. We could even enforce it while keeping all data in a single table, using DynamoDB's field-level permissions. The problem with that idea (aside from the permissions nightmare) is that we wouldn't be able to scale the services independently, since DynamoDB capacity is managed per table.

If you're doing this for a database which already has data, but you can tolerate the system being offline during the migration, you can export the data to S3, use Glue to filter the data, and then import it back to DynamoDB.

If the system is live, this step gets trickier. Here's how you can split a DynamoDB table with minimal downtime:

• First, add a timestamp to your data if you don't have one already.

• Next, create the new tables.

• Then set up DynamoDB Streams to replicate all future writes to the new table. You'll need to set one stream per microservice. It's easier if you set it up to copy all the data and after the switchover you delete the irrelevant data. But if you're performing a lot of writes, it will be cheaper to selectively copy only the data that belongs to the microservice.

• Then copy the old data, either with a script or with an S3 export + Glue (don't use the DynamoDB import, it only works for new tables, write the data manually instead). Make sure this can handle duplicates.

• Finally, switch over to the new tables.

I picked DynamoDB for this example because DynamoDB tables are easy to create and manage (other than designing the data model). In a relational database we would need to consider the tradeoff between having to manage (and pay for) one DB cluster per microservice, or having different databases in the same cluster. The latter is definitely cheaper, but it can get harder to manage permissions, and we lose the ability to scale the data stores independently. Aurora Serverless is a viable alternative, it scales very similarly to DynamoDB in Provisioned Mode. However, it's 4x more expensive than serverful Aurora.

Step 7: Build and Deploy the Microservices

We're using ECS for this example, just so we can focus on the microservices part, instead of debating over how to deploy an app. These are the steps to deploy in ECS, which you'll need to do separately for each microservice:

• Write a Dockerfile specifying the base image, copying the source code, installing packages, and setting the appropriate entry point. Test this, obviously.

• Build and push the Docker image to an Amazon Elastic Container Registry (ECR) registry. You'll use a separate registry for each microservice (remember they're separate apps).

• Create a Task Definition in Amazon ECS, specifying the required CPU, memory, environment variables, and the ECR image URL.

• Create an ECS service, associating it with the corresponding Task Definition. Make sure this is working properly.

I don't want to dive too deep into how to deploy an app to ECS. If you're not sure how to do it, here's an article I wrote about it.

Step 8: Update API Gateway

For each API in API Gateway, you'll need to update the routes to point to the newly deployed microservice, instead of to the monolith. First do it on a testing stage, even if you already ran everything in a separate dev environment. Then configure a canary release, and let the microservice gradually take traffic.

You might want to preemptively scale the microservice way beyond the expected capacity requirement. One hour of overprovisioning will cost you a lot less than angry customers.

User Interaction in the Monolith vs in Microservices

Here's the journey for a user viewing a course in our monolith:

1. The user sends a login request with their credentials to the monolithic application.

2. The application validates the credentials and, if valid, generates an authentication token for the user.

3. The user sends a request to view a course, including the authentication token in the request header.

4. The application checks the authentication token and retrieves the course details from the Courses table in DynamoDB.

5. The application retrieves the course content metadata from the Content table in DynamoDB, including the S3 object key.

6. Using the S3 object key, the application generates a pre-signed URL for the course content from Amazon S3.

7. The application responds with the course details and the pre-signed URL for the course content.

8. The user's browser displays the course details and loads the course content using the pre-signed URL.

And here's the same functionality in our microservices:

1. The user sends a login request with their credentials to the authentication service (not covered in the previous microservices example).

2. The authentication service validates the credentials and, if valid, generates an authentication token for the user.

3. The user sends a request to view a course, including the authentication token in the request header, to the Course Catalog microservice through API Gateway.

4. The Course Catalog microservice checks the authentication token and retrieves the course details from its Course Catalog table in DynamoDB.

5. The Course Catalog microservice responds with the course details.

6. The user's browser sends a request to access the course content, including the authentication token in the request header, to the Content Delivery microservice through API Gateway.

7. The Content Delivery microservice checks the authentication token and retrieves the course content metadata from its Content table in DynamoDB, including the S3 object key.

8. Using the S3 object key, the Content Delivery microservice generates a pre-signed URL for the course content from Amazon S3.

9. The Content Delivery microservice responds with the pre-signed URL for the course content.

10. The user's browser displays the course details and loads the course content using the pre-signed URL.

Best Practices for Microservices on AWS

Operational Excellence

• Centralized logging: You're basically running 3 apps. Store the logs in the same place, such as CloudWatch Logs (which ECS automatically configures for you).

• Distributed tracing: These three services don't call each other, but in a real microservices app it's a lot more common for that to happen. In those cases, following the trail of calls becomes rather difficult. Use X-Ray to make it a lot simpler.

Security

• Least privilege: It's not enough to not write the code to access another service's data, you should also enforce it via IAM permissions. Your microservices should each use a different IAM role, that lets each access its own DynamoDB table, not *.

• Networking: If a service doesn't need network visibility, it shouldn't have it. Enforce it with security groups.

• Zero trust: The idea is to not trust agents inside a network, but instead authenticate at every stage. Exposing your services through API Gateway gives you an easy way to do this. Yes, you should do this even when exposing them to other services.

Reliability

• Circuit breakers: User calls Service A, Service A calls Service B, Service B fails, the failure cascades, everything fails, your car is suddenly on fire (just go with it), your boss is suddenly on fire (is that a bad thing?), everything is on fire. Circuit breakers act exactly like the electric versions: They prevent a failure in one component from affecting the whole system. I'll let Fowler explain.

• Consider different scaling speeds: If Service A depends on Service B, consider that Service B scales independently, which could mean that instances of Service B are not started as soon as Service A gets a request. Service B could be implemented in a different platform (EC2 Auto Scaling vs Lambda), which scales at a different speed. Keep that in mind for service dependencies, and decouple the services when you can.

Performance Efficiency

• Scale services independently: Your microservices are so independent that even their databases are independent! You know what that means? You can scale them at will!

• Rightsize ECS tasks: Now that you split your monolith, it's time to check the resource usage of each microservice, and fine-tune them independently.

• Rightsize DynamoDB tables: Same as above, for the database tables.

Cost Optimization

• Optimize capacity: Determine how much capacity each service needs, and optimize for it. Get a savings plan for the baseline capacity.

• Consider different platforms: Different microservices have different needs. A user-facing microservice might need to scale really fast, at the speed of Fargate or Lambda. A service that only processes asynchronous transactions, such as a payments-processing service, probably doesn't need to scale as fast, and can get away with an Auto Scaling Group (which is cheaper per compute time). A batch processing service could even use Spot Instances! Every service is independent, so don't limit yourself.

• Consider the increased management efforts: It's easier (thus cheaper) to manage 10 Lambda functions than to manage 5 Lambda functions, 1 ECS cluster and 2 Auto Scaling Groups.

Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.

• Real scenarios and solutions

• The why behind the solutions

• Best practices to improve them

Subscribe for free

If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com