Search for a command to run...
Hi! I'm Guille Ojeda, Cloud Architecture Consultant To put it simply, I create simple solutions for complex technical problems.
My focus is on designing scalable, secure, and cost-effective cloud solutions that startups, scaleups and SMBs can lean on to grow.
I lean on years of experience in cloud architecture and consulting to deliver solutions that are easy to understand and manage.
No comments yet. Be the first to comment.
What Are Agentic AI Architectures? I won't waste your time with a long, fluffy introduction about how AI is changing the world. Let's get straight to the point: agentic AI architectures are fundamentally different from the prompt-response pattern you...
Building distributed systems requires putting a lot of attention on communication between components. These components often need to exchange information asynchronously, and that's where message queues and pub/sub systems are the go-to solution. AWS ...
Database failures are inevitable. Even with the most reliable hardware and software, something will eventually break. AWS RDS Multi-AZ deployments promise to handle these failures gracefully, automatically failing over to a standby database when prob...
There are two managed relational database services in AWS: Amazon Relational Database Service (RDS) and Amazon Aurora. Both provide the benefits of a fully managed database solution, but they have distinct features and use cases. In this article, we'...
Containers offer a lightweight, portable, and scalable solution for running software consistently across different environments. But as the number of containers grows, managing them becomes increasingly complex. That's where container orchestration c...
Note: This content was originally published at the Simple AWS newsletter.
Imagine this scenario: You're careful with security, and you set up Multi-Factor Authentication for your AWS IAM or IAM Identity Center user. At one point, a malicious agent of evilness figures out your password, either via phishing, keylogging, or any other technique. The only thing standing between them and $50k in bitcoin mined on your AWS account (and the $500k AWS bill paid with your credit card) is your MFA device. They don't have access it (yet!), so you're safe (for now!).
The correct response is obvious: Change your password, so they'll be back to square one. The problem with this is that most of the times you don't know that a password has become known until someone uses it. This is the reason why we should rotate passwords regularly!
Let me propose an extra layer of security: Get notified every time a login attempt fails because of a failed MFA check.
Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.
AWS CloudTrail is a service that logs all requests performed against AWS in your account. This includes all actions and requests (even unauthorized ones) done through the Console, CLI, AWS SDKs, and APIs.
By default (already enabled when you create your account) and for free, CloudTrail offers you a viewable, searchable, downloadable, and immutable record of all events that happened in the past 90 days, called Event History.
You can also create Trails, which let you export to S3 or CloudWatch Logs all or a subset of CloudTrail events. This way, you can use other AWS services like Athena and OpenSearch, or external tools like ElasticSearch, to analyze CloudTrail events. Event history is per region, while Trails can be created for all regions in an account, and even for all accounts in an Organization.
Events are JSON objects, which contain all the information of what was attempted, when, by whom, some other details on the request, what was the result, and sometimes some other details on the response, such as the reason.
For example, this is what a ConsoleLogin event that fails looks like (I hid a few details replacing them with HIDDEN):
{
"eventVersion": "1.08",
"userIdentity": {
"type": "IAMUser",
"principalId": "HIDDEN",
"accountId": "HIDDEN",
"accessKeyId": "HIDDEN",
"userName": "HIDDEN"
},
"eventTime": "2023-08-24T21:07:08Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "HIDDEN",
"userAgent": "HIDDEN",
"errorMessage": "Failed authentication",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Failure"
},
"additionalEventData": {
"LoginTo": "https://console.aws.amazon.com/console/home?HIDDEN",
"MobileVersion": "No",
"MFAUsed": "Yes"
},
"eventID": "HIDDEN",
"readOnly": false,
"eventType": "AwsConsoleSignIn",
"managementEvent": true,
"recipientAccountId": "HIDDEN",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.3",
"cipherSuite": "TLS_AES_128_GCM_SHA256",
"clientProvidedHostHeader": "signin.aws.amazon.com"
}
}
Naturally, CloudTrail logs events when an IAM or IAM IC user logs in to AWS. However, it's not just one event. Let me show you what happens behind the scenes with IAM IC and IAM.
These are the events that CloudTrail logs when an IAM or IAM IC user logs in:
CredentialChallenge: AWS requested some form of credential, such as password or MFA device. Each of these is followed by UserAuthentication and one CredentialVerification event, and this sequence of three is repeated until either all necessary credentials are provided, or CredentialVerification fails.
UserAuthentication: AWS receives the requested credentials.
CredentialVerification: AWS checks whether the received credentials are valid. If they are, this event contains: "serviceEventDetails":{ "CredentialChallenge":"Success" }, and the process continues either by requesting the next credential or by authenticating. If the credentials received are invalid, this event contains "serviceEventDetails":{ "CredentialChallenge":"Failure" } and the process stops. The type of credential requested can be found in "additionalEventData":{ "CredentialType":"PASSWORD" }, the value of which can be PASSWORD for the regular password, TOTP for MFA devices that produce temporary codes, WEBAUTHN for web apps using WebAuthn, EXTERNAL_IDP for external identity providers, or RESYNC_TOTP to re-synchronize TOTP devices.
Authenticate: Once CredentialVerification succeeds and no more credentials are required, this event is logged. This means the user successfully authenticated to IAM Identity Center. If you're authenticating with an external Identity Provider such as Google Workspaces, Microsoft AD or Okta, this is the only event you'll see.
Federate: This event means the IAM IC user assumed an IAM role in an AWS account.
ConsoleLogin: This event means the IAM IC user logged in to the AWS Console using the assumed IAM role.
The first four events will be logged on the AWS account and region where IAM IC is configured. The last two are logged in the AWS account where the user signs in, in the default region for that user (i.e. the one that's selected when the user signs in).
IAM is much simpler. The only event is ConsoleLogin, and this is how you can identify what happened:
If the user signed in successfully, the event contains "responseElements": { "ConsoleLogin": "Success" }
If sign in failed, the event contains "responseElements": { "ConsoleLogin": "Failure" } and can contain "errorMessage": "Failed authentication"
If the user used MFA (regardless of success or failure), the event contains "additionalEventData": { "MFAUsed": "Yes" }
If the user is the root user, the event contains "userIdentity": { "type": "Root" }
All of these events are always logged in the us-east-1 region.
The last section was a bit detailed, but it contains all the information we need to identify when a login attempt fails the MFA check!
For IAM IC, we need to find CredentialVerification events which contain either "additionalEventData":{ "CredentialType":"TOTP" } or "additionalEventData":{ "CredentialType":"WEBAUTHN" }, and contain "serviceEventDetails":{ "CredentialChallenge":"Failure" }.
For IAM, we need to find ConsoleLogin events which contain "responseElements": { "ConsoleLogin": "Failure" } and "additionalEventData": { "MFAUsed": "Yes" }.
You can do this manually in the CloudTrail event history, but if we want to automate it and get notifications, we need to send it to CloudWatch Logs.
Sign in to the AWS Console. For IAM IC, sign in to the root account of the Organization
Go to CloudTrail
Click Create trail
For Trail name, enter management-events
If you're configuring this for an organization, check Enable for all accounts in my organization
Select Create new S3 bucket and under Trail log bucket and folder enter a name, or leave it as default
Under AWS KMS alias, enter a name for a KMS key to encrypt the logs
Under CloudWatch Logs, check Enabled
Under Log group name, enter a name for the CloudWatch Logs log group
Under Role name, enter a name for the IAM Role that'll let CloudTrail put logs to CloudWatch Logs
Click Next
Leave these options as default and click Next again
Click Create trail
Wait until the Status column changes to Logging (green)
First, let's create a filter to view the failed logins in CloudWatch Logs. Follow the steps, and on Step 6 choose the right filter depending on whether you're using IAM or IAM IC.
Open the CloudWatch console
In the panel on the left, under Logs, click Log groups.
Click on the name of the log group that you created for the trail
Click the Metrics filters tab, and click the button Create metric filter
Under Filter pattern, enter the pattern that corresponds to IAM or IAM IC, depending on which you're using:
For IAM: { ($.eventName = ConsoleLogin) && ($.additionalEventData.MFAUsed = "Yes") && ($.responseElements.ConsoleLogin = "Failure") }
For IAM IC: { ($.eventName = CredentialVerification) && (($.additionalEventData.CredentialType = "TOTP") || ($.additionalEventData.CredentialType = "WEBAUTHN")) && ($.serviceEventDetails.CredentialChallenge = "Failure")}
Click Next
For Filter name, enter SignInFailedMFA
Under Metric namespace, enter CloudTrailMetrics
For Metric name, enter SigninMFAFailureCount
For Metric value, enter 1.
Click Next
Click Create metric filter
On the Metric filters tab, find the metric filter you just created, select it and click Create alarm
Under Whenever SigninMFAFailureCount is..., select Greater/Equal
Under than…, enter 1
Click Next
Under Send a notification to the following SNS topic, select Create new topic
Under Create a new topic…, enter SignInFailedMFAAlarm
Under Email endpoints that will receive the notification…, enter your email address
Click the Create topic button that's right below where you just entered your email address
Click Next
Under Alarm name, enter SignInFailedMFAAlarm
Click Next
Click Create alarm
Open your email inbox, open the email titled AWS Notification - Subscription Confirmation, and click Confirm subscription
To test it, you'll need to attempt to sign in, and enter the incorrect MFA code. There's going to be a delay of a few minutes between when the event actually occurs and when you receive the notification. CloudTrail usually takes an average of 5 minutes to export the event logs to CloudWatch Logs, and the alarm could take a couple more minutes to fire.
You can also monitor the alarm in CloudWatch Alarms:
Stop copying cloud solutions, start understanding them. Join over 45,000 devs, tech leads, and experts learning how to architect cloud solutions, not pass exams, with the Simple AWS newsletter.
Real scenarios and solutions
The why behind the solutions
Best practices to improve them
If you'd like to know more about me, you can find me on LinkedIn or at www.guilleojeda.com