Introduction to IAM Troubleshooting
Identity and Access Management (IAM) is the backbone of cloud security, controlling who is authenticated and authorized to access resources. Whether you're working with AWS IAM, Azure RBAC, or Google Cloud IAM, misconfigurations can lead to broken applications, security vulnerabilities, and frustrated developers. This tutorial walks you through the most common IAM issues and provides practical solutions to resolve them quickly.
What Is IAM?
IAM is a framework of policies, roles, and permissions that governs access to cloud resources. It ensures that users and services can only perform actions they are explicitly allowed to perform. IAM systems typically use a combination of authentication (verifying identity) and authorization (granting permissions) to enforce security boundaries.
Why IAM Troubleshooting Matters
A misconfigured IAM policy can cause application outages, data breaches, and compliance violations. According to industry reports, over 60% of cloud security incidents stem from misconfigured permissions rather than sophisticated attacks. Understanding how to diagnose and fix IAM issues is an essential skill for any developer or DevOps engineer working in the cloud.
Common IAM Issues and How to Diagnose Them
1. Access Denied Errors
The most frequent IAM problem is the dreaded "Access Denied" error. This occurs when a user or service attempts an action they are not authorized to perform. The challenge is identifying which specific permission is missing.
Symptoms:
- API calls return 403 Forbidden or AccessDenied errors
- Applications fail to read or write to resources like S3 buckets or databases
- CLI commands fail with permission errors
Diagnosis with AWS IAM Policy Simulator:
# Use the AWS CLI to simulate a policy evaluation
aws iam simulate-principal-policy \
--policy-source-arn arn:aws:iam::123456789012:user/dev-user \
--action-names s3:GetObject \
--resource-arns arn:aws:s3:::my-bucket/* \
--output table
The simulator will tell you whether the action is allowed or denied, and which policy statement caused the decision. This eliminates guesswork when debugging permissions.
Using CloudTrail to Trace Denied Requests:
# Query CloudTrail for denied events in the last 24 hours
aws logs filter-log-events \
--log-group-name CloudTrail/DefaultLogGroup \
--filter-pattern '{ ($.errorCode = "AccessDenied") }' \
--start-time $(date -d '1 day ago' +%s)000 \
--end-time $(date +%s)000 \
--output json
2. Implicit Deny and Policy Conflicts
IAM follows a default-deny model: if no policy explicitly allows an action, it is denied. Additionally, an explicit deny in any policy always overrides an allow. This can create confusing situations where a user has an allow policy but still cannot access a resource.
Example of a conflicting policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-bucket/*"
},
{
"Effect": "Deny",
"Action": "s3:DeleteObject",
"Resource": "arn:aws:s3:::my-bucket/*",
"Condition": {
"StringNotEquals": {
"aws:PrincipalTag/Department": "Finance"
}
}
}
]
}
In this example, even though the first statement allows all S3 actions, the second statement denies s3:DeleteObject unless the principal has a Department tag set to Finance. If a user without that tag tries to delete an object, they will receive an Access Denied error despite the broad allow statement.
Solution: Review all attached policies, including inline policies, managed policies, and permission boundaries. Use the following command to list all policies attached to a role:
# List all managed policies attached to a role
aws iam list-attached-role-policies --role-name MyRole
# List inline policies embedded in a role
aws iam list-role-policies --role-name MyRole
# Retrieve a specific inline policy document
aws iam get-role-policy \
--role-name MyRole \
--policy-name MyInlinePolicy
3. Trust Relationship Issues with Roles
Roles rely on trust policies to define which principals can assume them. A common issue occurs when the trust policy does not include the correct principal or conditions, preventing role assumption.
Example of a broken trust policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
This trust policy only allows the EC2 service to assume the role. If you try to assume this role from a Lambda function or another AWS account, it will fail. To fix this, update the trust policy to include the necessary principals:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": ["ec2.amazonaws.com", "lambda.amazonaws.com"],
"AWS": "arn:aws:iam::987654321098:root"
},
"Action": "sts:AssumeRole"
}
]
}
Testing role assumption:
# Attempt to assume a role and capture any errors
aws sts assume-role \
--role-arn arn:aws:iam::123456789012:role/MyRole \
--role-session-name TestSession \
--duration-seconds 3600
4. Permission Boundaries Blocking Access
Permission boundaries are advanced IAM features that set the maximum permissions a user or role can have. Even if an identity-based policy grants broad access, a permission boundary can restrict it. This often confuses developers who see an allow policy but still get denied.
Checking for permission boundaries:
# Check if a role has a permissions boundary attached
aws iam get-role --role-name MyRole --query 'Role.PermissionsBoundary'
# Check if a user has a permissions boundary attached
aws iam get-user --user-name MyUser --query 'User.PermissionsBoundary'
If a permissions boundary ARN is returned, retrieve and review its policy document to understand the restrictions:
# Get the permissions boundary policy document
aws iam get-policy \
--policy-arn arn:aws:iam::123456789012:policy/MyBoundaryPolicy
aws iam get-policy-version \
--policy-arn arn:aws:iam::123456789012:policy/MyBoundaryPolicy \
--version-id v1
5. Service Control Policies (SCPs) in AWS Organizations
If your account is part of an AWS Organization, Service Control Policies (SCPs) can restrict what actions are allowed at the organizational level. SCPs act as guardrails and can override individual account permissions. A developer might have full admin access in an account but still be unable to perform certain actions due to an SCP.
Diagnosing SCP issues:
# List SCPs attached to an organizational unit
aws organizations list-policies-for-target \
--target-id ou-abcd-12345678 \
--filter SERVICE_CONTROL_POLICY
# Retrieve the SCP content
aws organizations describe-policy --policy-id p-1234567890
Review the SCP for any deny statements that might be blocking the desired action. Note that SCPs do not grant permissions; they only set the maximum available permissions. You still need identity-based policies to actually grant access.
6. Temporary Credentials Expiration
When using IAM roles with temporary credentials, the credentials expire after a set duration (typically 1 to 12 hours). Applications that cache credentials without refreshing them will eventually fail with authentication errors.
Handling credential refresh in Python with Boto3:
import boto3
from botocore.session import Session
# Use AssumeRoleProvider for automatic credential refresh
session = Session()
session.set_config_variable('region', 'us-east-1')
# Create a client with automatic credential refresh
sts = session.create_client('sts')
response = sts.assume_role(
RoleArn='arn:aws:iam::123456789012:role/MyRole',
RoleSessionName='MyAppSession',
DurationSeconds=3600
)
# Use the temporary credentials
credentials = response['Credentials']
print(f"Credentials expire at: {credentials['Expiration']}")
For production applications, use the AWS SDK's built-in credential provider chain, which automatically handles refresh logic. Avoid hardcoding temporary credentials in configuration files.
7. Resource-Based Policy Issues
Some AWS services, like S3, KMS, and SQS, support resource-based policies attached directly to the resource. These policies work in conjunction with identity-based policies. For most services, both the identity-based policy and the resource-based policy must allow the action. However, for S3, a resource-based policy alone can grant access to anonymous users.
Checking an S3 bucket policy:
# Retrieve the bucket policy
aws s3api get-bucket-policy --bucket my-bucket --output json
# Common issue: bucket policy denies access from outside a specific VPC
# Example restrictive bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-bucket/*",
"Condition": {
"StringNotEquals": {
"aws:sourceVpce": "vpce-1234567890abcdef"
}
}
}
]
}
If your application runs outside the specified VPC, it will be denied access regardless of IAM permissions. Ensure that your resource-based policies align with your network architecture.
Best Practices for IAM Troubleshooting
Follow the Principle of Least Privilege
Grant only the permissions required to perform a task. Start with a minimal policy and add permissions as needed. This reduces the attack surface and makes troubleshooting easier because there are fewer policies to review.
Use IAM Access Analyzer
AWS IAM Access Analyzer identifies resources in your organization and accounts that are shared with external entities. It helps you detect unintended access and validate policies before deployment.
# Create an analyzer
aws accessanalyzer create-analyzer \
--analyzer-name MyAnalyzer \
--type ACCOUNT
# List findings
aws accessanalyzer list-findings \
--analyzer-arn arn:aws:access-analyzer:us-east-1:123456789012:analyzer/MyAnalyzer
Validate Policies Before Applying
Always validate IAM policies using the IAM policy validator or Access Analyzer policy validation before applying them. This catches syntax errors and identifies security issues proactively.
# Validate a policy using Access Analyzer
aws accessanalyzer validate-policy \
--policy-type IDENTITY_POLICY \
--policy-document file://my-policy.json \
--output json
Implement Tag-Based Access Control
Tags provide a scalable way to manage permissions. Use condition keys like aws:ResourceTag and aws:PrincipalTag to create flexible policies that adapt to your resource organization.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["ec2:StartInstances", "ec2:StopInstances"],
"Resource": "*",
"Condition": {
"StringEquals": {
"aws:ResourceTag/Environment": "${aws:PrincipalTag/Environment}"
}
}
}
]
}
Enable Detailed Logging
CloudTrail and CloudWatch logs are essential for diagnosing IAM issues. Ensure that CloudTrail is enabled in all regions and that management events are logged. Set up CloudWatch alarms for anomalous API activity to catch issues early.
# Create a CloudWatch alarm for AccessDenied events
aws cloudwatch put-metric-alarm \
--alarm-name IAMAccessDenied \
--alarm-description "Alert on AccessDenied events" \
--metric-name AccessDeniedCount \
--namespace CloudTrailMetrics \
--statistic Sum \
--period 300 \
--threshold 5 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 1 \
--alarm-actions arn:aws:sns:us-east-1:123456789012:MyAlertTopic
Use Named Profiles for Testing
When troubleshooting, use named AWS CLI profiles to test permissions as different users or roles. This helps you reproduce issues and verify fixes without modifying your primary credentials.
# Configure a named profile
aws configure set region us-east-1 --profile test-user
aws configure set aws_access_key_id AKIAIOSFODNN7EXAMPLE --profile test-user
aws configure set aws_secret_access_key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --profile test-user
# Test an action as the test user
aws s3 ls s3://my-bucket/ --profile test-user
Conclusion
Troubleshooting IAM issues requires a systematic approach: identify the failing action, trace the request through CloudTrail, evaluate all applicable policies (identity-based, resource-based, permission boundaries, and SCPs), and use tools like the IAM Policy Simulator and Access Analyzer to pinpoint the root cause. By following the principle of least privilege, implementing proper logging, and validating policies before deployment, you can minimize IAM-related issues and maintain a secure, reliable cloud environment. Remember that IAM is not a set-it-and-forget-it configuration — it requires ongoing review and refinement as your infrastructure and team evolve.