Troubleshooting S3: Common Issues and Solutions

Introduction to S3 Troubleshooting

Amazon Simple Storage Service (S3) is one of the most widely used object storage services in the cloud, powering everything from static website hosting to data lakes and backup archives. However, despite its reliability and ease of use, developers frequently encounter issues related to permissions, configuration, performance, and data consistency. Understanding how to diagnose and resolve these common problems is an essential skill for any engineer working with AWS.

This tutorial walks through the most frequent S3 issues developers face, explains why they occur, and provides practical solutions with code examples you can apply immediately in your own projects.

Why S3 Troubleshooting Matters

S3 is deceptively simple on the surface — you create a bucket, upload objects, and retrieve them. But beneath that simplicity lies a complex system of access policies, CORS rules, encryption settings, lifecycle configurations, and eventual consistency guarantees. A single misconfigured policy can block an entire application from reading critical files, while a poorly structured key naming scheme can throttle performance at scale.

Effective troubleshooting matters because S3 often sits at the center of application architecture. When S3 fails or behaves unexpectedly, downstream services fail too. Quick diagnosis minimizes downtime, reduces support costs, and prevents data loss. Moreover, many S3 issues are silent — they do not throw errors but instead degrade performance or expose security vulnerabilities over time.

Common S3 Issues and How to Resolve Them

1. Access Denied Errors

The "403 Access Denied" error is arguably the most common S3 problem. It occurs when a user or application attempts an operation without sufficient permissions. The challenge is that multiple layers of policy can block access: IAM user policies, bucket policies, S3 Access Point policies, and bucket ACLs. AWS evaluates all of them, and the most restrictive rule wins.

To diagnose, start by checking the IAM policy attached to the user or role:

aws iam get-user-policy --user-name my-app-user --policy-name S3AccessPolicy

Next, inspect the bucket policy:

aws s3api get-bucket-policy --bucket my-application-bucket

A common mistake is granting s3:GetObject but forgetting s3:ListBucket. Without list permissions, the application cannot enumerate objects, even if individual reads are allowed. Here is a corrected policy that grants both:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-application-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-application-bucket"
    }
  ]
}

If the bucket uses KMS encryption, also verify that the IAM principal has kms:Decrypt and kms:GenerateDataKey permissions on the associated KMS key. Many access denied errors on encrypted buckets trace back to missing KMS permissions rather than S3 itself.

2. CORS Errors in Browser Applications

When a web application tries to fetch objects from S3 using JavaScript, browsers enforce Cross-Origin Resource Sharing (CORS) rules. Without proper CORS configuration, the browser blocks the request and you see errors in the console like "No 'Access-Control-Allow-Origin' header is present."

Configure CORS on the bucket using the following JSON:

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST", "HEAD"],
    "AllowedOrigins": ["https://myapp.example.com"],
    "ExposeHeaders": ["ETag", "x-amz-request-id"],
    "MaxAgeSeconds": 3000
  }
]

Apply it with the AWS CLI:

aws s3api put-bucket-cors --bucket my-application-bucket --cors-configuration file://cors.json

Be specific with AllowedOrigins. Using a wildcard * in production can expose your bucket to unauthorized cross-origin requests, especially if the bucket also allows public reads.

3. Slow Upload and Download Performance

S3 supports very high throughput, but single-threaded uploads of large objects will always be slow. The solution is multipart upload, which splits a large file into parts that are transferred in parallel.

Using the AWS SDK for Python (Boto3), you can enable transfer acceleration and multipart automatically:

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client('s3')

config = TransferConfig(
    multipart_threshold=8 * 1024 * 1024,
    max_concurrency=10,
    multipart_chunksize=8 * 1024 * 1024,
    use_threads=True
)

s3.upload_file(
    'large-file.zip',
    'my-application-bucket',
    'large-file.zip',
    Config=config
)

Another performance pitfall is key naming. S3 partitions the keyspace based on prefix. If all your keys start with the same characters, such as 2024-01-15-log-001, they all land in the same partition and you hit a throughput limit per partition. Instead, add a hash or random prefix:

# Bad: all keys share a prefix
uploads/2024-01-15/file1.jpg
uploads/2024-01-15/file2.jpg

# Better: distribute across prefixes
uploads/7a/2024-01-15/file1.jpg
uploads/3f/2024-01-15/file2.jpg

4. Bucket Not Found or Wrong Region Errors

S3 bucket names are globally unique, but the bucket itself lives in a specific region. If your client is configured for a different region, you may receive a redirect or a "PermanentRedirect" error. Always specify the region explicitly when creating the client:

import boto3

s3 = boto3.client('s3', region_name='us-east-1')

If you are using the REST API directly, include the correct regional endpoint, for example s3.eu-west-1.amazonaws.com instead of the global s3.amazonaws.com.

5. Object Not Found Despite Successful Upload

Developers sometimes upload an object and immediately try to read it, only to get a 404. This can happen when the upload is asynchronous or when a CDN like CloudFront is caching a previous 403 or 404 response. CloudFront caches errors by default for five minutes.

To fix this, configure CloudFront to not cache error responses, or invalidate the cache after upload:

aws cloudfront create-invalidation \
  --distribution-id E1ABC23DEF4GHI \
  --paths "/uploads/new-file.jpg"

For direct S3 access, verify the key name exactly. S3 keys are case-sensitive and include any leading slashes. A key stored as /uploads/file.jpg is different from uploads/file.jpg.

6. Public Access Blocked Despite Public Bucket Policy

AWS introduced S3 Block Public Access at the account and bucket level to prevent accidental data exposure. Even if your bucket policy grants s3:GetObject to everyone, Block Public Access will override it. Check the setting:

aws s3api get-public-access-block --bucket my-application-bucket

If you intentionally need public read access, disable the relevant block settings:

aws s3api put-public-access-block \
  --bucket my-application-bucket \
  --public-access-block-configuration \
  BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false

Only do this when the bucket genuinely needs to be public, such as for static website hosting, and never store sensitive data in the same bucket.

7. Lifecycle Rules Not Transitioning Objects

Lifecycle rules automate moving objects to cheaper storage classes or deleting them after a set time. If objects are not transitioning as expected, check the rule scope and filters. A common mistake is applying a prefix filter that does not match the actual keys.

aws s3api get-bucket-lifecycle-configuration --bucket my-application-bucket

Verify that the rule is enabled and that the prefix matches. Also note that lifecycle transitions can take up to 48 hours to complete after the trigger date, which is normal S3 behavior.

Best Practices for Avoiding S3 Issues

Use the principle of least privilege: Grant only the specific S3 actions and resources an application needs. Avoid wildcard resources unless absolutely necessary.
Enable server access logging or AWS CloudTrail: Logs provide an audit trail that is invaluable when diagnosing permission or access problems after the fact.
Enable versioning on important buckets: Versioning protects against accidental deletes and overwrites. Combine it with MFA delete for critical data.
Use S3 Storage Lens: This built-in dashboard reveals usage patterns, performance metrics, and potential cost optimizations across your buckets.
Test policies with the IAM Policy Simulator: Before deploying a new policy, simulate it against expected actions to catch permission gaps early.
Automate infrastructure with CloudFormation or Terraform: Manual configuration drift is a leading cause of S3 misconfigurations. Infrastructure as code makes changes reviewable and repeatable.
Monitor with CloudWatch alarms: Set alarms on 4xx and 5xx error rates to catch issues before users report them.
Use presigned URLs for temporary access: Instead of making objects public or sharing long-lived credentials, generate presigned URLs that expire after a short window.

Here is an example of generating a presigned URL in Python:

import boto3

s3 = boto3.client('s3', region_name='us-east-1')

url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-application-bucket', 'Key': 'report.pdf'},
    ExpiresIn=3600
)

print(url)

Conclusion

Troubleshooting S3 effectively requires understanding the layered permission model, the eventual consistency behavior, and the performance characteristics of the service. Most issues fall into a handful of categories — access denied, CORS, performance, region mismatch, and lifecycle misconfiguration — and each has a reliable diagnostic path. By combining the CLI commands and SDK examples in this tutorial with the best practices of least privilege, logging, and infrastructure as code, you can resolve S3 problems quickly and prevent them from recurring. As your usage of S3 grows, invest in monitoring and automation early; the cost of setting them up is far lower than the cost of debugging a production outage under pressure.