Python FastAPI Tutorial (Part 16): AWS S3 and Boto3 - Moving File Uploads to the Cloud
TL;DR
This tutorial demonstrates how to migrate a FastAPI application from local disk storage to AWS S3 for production file uploads, covering S3 bucket setup, IAM security configuration, and Boto3 integration while maintaining separation between image processing and storage layers.
🏗️ Production Storage Architecture 3 insights
Local storage fails in containerized environments
Container filesystems are ephemeral, meaning uploaded files disappear on app restart or redeployment, and local files cannot be easily shared across multiple server instances.
Object storage provides durability and scale
AWS S3 stores files independently from application servers, ensuring persistence during deployments and enabling horizontal scaling without file synchronization issues.
Separate processing from storage concerns
Refactor the application so Pillow handles image processing (resizing, format conversion) while a distinct function manages S3 upload, allowing the storage layer to change without affecting image logic.
⚙️ S3 Bucket Configuration 3 insights
Bucket naming follows strict global rules
Names must be globally unique across all AWS, lowercase only, with no underscores; users can choose between global namespace or the newer account-regional option.
Disable ACLs in favor of bucket policies
Modern S3 setups use 'Bucket owner enforced' with ACLs disabled, managing access exclusively through IAM and bucket policies rather than legacy access control lists.
Scoped public access for image serving
Disable 'block all public access' to allow browsers to load images, but restrict the bucket policy to only allow s3:GetObject on a specific prefix like 'profile-pics/*' rather than the entire bucket.
🔐 IAM Security & Credentials 3 insights
Apply least privilege to upload permissions
Create an IAM policy granting only PutObject and DeleteObject actions on the specific bucket prefix, deliberately excluding GetObject since the application generates URLs rather than downloading files.
Secure access keys immediately upon creation
AWS displays the secret access key only once during creation; copy both the key ID and secret immediately to environment variables and ensure .env files are in .gitignore to prevent credential leaks.
Support multiple deployment patterns
Make access key settings optional in the application configuration to support both explicit credentials (for local development or VPS hosting) and IAM roles (for AWS EC2/ECS where Boto3 picks up temporary credentials automatically).
💻 Application Integration 3 insights
Install Boto3 as the standard AWS SDK
Add the official Boto3 library to interact with S3, which serves as the industry standard for Python AWS integration across all services.
Configure flexible S3 settings
Add environment variables for bucket name, region, optional credentials, and optional endpoint URL to enable testing with local S3-compatible services like MinIO without code changes.
Modify image utilities for cloud upload
Update the process_profile_image function to return image bytes instead of saving to disk, then pass those bytes to a new S3 upload function that uses Boto3 to stream the file to cloud storage.
Bottom Line
Move production file uploads to AWS S3 instead of local storage, using IAM policies with minimal required permissions (PutObject and DeleteObject only) and never commit AWS credentials to version control.
More from Corey Schafer
View all
Python FastAPI Tutorial (Part 18): Deploy to a VPS - Security, Nginx, SSL, and Custom Domain
Corey Schafer demonstrates how to deploy a production-ready FastAPI application to a Virtual Private Server (VPS), emphasizing fundamental deployment concepts including security hardening, SSH key authentication, and health check implementation before moving to managed cloud solutions.
Python FastAPI Tutorial (Part 17): Testing the API - Pytest, Fixtures, and Mocking External Services
This tutorial demonstrates how to implement comprehensive testing for FastAPI applications using pytest with async support, covering critical setup patterns like environment variable configuration before app imports, using AsyncClient for async endpoints, mocking AWS S3 with Moto, and maintaining a separate PostgreSQL test database to ensure production parity.
Python FastAPI Tutorial (Part 15): PostgreSQL and Alembic - Database Migrations for Production
This tutorial transitions a FastAPI application from SQLite to PostgreSQL for production readiness, implementing Alembic for database migrations to enable safe, version-controlled schema changes without data loss.
Python FastAPI Tutorial (Part 14): Password Reset - Email, Tokens, and Background Tasks
This tutorial demonstrates implementing a secure password reset flow in FastAPI using cryptographically secure hashed tokens stored in a database, asynchronous email sending via aiostmplib to prevent blocking, and Jinja2 templates for HTML emails, following security best practices like one-hour expiration and single-use validation.