Skip to main content
object-storage

Object Storage (S3, GCS, Azure Blob)

Durable, cheap, flat-namespace storage for blobs — images, videos, backups, logs, and any large binary object.

Why it exists

Storing large blobs (>1MB — images, videos, PDFs, backups) in a database is a bad idea: databases are optimized for structured queries and page-cache-fitting rows, not for streaming megabytes. Object storage is purpose-built for exactly this: cheap disk-backed storage with an HTTP API, eleven 9s of durability (S3 default), and a flat namespace (no filesystem hierarchy). It's the default answer for 'where do I store user uploads?'

How it works

Objects are stored as key → blob + metadata in buckets. There's no directory structure — 'folders' are just prefixes in the key (e.g., users/123/avatar.jpg). Data is replicated across ≥3 availability zones. Access is via HTTP GET/PUT/DELETE. Consistency is strong per-object (S3 as of 2020) — a PUT is visible to subsequent GETs immediately. There's no partial updates: you always overwrite the whole object or upload a new version.

Scaling characteristics

Effectively infinite storage and throughput per bucket (S3 auto-partitions internally). Practical limits: ~5,500 GETs/sec/prefix and ~3,500 PUTs/sec/prefix before AWS starts throttling — mitigate with random prefixes for high-write workloads. Latency is 20-200ms per request depending on size. Bandwidth is cheap; egress across regions is expensive.

When to use it

  • User uploads (photos, videos, documents)
  • Static asset hosting (with CDN in front)
  • Backup and archive (S3 Glacier for cold data)
  • Big data lakes (Parquet files, Kafka topic dumps)
  • Serverless / stateless architectures — objects are the durable state, functions are ephemeral
  • Machine-learning datasets and model artifacts

When NOT to use it

  • Sub-100ms interactive reads for small objects — Redis or a DB is faster
  • Small (KB-scale) rows with structured queries — SQL/NoSQL wins
  • Datasets with heavy random-access patterns on individual bytes within a file — object storage is optimized for whole-object read/write
  • Extreme write latency requirements — object storage adds 20-100ms per write, which is a lot for real-time paths

Failure modes

  • Bucket misconfiguration exposes data publicly — S3 default is private, but IAM policies can leak. Mitigate with default-deny + linters + org SCP.
  • Hot prefix throttling: too many requests hit one auto-partition. Mitigate with random-hash prefixes.
  • Cost surprise: cross-region egress or Glacier retrieval bills you didn't plan for. Set billing alarms.
  • Multi-object updates aren't atomic. If you write image + metadata separately, they can diverge on failure.
  • Data corruption on partial write is not automatically detected — always verify ETag / checksum.

Alternatives

  • Filesystem (NFS, EFS) — POSIX semantics, works for legacy apps; expensive at scale
  • Block storage (EBS) — for databases and stateful workloads that need low-latency random IO
  • Database BLOB columns — works for <10MB files; usually a mistake at scale
  • Peer-to-peer / IPFS — for content-addressed storage; niche use cases

Interview questions

  • How do you design a file-upload service? Where does the file live? What's the write flow?
  • You need to serve 1TB/sec of video. Where does the video live, and how do you deliver it?
  • How would you migrate from local filesystem to S3 with zero downtime?
  • Explain multi-part upload. When do you use it, and what's the failure story?
  • How do you prevent your app from becoming a proxy for object downloads? (Hint: presigned URLs)
  • You have 100M objects in one prefix. Reads are getting throttled. What's the fix?

Systems that use this component

See how the real designs on this platform put object-storage to work — concrete usage context per system.

YouTube

intermediate

S3 for uploaded videos; Glacier for cold archive

Open system

Dropbox

intermediate

Magic Pocket (custom S3 replacement) at exabyte scale

Open system

Instagram

intermediate

Photos in S3; hot photos on CDN

Open system

Netflix

intermediate

Master files in S3; encoded variants in Open Connect

Open system

URL Shortener

beginner

Backup snapshots + CSV analytics exports

Open system
The story of infinite storage

2006, Amazon: a service so simple it made the cloud possible

In March 2006, AWS launched S3 before EC2, before any other significant service. That wasn't an accident. Amazon knew that if they could give developers infinite, cheap, reliable storage, everything else would follow. A single API: PUT, GET, DELETE. Files (called "objects") up to 5TB each. Priced at pennies per GB. Eleven 9's of durability. No pre-provisioning, no capacity planning, no filesystem to manage.

S3 broke the traditional storage mental model. Where NAS/SAN required you to pre-allocate volumes, worry about inodes, and manage RAID, S3 offered a flat key-value namespace — buckets and keys. Where filesystems limited you to millions of files, S3 supported trillions. And where the durability of your homelab NAS depended on you noticing a drive failure, S3 asynchronously replicated to multiple AZs and self-healed silently.

The pattern replicated everywhere. Google Cloud Storage (2010) and Azure Blob Storage (2010) offered near-identical APIs. Then in 2020, Cloudflare R2 attacked S3's biggest cost — egress fees — with a $0 egress model. Backblaze B2 undercut on price. MinIO gave the enterprise self-hosted S3 semantics. Today, the S3 API is a de facto standard. Thousands of tools speak "S3" without caring which vendor is behind it.

The core insight: object storage isn't "disk for the cloud." It's a fundamentally different abstraction — immutable objects addressed by key, with eventual metadata propagation, backed by erasure coding. You don't open, seek, and write. You PUT a whole object, then GET it back. This constraint is what makes 11 nines of durability + effectively-unlimited scale economically viable. Give up random-access mutation, gain planet-scale reliability.

Object storage in one page
Data model
Flat namespace: bucket + key → object (blob + metadata).
Durability
11 nines (99.999999999%) — via erasure coding + geo-replication.
Scale
Trillions of objects. Petabytes-to-exabytes per bucket.
API
PUT, GET, DELETE, LIST, HEAD. Not open/read/seek/write.
Rule: anything write-once-read-many belongs in object storage. Anything read-modify-write belongs in a database.

Historical timeline

  1. 1990s
    SAN/NAS dominate
    Enterprise storage means EMC, NetApp, Hitachi. Expensive. Fragile. Requires storage admins.
  2. 2001
    First cloud NAS attempts
    IBM Cloud Storage, various failed attempts. Model doesn't match cloud economics.
  3. 2003
    Amazon Dynamo internal
    Amazon builds Dynamo for shopping cart — proves KV works at scale. Later inspires S3 model.
  4. 2006
    AWS S3 launches
    March 14, 2006. Simple API. 15GB per object initially. Prices start at $0.15/GB. Cloud storage as a concept is born.
  5. 2010
    Google Cloud Storage GA
    Google offers direct competitor. Uses Google's Colossus file system underneath.
  6. 2010
    Azure Blob Storage
    Microsoft joins with block, page, and append blob types.
  7. 2011
    OpenStack Swift released
    First widely-used open-source object store. Rackspace-led. Foundation for private clouds.
  8. 2014
    Ceph reaches production maturity
    Multi-purpose OSS storage: object, block, file. Powers many private clouds.
  9. 2015
    MinIO founded
    Anand Babu Periasamy builds S3-compatible object store for on-prem. Popular for k8s + private clouds.
  10. 2018
    S3 achieves strong read-after-write
    AWS quietly removes the last consistency caveat. All operations now strongly consistent.
  11. 2018
    Backblaze B2 gains traction
    Cheap alternative to S3. $0.005/GB. Powers many indie devs and backup workloads.
  12. 2020
    S3 Intelligent-Tiering matures
    Auto-move cold objects to cheaper tiers. Removes the manual lifecycle-policy burden.
  13. 2021
    Cloudflare R2 announced
    S3-compatible + zero egress fees. Attacks S3's biggest revenue moat. Ships GA 2022.
  14. 2022
    S3 introduces Express One Zone
    Single-AZ ultra-low-latency tier. 10x lower latency than S3 Standard. Aimed at AI training.
  15. 2024
    AI reshapes object storage
    Training on S3 becomes standard. Iceberg + Delta Lake + Parquet on S3 becomes THE data lake pattern.

Inside S3: erasure coding + metadata index

The magic of object storage is that a single PUT survives 2+ disk failures with no visible impact. The mechanism is erasure coding: chop the object into K data shards + M parity shards, spread across K+M drives. Lose any M and reconstruct from the rest. Combined with cross-AZ placement, you get 11 nines.

Client side
PUT bucket/key.jpg (10MB)
↓
API frontend
Auth (SigV4) → compute object key hash → route to placement service
↓
Erasure coding + placement
Split 10MB → 6 data shards (1.67MB each) + 3 parity shards = 9 shards total
Place shards on 9 different drives across 3 AZs
Any 3 shards can be lost → object still recoverable
↓
Metadata index
bucket + key → shard locations, checksum, size, ACL, encryption key ID
The metadata service is the single most important + smallest layer. Highly replicated. LSM-tree backed.
Erasure coding vs replication: 6+3 EC gives 3-drive fault tolerance at ~1.5x storage overhead. Full replication would be 3x. EC wins on cost.
Self-healing: S3 constantly reads sample objects to detect bit rot. When it finds a bad shard, it regenerates from surviving shards. You never notice.
The bottleneck is metadata: object data is trivial to shard. The index of "where is bucket/key?" must be strongly consistent + hyper-scalable. That's the hard part.

Storage classes: pay for the access pattern, not the bytes

S3's biggest insight after the initial "infinite storage" play was tiered pricing. Hot data pays more per GB but retrieval is free. Cold data is 20x cheaper per GB but retrieval costs money. Get this right and you save 80% of storage bill:

ClassCost/moAccessRetrieval feeUse for
S3 Standard$0.023/GB/moImmediateFreeHot data. Default for anything you touch daily.
S3 Intelligent-Tiering$0.023 (hot) → $0.0125 (cool) → $0.004 (archive) autoImmediate for hot, minutes for archiveFree (monitoring fee)Unknown access patterns. AWS moves objects automatically.
S3 Standard-IA$0.0125/GB/moImmediate$0.01/GB (per-retrieval)Infrequently accessed but still needed sometimes. Backups, log archives.
S3 One Zone-IA$0.01/GB/moImmediate$0.01/GBReproducible data. If AZ dies, you regenerate from source.
S3 Glacier Instant Retrieval$0.004/GB/moImmediate$0.03/GBLong-term archive with occasional need for instant access. Medical imaging, media libraries.
S3 Glacier Flexible Retrieval$0.0036/GB/moMinutes-hours (Expedited/Standard/Bulk)$0.01/GB (Bulk) — $0.10/GB (Expedited)Compliance archives, backups you rarely touch.
S3 Glacier Deep Archive$0.00099/GB/mo12 hours$0.02/GB7+ year retention. Tape-replacement. Financial records, legal holds.
S3 Express One Zone$0.16/GB/mo<10ms P50FreeAI training checkpoints, ML data loaders. Where every ms matters.
The gotcha: minimum storage duration. Glacier has 90-day min. Deep Archive has 180-day min. Delete before that = still charged. Always lifecycle-transition, don't delete-then-recreate.

Multipart upload: how to reliably PUT a 5TB object

Sending a 5TB blob in one HTTP PUT would take hours and any network blip kills it. S3's answer is multipart upload: chop the object into parts, upload each independently (in parallel), then commit atomically. Retry only failed parts.

1. Initiate
CreateMultipartUpload → uploadId
2. Upload parts (parallel!)
UploadPart(1, bytes 0-5MB) → ETag1
UploadPart(2, bytes 5-10MB) → ETag2 (parallel)
UploadPart(3, bytes 10-15MB) → ETag3 (parallel)
...up to 10,000 parts, 5MB-5GB each...
3. Complete
CompleteMultipartUpload(uploadId, [ETag1, ETag2, ETag3, ...])
S3 atomically assembles the parts. Object appears in single-request-consistent form.
Parallelism boost: 100 parallel part uploads = ~100x faster than serial. Use SDK's TransferManager or aws s3 cp.
Retry safety: if part 5 fails, retry just part 5 with same part number. Other parts unaffected.
Cost gotcha: incomplete uploads still store the parts (billed!). Set a bucket lifecycle rule: "abort multipart uploads older than 7 days."

Consistency: from eventual to strong (2018 upgrade)

Original S3 (2006-2020) was eventually consistent for overwrites and DELETEs — a GET immediately after PUT might return the old object for a few seconds. This bit many teams. In December 2020, AWS quietly delivered strong read-after-write consistency for all operations, everywhere, at no cost.

Pre-2020: eventual consistency for overwrites

PUT bucket/foo.txt (new version)
GET bucket/foo.txt
  → might return OLD version for a few seconds

Root cause: multi-AZ replication was async.
Reads could hit a not-yet-updated replica.

Read-after-write for NEW objects was strong. Overwrites + deletes were eventually consistent (usually <1s, sometimes 30+s). Devs shipped complex workarounds.

2020+: strong consistency, everywhere

PUT bucket/foo.txt (new version)
GET bucket/foo.txt
  → ALWAYS returns new version.
LIST bucket/
  → ALWAYS reflects PUT/DELETE.

AWS added a distributed lease + fencing mechanism for the metadata layer. Zero cost, zero API changes. One of the largest silent upgrades in cloud history.

Modern reality: S3 is strongly consistent. GCS + Azure Blob were strongly consistent from day one. You can trust "write-then-read" without extra logic — unless you're using a service that caches S3 (CloudFront, athena, EMR-caching-tables). Those still have their own caching semantics.

Security: 8 layers that keep your bucket private

Nearly every "S3 breach" you've read about was a misconfigured bucket policy. Object storage security is configuration security: get the layers right, breaches are near-impossible.

Bucket policy

IAM-style JSON policy attached to the bucket. Coarse-grained: allow all users in org to read /public/, deny outside VPC.

Best practice: Deny by default. Explicit allow-list. Never make buckets fully public unless static website.

IAM policy (identity-based)

Policy on the user/role. Grants them access to specific buckets/keys.

Best practice: Least privilege. Use IAM roles (not access keys) for services.

Object ACL (legacy)

Per-object ACLs. AWS actively recommends disabling and using bucket policy only.

Best practice: Turn OFF via Object Ownership setting.

Block Public Access

Master switch at account + bucket level. Blocks ALL public access even if bucket policy allows.

Best practice: Enable at account level unless you have a documented reason not to.

Encryption at rest (SSE-S3, SSE-KMS, SSE-C)

S3 encrypts every object on disk. SSE-S3 = AWS keys. SSE-KMS = your KMS. SSE-C = your own key.

Best practice: SSE-KMS for auditable compliance. SSE-S3 is default and free.

Encryption in transit (HTTPS)

TLS 1.2+ enforced by bucket policy. Denies aws:SecureTransport = false.

Best practice: Enforce via bucket policy. Never allow plaintext HTTP.

Access logs + CloudTrail data events

Every request logged for audit. Detect anomalies + prove compliance.

Best practice: Enable for regulated workloads. Sample for cost.

S3 Object Lock (WORM)

Prevent object deletion for a defined period. Ransomware defense. Compliance requirement (SEC 17a-4).

Best practice: Enable Governance or Compliance mode for backups + financial records.
The Capital One breach (2019): not S3's fault — it was an over-permissive IAM role exploited via SSRF. But the data was in S3. Layered defense (block public access + VPC endpoint + KMS + object lock) would have stopped it. Do all of them.

Product comparison

ProductPricingStrengthWeakness
AWS S3$0.023/GB + egress feesEcosystem — deepest integration. Every AWS service works with S3.Egress fees ($0.09/GB out to internet) can dominate cost.
Google Cloud Storage$0.020/GB + egressNative BigQuery + Vertex integration. Global buckets.Smaller ecosystem outside GCP. Similar egress costs.
Azure Blob Storage$0.018/GB + egressDeep .NET / M365 integration. Hierarchical namespace (Data Lake Gen2) for analytics.Azure-only. Complex tier model.
Cloudflare R2$0.015/GB + $0 egressZero egress fees — huge for high-egress workloads. S3-compatible API.Newer. Smaller ecosystem. Not for compliance-only-in-region.
Backblaze B2$0.005/GB + $0.01/GB egressCheapest managed option. S3-compatible.Fewer features. Slower on cold data. No advanced tiering.
Wasabi Hot Cloud Storage$0.0059/GB + $0 egressCheap + no egress + no retrieval fees. All-hot.Fewer regions. Fewer integrations. Data locking policies stricter.
MinIOSelf-hosted (compute + disk)S3 API on your infra. Kubernetes-native. On-prem for compliance.You run the operations. Multi-region requires paid version.
Ceph Object Gateway (RGW)Self-hostedOSS. Works with block + file too. Behind OpenStack Swift + others.Complex ops. Requires skilled team.
OpenStack SwiftSelf-hostedMature OSS. Battle-tested at Rackspace + telco private clouds.Ancient by cloud standards. Not as feature-rich as S3.
IBM Cloud Object Storage (Cleversafe)Enterprise pricingVery high durability (SecureSlice erasure coding). Regulated workloads.Vendor lock-in. IBM-centric ecosystem.
Oracle Cloud Object Storage$0.026/GB (Std) → $0.0026 (Archive)Deep Oracle DB integration. Free egress up to 10TB/mo.Ecosystem smaller. Less community.
Storj DCS$0.004/GB + $0.007/GB egressDecentralized (peer storage nodes). End-to-end encryption. S3-compatible.Newer. Perf can vary. Trust model different.

How to choose: AWS shop → S3. Cheapest + high egress → R2. Cheapest + low egress → B2 or Wasabi. On-prem/compliance → MinIO or Ceph. Analytics-heavy → Azure Data Lake Gen2 or GCS.

12 real-world object storage deployments

Netflix

S3 as the master archive

Every Netflix video encoding lives in S3. Petabytes per title × thousands of titles. Multiple resolutions, HDR variants, per-region masters. Open Connect edge boxes cache the hot subset. S3 is the immutable source of truth.

Airbnb

S3 for all listing images

Airbnb stores >1B property photos in S3. Cloudfront serves them globally. Every upload goes through image processing (thumbnails, WebP variants) with results written back to S3.

Dropbox

S3 → Magic Pocket migration

Dropbox stored all user files in S3 from 2007-2016. Then migrated 500PB to their own storage system (Magic Pocket) to save $75M/year. Perfect case study of when to leave managed storage.

Spotify

GCS for the entire music library

Every song Spotify serves lives in Google Cloud Storage. Multiple encodings (Ogg Vorbis 96/160/320kbps). Global buckets so Sydney users hit local replicas. Backed by CDN edge for hot tracks.

Reddit

S3 for user uploads + backups

Reddit puts everything uploaded — images, videos, avatars — in S3. Also uses S3 Glacier for compliance backups. Peak: millions of writes/day for image uploads.

Snowflake

S3 as the storage layer

Snowflake's entire storage layer is S3 (or GCS/Azure Blob depending on region). Their compute-storage separation architecture would be impossible without cheap object storage. Every Snowflake query reads Parquet files from S3.

Databricks

Delta Lake on S3

Databricks' Delta Lake stores data as Parquet files in S3 + a transaction log. ACID semantics on S3! Basis for modern lakehouses. Used by tens of thousands of enterprises.

Cloudflare

R2 dogfooding — $0 egress win

Cloudflare uses R2 for their own storage — logs, images, backups. Zero egress means moving TB between services is free. This is what enables cheap Workers KV, R2, and other data-heavy products.

GitHub

S3 + Alexandria for LFS

GitHub stores Git LFS (Large File Storage) and release binaries in S3. Every repo release, every LFS pointer resolves to an S3 object. Petabytes across all repos.

Twitch

S3 for VOD storage

Twitch stores video-on-demand recordings in S3. Every past broadcast (subject to retention) sits there. Multiple HLS quality variants. Glacier for long-tail archives.

OpenAI

S3 for training data + checkpoints

OpenAI stores training corpora, model checkpoints, and Fine-tuning uploads in S3. Multi-TB checkpoints per training run. Express One Zone for training I/O throughput.

Backblaze

B2 as an S3 alternative for indie devs

Countless indie SaaS founders back onto Backblaze B2 — same S3 API, 1/4 the cost. Popular for backup services, media sites, static hosting. Their per-GB pricing is transparent + brutal to compete with.

Key takeaways

  • 1Object storage is immutable objects addressed by key. Not a filesystem. Give up random-access mutation, gain planet-scale durability.
  • 211 nines of durability comes from erasure coding + cross-AZ placement. Any 3 of 9 shards can die; object survives.
  • 3Tiered pricing is where the savings hide. Move cold data to Infrequent Access or Glacier. Save 80% of storage bill.
  • 4Multipart upload is the standard for >100MB objects. Parallelism + granular retries. Don't forget to abort incomplete uploads.
  • 5Since Dec 2020, S3 has been strongly consistent. No more read-after-write bugs. GCS + Azure Blob always were.
  • 6Security is layered configuration. Block Public Access + bucket policy + VPC endpoint + KMS + Object Lock. Every breach is a config bug.
  • 7Egress fees dominate cost for read-heavy workloads. Consider Cloudflare R2 or Backblaze B2 if you serve lots of traffic.
  • 8Modern lakehouses (Snowflake, Databricks, Iceberg) all treat object storage as the substrate. This is the AI-era data lake.

References & further reading

  • • Vogels, W. (2006). "Amazon S3 launch." AWS blog. Historical note that changed cloud economics.
  • • Vogels, W. (2020). "Diving Deep on S3 Consistency." AWS blog. How AWS achieved strong consistency.
  • • Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). "The Google File System." SOSP. Foundational for object storage design.
  • • Weil, S. et al. (2006). "Ceph: A Scalable, High-Performance Distributed File System." OSDI.
  • • Zhang, Z. et al. (2019). "Building a Realtime Data Warehouse at Uber." VLDB. Uber's HDFS+object-storage architecture.
  • • AWS S3 Documentation: The definitive reference — read at least "Performance guidelines," "Consistency model," and "Security best practices."
  • • Dropbox Tech Blog: "Scaling to Exabytes and Beyond" — Magic Pocket migration story.
  • • Cloudflare Blog (2022): "R2: Object Storage Without Egress Fees." Fee-model disruption story.
  • • Kleppmann, M. (2017). DDIA. Chapters 6-10 discuss the primitives underlying object storage.
  • • NIST SP 800-88: Data sanitization guidance for archived object storage.