Syncing Uploaded Media to the New Origin

Problem Statement

Databases get a migration plan; the media directory usually gets a cp -r the night before. It is the larger of the two, it changes continuously while you copy it, and the failure it produces is uniquely bad — a page that renders perfectly with a broken image in it, returning 200, invisible to every uptime check. Uploads made during the copy window are the ones that go missing, and they are by definition the most recent and most noticed. This page sits under Staging to Production Sync.

Seed, delta, dual-write, verify Four phases of a media migration: a bulk seed copy, repeated delta syncs, a dual-write window across the cutover, and a final byte-level verification. The Window Is Closed by Dual Writes 1. Seed bulk copy, days early slow, unattended 2. Delta sync repeated, cheap shrinks the gap 3. Dual write across the cutover closes the gap 4. Verify counts and hashes before decommission Without phase three there is always a window in which an upload lands on only one store. That window is where the newest, most-noticed files go missing.
Phases one and two are bandwidth; phase three is the design decision that makes the cutover lossless.

When to Use This Approach

  • User-uploaded media lives on a filesystem or object store that is moving to a new origin or provider.
  • The volume is large enough that a single copy takes longer than an acceptable downtime window.
  • Uploads continue during the migration — any site with user accounts, a CMS, or a support portal.
  • You need to prove completeness before decommissioning the old store rather than waiting for complaints.

Why Media Is Harder Than the Database

A database migration has a well-understood shape: dump, restore, replicate, cut over, and a transaction log that can prove nothing was lost. Media has none of that. There is no transaction log, no consistent snapshot, and no equivalent of replication lag to measure. What there is, instead, is volume — usually an order of magnitude more bytes than the database — and a write pattern that continues throughout.

The failure mode is also uniquely difficult to detect. A missing database row produces an error somewhere: a page that will not render, an API call that returns 500, a report whose totals do not add up. A missing image produces a page that renders perfectly, returns 200, passes every uptime check, and simply has a hole in it. Nothing in the monitoring stack is watching for holes.

That combination — no consistency guarantee, large volume, silent failure — is why the phase ordering below is worth following exactly rather than compressing. The seed handles the volume, the deltas shrink the moving target, and the dual-write closes the only remaining gap. Skip the dual-write and you are relying on the gap being small, which is true, and on nothing important landing in it, which is not.

# The number that matters, and the reason a gap is unacceptable: uploads per minute
awk '$7 ~ /^\/uploads\// && $1 == "POST"' /var/log/nginx/access.log | wc -l

Step-by-Step Instructions

1. Seed the Bulk Early and Unattended

Run the first full copy days ahead. It is slow, it is safe, and everything afterwards is a small delta.

# Seed: preserve times so later delta passes can compare cheaply
rsync -aH --info=progress2 --partial \
  /srv/uploads/ deploy@new-origin:/srv/uploads/

For object storage the equivalent is a bucket-to-bucket sync, which parallelises far better than a file-by-file copy:

# Object store seed; --size-only avoids re-copying on clock skew between providers
aws s3 sync s3://legacy-uploads s3://new-uploads --size-only --no-progress

2. Delta Sync on a Schedule

Repeat the sync nightly until the delta is small and predictable. The last delta before the cutover should complete in minutes, and you should know that number before the night itself.

# Nightly delta; --delete only once you are confident deletions are intentional
rsync -aH --delete --stats \
  /srv/uploads/ deploy@new-origin:/srv/uploads/ | tail -20

3. Dual-Write Across the Cutover Window

This is the step that makes the migration lossless. For the duration of the window, the application writes every new upload to both stores; reads continue from the old one until the switch, then from the new one.

# Dual write: the new store must not be able to fail the request
def store_upload(key: bytes, data: bytes) -> None:
    legacy.put(key, data)                 # authoritative during the window
    try:
        new.put(key, data)                # best-effort mirror
    except Exception:
        log.warning("mirror write failed", extra={"key": key})
        mirror_backlog.push(key)          # reconciled by the delta sync
Read and write paths through the cutover Before the switch, reads and the authoritative write go to the legacy store with a mirrored write to the new store; after the switch the roles reverse. Which Store Is Authoritative, and When PHASE READS FROM AUTHORITATIVE WRITE MIRROR Seed and delta legacy legacy none Dual-write window legacy legacy new, best effort After the switch new new legacy, until verified Keeping the reverse mirror after the switch is what makes the rollback path real.
Reversing the mirror rather than stopping it is the difference between a rollback that works and one that loses a day of uploads.

4. Switch Reads, Keep the Reverse Mirror

Flip reads to the new store and reverse the mirror so writes also land on the legacy store for a further day or two. That keeps the DNS-level rollback described in Reverting DNS Records During a Failed Cutover genuinely available rather than nominally available.

5. Verify Before Decommissioning

Counts, then sizes, then hashes on a sample. Anything less and you are trusting a sync tool’s exit code with data you cannot recreate.

Worked Example

A community platform held 2.1 TB of user uploads across 4.6 million files, moving from a self-managed NFS volume to object storage in a different region.

The plan and what it produced:

  1. Seed started nine days out and took 41 hours over a saturated link.
  2. Nightly deltas ran for a week; the last one before cutover moved 12 GB and completed in nine minutes — the number quoted in the runbook.
  3. Dual writes were enabled 24 hours before the switch. Over that period 61,000 files were written to both stores, and 47 mirror writes failed and were reconciled from the backlog.
  4. Reads switched at the DNS cutover; the reverse mirror stayed on for 48 hours.
  5. Verification found 19 files present on the legacy store and absent on the new one, all written in the four-minute period between the last delta and the dual-write deployment — precisely the gap the dual-write phase exists to close, and evidence that the phase ordering mattered.

Those 19 files were copied by hand. The old volume was decommissioned two weeks later, after a second verification pass returned clean.

Verification

# 1. File counts must match exactly
find /srv/uploads -type f | wc -l
aws s3 ls s3://new-uploads --recursive | wc -l

# 2. Total bytes should match; a difference points at truncated transfers
du -sb /srv/uploads
aws s3 ls s3://new-uploads --recursive --summarize | grep 'Total Size'

# 3. Byte-for-byte on a random sample — the check that catches silent corruption
find /srv/uploads -type f | shuf -n 500 | while read -r f; do
  key="${f#/srv/uploads/}"
  a=$(md5sum "$f" | cut -d' ' -f1)
  b=$(aws s3api head-object --bucket new-uploads --key "$key" --query ETag --output text | tr -d '"')
  [ "$a" = "$b" ] || echo "MISMATCH $key"
done

Note that a multipart-uploaded object’s ETag is not a plain MD5, so for large files compare with a checksum the provider computes on request rather than the ETag. Getting this wrong produces a wall of false mismatches and a team that stops trusting the check.

Verification ladder from cheap to conclusive Three rungs — file count, total bytes and sampled hashes — with what each proves and what it cannot detect. Each Rung Catches What the Last One Missed 1. File count catches missing files blind to corruption 2. Total bytes catches truncation blind to swapped files 3. Sampled hashes catches corruption conclusive on the sample
Counts and sizes are cheap enough to run continuously; hashes are sampled because they are not.

FAQ

Can I skip the dual-write phase if the delta sync is fast? Only if you can accept losing uploads made in the gap between the final sync and the read switch. That gap is minutes at best, and the files in it are the newest ones — the ones whose absence is noticed immediately.

Should the mirror write be allowed to fail the request? No. During the window the legacy store is authoritative, so a failed mirror write should be logged and reconciled, never surfaced to the user. Making the mirror mandatory turns a migration convenience into a new single point of failure.

How do I handle files deleted during the migration? Decide the policy before enabling --delete on the sync. Mirroring deletions keeps the stores identical; deferring them keeps a recovery option. Most teams defer during the window and reconcile once afterwards.

Why do ETags differ for files that are definitely identical? Because a multipart upload’s ETag is a hash of part hashes, not of the object. Compare using a checksum the provider computes on demand, or compare sizes and a locally computed hash of the downloaded bytes.

When can the legacy store be switched off? After a second clean verification pass, and after the reverse mirror has been off long enough that a rollback is no longer on the table. Two weeks is a common choice, and the storage cost of that delay is almost always smaller than the cost of being wrong.

Should media URLs change during the migration? Avoid it if you can. Keeping the paths identical means the database rows referencing them need no rewriting, and a rollback does not have to undo a data migration as well as a storage one. Where the host must change, serve the old host as a redirect rather than rewriting every reference at once.

How do I handle files added to the legacy store after the read switch? They should not exist — the application is writing to the new store by then. If they do appear, something is still writing to the old path, and the reverse mirror will be reflecting them in the wrong direction. Treat any post-switch write to the legacy store as a defect to be traced, not as a file to be copied.

Is rsync or a provider’s sync tool the better choice? Whichever matches the storage. rsync is excellent between filesystems and poor against object storage, where per-object API calls dominate and a provider’s parallel sync is dramatically faster. Using the wrong one turns a two-hour seed into a two-day one.

What about thumbnails and derived images? Regenerate them rather than copying, if regeneration is cheap and deterministic. They are typically the majority of the file count and the minority of the value, and excluding them from the sync can halve the transfer time. Copy them if regeneration is expensive or if the originals are gone.

Do I need to preserve timestamps and permissions? Timestamps, yes — later delta passes use them to decide what has changed, and losing them forces a full re-comparison. Permissions matter only if the application depends on them, which is worth checking before assuming either way.

Related

← Back to Staging to Production Sync