Switch language한국어
Back to the list

I built a faster alternative to cp and rsync — here's how it works

TL;DR AI

Key summary

2 min read
  1. The author built fast-copy, a Python CLI that speeds file copies by reading files in physical disk order and deduplicating duplicates.

  2. fast-copy obtains file physical offsets (FIEMAP on Linux, fcntl on macOS, FSCTL on Windows) and sorts files by block position to read sequentially.

  3. It hashes files with xxHash-128 (SHA-256 fallback), stores hashes in SQLite, creates hard links for duplicates, and streams ~100MB tar batches over raw SSH for remote transfers.

  4. Benchmarks on 92K files showed local copies wrote 509.8 MB in 17.9 seconds (28.5 MB/s) and remote transfers were reported about 3× faster than SFTP.

Read the original