hashsync Cut an rsync Copy Over SSH From 19 Minutes to 4 by Matching Data the Receiver Already Had

rsync already skips files that haven’t changed. Where it falls down is data that moved: a file renamed, split, or copied into a new format, which rsync treats as brand new. hashsync goes after that case.

You run it the way you’d run ssh, and it watches the stream. The receiving side keeps an index of folders you point it at, which the README calls roots. When the data going over the wire contains a block from one of those files, “hashsync sends a reference to the block instead of its contents.” Blocks are 16 KiB by default and can match at any offset, so a block found halfway into a different file still counts. The sender finds candidates with a rolling hash, then checks each one with a strong hash, so a match is “Always exact.”

The usual way in is through rsync: rsync -a --delete-delay -e 'hashsync ssh' …. The README asks for two more things. It recommends patching rsync with a file shipped in the repo’s extras/ folder, because rsync sends data in 32 KiB pieces that split blocks, so “it can miss up to half of the blocks.” And it says not to use rsync’s -z, since compressed data can’t be matched.

The benchmark used two versions of the GPT-2 repository from Hugging Face, one from May 19, 2021 and one from February 19, 2024. The newer one “adds a safetensors file and three ONNX files with the same weights as the older files,” which is the kind of change rsync can’t see. rsync alone moved 2.12 GB; with hashsync the receiver got 416.47 MiB.

There’s no daemon. The README says “There’s no service or network port,” but both machines need hashsync installed, and it needs Linux 5.8 or newer. The index is a RocksDB store; the design notes say SQLite was tried and rejected after an update adding 262,000 rows took 18 to 23 seconds, against 2.4 to 4 seconds on RocksDB. The repo had one star the day after release.

hashsync matches content wherever it sits instead of matching file names, and the README is clear that it changes only what crosses the network: “the commands at both ends see exactly the bytes that they would see through ssh.” That means rsync, tar or a nix-store --export pipe can use it without a server process. What the project hasn’t shown yet is how the index holds up on a receiver with many terabytes of roots, or how long that first indexing pass takes.