4 Fast stateful file/disk data syncer.
8 The main purpose of this utility is fast data synchronizing between two
9 hard drives: one is fast (SSD, SATA HDD), another is connected through
10 slow USB interface. Task is to lower amounts of data needed to be
13 This utility is stateful: it keeps precomputed data hashes in separate
14 statefile and use it to determine if we need to update block of data.
17 # sync from very fast SSD to slow USB connected HDD
18 % ./syncer -src /dev/ada0 -dst /dev/da0 -state state.bin
20 # all blocks were transferred to da0
23 Now we have statefile containing cryptographic hashes of the blocks from
24 source and copy of all read data in destination. Now if we run it again:
27 % ./syncer -src /dev/ada0 -dst /dev/da0 -state state.bin
29 # only one block was transferred to da0
32 Only one modified block was transferred during this session. We read all
33 data from source again, compute hashes and understand what was updated
34 since the last run. Statefile is updated at the end.
36 Utility parallelize hash computations among all found CPUs. It updates
37 statefile atomically (saves data in temporary file and then renames it).
38 You can configure the blocksize: shorter transfers but bigger statefile
39 (it is kept in memory), or larger transfer and smaller statefile. All
40 writes are sequential.
42 syncer is free software: see the file COPYING for copying conditions.
47 % go get github.com/dchest/blake2b
49 # syncer executable file should be in current directory
54 `SRC_SIZE || BLK_SIZE || HASH0 || HASH1 || ...`
56 SRC_SIZE contains size of the source, when it was firstly read. BLK_SIZE
57 is the blocksize used. Both are 64-bit big-endian unsigned integers. If
58 either size or blocksize differs, then syncer will deny using that
59 statefile as a precaution. HASHx is BLAKE2b-512 hash output, 64 bytes.