1 glocate -- ZFS-diff-friendly locate-like utility
3 This utility is intended to keep the database of filesystem hierarchy
4 and quickly display some part of it. Like ordinary *locate utilities.
5 But unlike others, it is able to eat zfs-diff's output and apply the
6 changes to existing database.
8 Why I wrote it? Indexing, just "find /big" can take a considerable
9 amount of time, like an hour or so, with many I/O operations spent. But
10 my home NAS has relatively few number of changes made every day. The
11 only possible quick way to determine what exactly was modified is to
12 traverse over ZFS'es Merkle trees to find a difference between
13 snapshots. Fortunately zfs-diff command does exactly that, providing
14 pretty machine-friendly output.
16 Why this utility is so complicated? Initially it kept all database in
17 memory, but that takes 2-3 GiBs of memory, that is huge amount. Moreover
18 it fully loads it to perform any basic searches. So current
19 implementation uses temporary files and heavy use of data streaming.
21 Its storage format is simple: Zstandard-compressed list of records:
23 * 16-bit BE size of the following name
24 * entity (file, directory, symbolic link, etc) name itself.
25 Directory has trailing "/"
26 * single byte indicating current file's depth
27 * 64-bit BE mtime seconds
28 * 64-bit BE file or directory (sum of all files and directories) size
30 Its indexing algorithm is following:
32 * traverse over all filesystem hierarchy in a *sorted* order. All
33 records are written to temporary file, without directory sizes,
34 because they are not known in advance during the walking
35 * during the walk, remember in memory each directory's total size
36 * read all records from that temporary file, writing to another one, but
37 replacing directory sizes with ones remembered
41 * searching is performed on each record streamed from the database
42 * if -root is specified, then search will stop after that hierarchy part
44 * by default all elements are printed, unless you provide a single
45 argument that becomes "*X*" pattern matched on case-lowered path
48 Updating algorithm is following:
50 * read all [-+MR] actions from zfs-diff, validating the whole format
51 * each file's "R" becomes "-" and "+" actions
52 * if there are directory "R", then collect them and stream from current
53 database to determine each path entity you have to "-" and "+"
54 * each "+" adds an entry to the list of "M"s
55 * sort all "-", "+" and "M" filenames in ascending order
56 * get entity's information for each "M" (remembering its size and mtime)
57 * stream current database records, writing them to temporary file
58 * if record exists in "-"-list, then skip it
59 * if any "+" exists in the *sorted* list, that has precedence over the
60 record from database, then insert it into the stream, taking size and
61 mtime information from "M"-list
62 * if any "M" exists for the read record, then use it to alter it
63 * all that time, directory size calculating algorithm also works, the
64 same one used during indexing
65 * create another temporary file to copy the records with actualized
71 $ cd /big ; glocate -db /tmp/glocate.db -index
73 $ glocate -db /tmp/glocate.db
76 $ glocate -db /tmp/glocate.db -machine
77 [machine parseable list of files with sizes and mtimes]
79 $ glocate -db /tmp/glocate.db -tree
80 [beauty tree-like list of files with sizes and mtimes]
82 $ glocate -db /tmp/glocate.db -root music
83 [just a music hierarchy path]
85 $ glocate -db /tmp/glocate.db -root music blasphemy | grep "/$"
86 music/Blasphemy-2001-Gods_Of_War_+_Blood_Upon_The_Altar/
87 music/Cryptopsy-1994-Blasphemy_Made_Flesh/
88 music/Infernal_Blasphemy-2005-Unleashed/
89 music/Ravenous-Assembled_In_Blasphemy/
90 music/Sect_Of_Execration-2002-Baptized_Through_Blasphemy/
91 music/Spectral_Blasphemy-2012-Blasphmemial_Catastrophic/
93 and update it carefully, providing the strip prefix to -update:
96 $ zfs diff -FH big@snap2 | glocate -db /tmp/glocate.db -update /big/
98 glocate is copylefted free software: see the file COPYING for copying