1 glocate -- ZFS-diff-friendly locate-like utility
3 This utility is intended to keep the database of filesystem hierarchy
4 and quickly display some part of it. Like ordinary *locate utilities.
5 But unlike others, it is able to eat zfs-diff's output and apply the
6 changes to existing database.
8 Why I wrote it? I have got ~18M files ZFS data storage, where even
9 "find /storage" takes considerable amount of time, up to an hour.
10 So I have to use separate indexed database and search against it.
11 "locate" family of utilities does exactly that. But none of them are
12 able to detect a few seldom made changes to the dataset, without
13 traversing through the whole dataset anyway, consuming much IO.
15 Fortunately ZFS design with Merkle trees is able to show us the
16 difference quickly and without notable IO. "zfs diff" command's
17 output is very machine friendly. So locate-like utility has to be able
18 to update its database with zfs-diff's output.
20 Why this utility is so relatively complicated? Initially it kept all
21 database in memory, but that took 2-3 GiBs of memory, that is a huge
22 amount. Moreover it fully loads it to perform any basic searches. So
23 current implementation uses temporary files and heavy use of data
24 streaming. Database in my case takes less than 128MiB of data. And
25 searching takes only several seconds on my machine.
27 It is free software: see the file COPYING for copying conditions.