2 @section WARCs management
4 To view WARC files, you have to load them in daemon. Responses will be
5 transparently replaced from those WARCs for corresponding URIs.
7 There is no strict validation or checking of WARCs correctness at all!
8 But built-in WARC support seems to be good enough for various sources.
9 Uncompressed, @command{gzip} (multiple streams and single stream are
10 supported) and @command{zstd} compressed ones are supported.
12 Searching in compressed files is @strong{slow} -- every request will
13 lead to decompression of the file from the very beginning, so keeping
14 uncompressed WARCs on compressed ZFS dataset is much more preferable.
15 @command{tofuproxy} does not take advantage of multistream gzip files.
23 $ tee fifos/add-warcs < warcs.txt
24 smth.warc-00000.warc.gz
25 smth.warc-00001.warc.gz
26 smth.warc-00002.warc.gz
31 Visit the URI you know, that exists in those WARCs, or go to
32 @url{http://warc/}, to view full list of known loaded URIs from
36 Pay attention that order of WARCs loading is important! WARC can be
37 segmented and single response can be split on multiple WARC files.
38 Each following WARC files will overwrite possibly already existing URIs.
41 To list and delete loaded known WARCs:
44 $ cat fifos/list-warcs
45 smth.warc-00000.warc.gz 154
46 smth.warc-00001.warc.gz 13
47 smth.warc-00002.warc.gz 0
49 $ echo another.warc > fifos/del-warcs
52 One possibility that @file{smth.warc-00002.warc.gz} has no URIs is that
53 it contains continuation segmented records.
57 Loading of WARC involves its whole reading and remembering where is each
58 URI response is located. You can @code{echo SAVE > fifos/add-warcs} to
59 save in-memory index to the disk as @file{....warc.idx.gob} file. During
60 the next load, if that file exists, it is used as index immediately,
61 without expensive WARC reading.
63 @code{redo warc-extract.cmd} builds @command{warc-extract.cmd} utility,
64 that uses exactly the same code for parsing WARCs. It can be used to
65 check if WARCs can be successfully loaded, to list all URIs after, to
66 extract some specified URI and to pre-generate @file{.idx.gob} indexes.
69 $ warc-extract.cmd -idx \
70 smth.warc-00000.warc.gz \
71 smth.warc-00001.warc.gz \
72 smth.warc-00002.warc.gz
73 $ warc-extract.cmd -uri http://some/uri \
74 smth.warc-00000.warc.gz \
75 smth.warc-00001.warc.gz \
76 smth.warc-00002.warc.gz
79 @url{https://www.gnu.org/software/wget/, GNU Wget} can be easily used to
83 $ wget ... [--page-requisites] [--recursive] \
84 --no-warc-keep-log --no-warc-digests [--warc-max-size=XXX] \
85 --warc-file smth.warc ...