@node WARCs
-@section WARCs management
+@unnumbered WARCs management
To view WARC files, you have to load them in daemon. Responses will be
transparently replaced from those WARCs for corresponding URIs.
Multi-frame format is properly indexed. Dictionary at the beginning
is also supported.
-It is processed with with @command{unzstd} (@command{redo
-cmd/unzstd/unzstd}) utility. It eats compressed stream from
-@code{stdin}, outputs decompressed data to @code{stdout}, and prints
-each frame size with corresponding decompressed data size to 3rd file
-descriptor (if it is opened). You can adjust path to it with @code{-X
-go.stargrave.org/tofuproxy/warc.UnZSTDPath} command line option during
-building.
+It is processed with with @command{unzstd} (@file{cmd/unzstd/unzstd})
+utility. It eats compressed stream from @code{stdin}, outputs
+decompressed data to @code{stdout}, and prints each frame size with
+corresponding decompressed data size to 3rd file descriptor (if it is
+opened). You can adjust path to it with
+@code{-X go.stargrave.org/tofuproxy/warc.UnZSTDPath} command line option
+during building.
@end table
Loading of WARC involves its whole reading and remembering where is each
URI response is located. You can @code{echo SAVE > fifos/add-warcs} to
-save in-memory index to the disk as @file{....idx.gob} file. During
-the next load, if that file exists, it is used as index immediately,
+save in-memory index to the disk as @file{....idx.gob} files. During
+the next load, if those files exists, they are used as index immediately,
without expensive WARC parsing.
-@code{redo warc-extract.cmd} builds @command{warc-extract.cmd} utility,
-that uses exactly the same code for parsing WARCs. It can be used to
-check if WARCs can be successfully loaded, to list all URIs after, to
-extract some specified URI and to pre-generate @file{.idx.gob} indexes.
+@code{redo warc-extract.cmd} utility uses exactly the same code for
+parsing WARCs. It can be used to check if WARCs can be successfully
+loaded, to list all URIs after, to extract some specified URI and to
+pre-generate @file{.idx.gob} indexes.
@example
$ warc-extract.cmd -idx \
Following example can be used to create multi-frame @file{.warc.zst}
from any kind of already existing WARCs. It has better compression ratio
-and much higher decompression speed.
+and much higher decompression speed, than @file{.warc.gz}.
@example
$ redo cmd/enzstd/enzstd