]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
5 years agoexamples/cgit-commit-filter.lua: some doc updates
Eric Wong [Wed, 24 Apr 2019 22:51:42 +0000 (22:51 +0000)]
examples/cgit-commit-filter.lua: some doc updates

It's been a while since I wrote this, and it needs to be kept
up-to-date with some advances in our Perl code.

5 years agoexamples: cgit filter for use with WwwHighlight
Eric Wong [Wed, 24 Apr 2019 22:44:04 +0000 (22:44 +0000)]
examples: cgit filter for use with WwwHighlight

I'm using this as the cgit about-filter and source-filter
in https://80x24.org/public-inbox.git

5 years agowwwhighlight: deal with no-op highlight
Eric Wong [Wed, 24 Apr 2019 21:56:53 +0000 (21:56 +0000)]
wwwhighlight: deal with no-op highlight

Not all inputs are highlight-able, so reuse the original
input and just linkify it if it can't be highlighted.

5 years agoexamples/cgit-commit-filter: remove unused variable
Eric Wong [Wed, 24 Apr 2019 21:22:11 +0000 (21:22 +0000)]
examples/cgit-commit-filter: remove unused variable

5 years agowwwhighlight: do linkification, too
Eric Wong [Wed, 24 Apr 2019 19:52:14 +0000 (19:52 +0000)]
wwwhighlight: do linkification, too

Otherwise, there's no reason to use this API over highlight(1).
Maybe this can be an option in the future; but I'm struggling to
find a reason to not do it by default.

5 years agoview: avoid "1+ messages" in per-message footer of /t/ and /T/
Eric Wong [Tue, 23 Apr 2019 03:33:41 +0000 (03:33 +0000)]
view: avoid "1+ messages" in per-message footer of /t/ and /T/

Try to appear gramatically correct and state:
"only message in thread" when there's only one known (to us)
message in the thread.

5 years agodoc: add standards reference
Eric Wong [Sat, 20 Apr 2019 00:20:04 +0000 (00:20 +0000)]
doc: add standards reference

Incomplete at the moment, but this ought to be a handy reference
for both implementers and users alike.

5 years agot/hl_mod: workaround w3m not handling '
Eric Wong [Fri, 19 Apr 2019 08:09:31 +0000 (08:09 +0000)]
t/hl_mod: workaround w3m not handling '

This fixes a test failure on my Debian buster system.

Bug report filed for w3m to handle "'":

https://bugs.debian.org/927409

and for "highlight" to favor "'" in case other browsers fail:

https://bugs.debian.org/927410

5 years agolinkify: require parentheses pairs in URLs
Eric Wong [Thu, 18 Apr 2019 23:49:42 +0000 (23:49 +0000)]
linkify: require parentheses pairs in URLs

Dangling parentheses with trailing punctuation usually means the
parentheses is not intended as part of the URL.

5 years agoview: show "(no subject)" consistently in HTML
Eric Wong [Thu, 18 Apr 2019 23:31:46 +0000 (23:31 +0000)]
view: show "(no subject)" consistently in HTML

Empty subjects ("") and undefined Subjects: are now both
displayed as "(no subject)" for now.

5 years agodoc: config: fix braino/typo :x
Eric Wong [Thu, 18 Apr 2019 22:53:20 +0000 (22:53 +0000)]
doc: config: fix braino/typo :x

5 years agolinkify: don't get confused by URLs in Perl code, at least
Eric Wong [Thu, 18 Apr 2019 22:47:01 +0000 (22:47 +0000)]
linkify: don't get confused by URLs in Perl code, at least

The URLs at the top of WwwStream.pm weren't getting linkified
correctly.

5 years agoinbox: add `modified' sub
Eric Wong [Thu, 18 Apr 2019 00:58:06 +0000 (00:58 +0000)]
inbox: add `modified' sub

For inboxes with SQLite enabled (all v2, and probably most v1);
we can use the overview DB to get the timestamp of the latest
message.  It's faster than scanning git branches for commit times,
but not always the same.

5 years agogit: calculate modified time of repository
Eric Wong [Wed, 17 Apr 2019 10:49:15 +0000 (10:49 +0000)]
git: calculate modified time of repository

This will be used for generating an HTML listing for v1 inboxes,
at least.  The logic for this follows that of grokmirror,
and we may dynamically generate manifest.js.gz natively...

5 years agoconfig: use '$ibx' instead of '$rv' to denote Inbox objects
Eric Wong [Wed, 17 Apr 2019 00:32:41 +0000 (00:32 +0000)]
config: use '$ibx' instead of '$rv' to denote Inbox objects

Followup-to: 6e6f7999361925e4
  ("cleanup: use '$ibx' consistently when referring to Inbox refs")

5 years agoTODO: add an item for hooks support
Eric Wong [Mon, 15 Apr 2019 18:28:00 +0000 (18:28 +0000)]
TODO: add an item for hooks support

and add a note for grokmirror support/integration, too

5 years agoHACKING: stop advertising mailing list subscription info
Eric Wong [Mon, 15 Apr 2019 07:32:49 +0000 (07:32 +0000)]
HACKING: stop advertising mailing list subscription info

It's still available, but no point in advertising something
which increases the dependency on a centralized subscriber list.
In fact, it's actively against the goal of promoting resistance
against centralization.

5 years agoimport: prefix 'inbox' with '-' to denote Inbox ref
Eric Wong [Tue, 16 Apr 2019 01:10:10 +0000 (01:10 +0000)]
import: prefix 'inbox' with '-' to denote Inbox ref

This is for consistency with other fields which follow
this pattern w.r.t. field-naming when referring to internal
fields.

5 years agocleanup: use '$ibx' consistently when referring to Inbox refs
Eric Wong [Tue, 16 Apr 2019 01:10:09 +0000 (01:10 +0000)]
cleanup: use '$ibx' consistently when referring to Inbox refs

'$inbox' is more human-readable, so that is for the more
human-readable name in most cases.  Making our variable naming
more consistent should make the code easier-to-review and
harder to screw up.

5 years agowww: remove unnecessary Git object reference
Eric Wong [Tue, 16 Apr 2019 01:19:42 +0000 (01:19 +0000)]
www: remove unnecessary Git object reference

We access the Git object via the Inbox object nowadays, so
there's no point in having a shortcut to it, anymore.

5 years agoINSTALL: document Crypt:CBC for PublicInbox::Unsubscribe
Eric Wong [Tue, 16 Apr 2019 01:17:21 +0000 (01:17 +0000)]
INSTALL: document Crypt:CBC for PublicInbox::Unsubscribe

I basically forgot about this module, but "make syntax" on a
new chroot trips it up.

5 years agoconfig: fix regression in repo.path => coderepo.dir mapping
Eric Wong [Mon, 15 Apr 2019 06:35:30 +0000 (06:35 +0000)]
config: fix regression in repo.path => coderepo.dir mapping

We parse cgitrc for "repo.path", while we use "coderepo.dir" to
mean the same thing for non-cgit users.  So I ended up confusing
myself, here.

But then again, git uses "--git-dir" and "GIT_DIR", so I suspect
"dir" is the better choice than "path", here

5 years agoviewdiff: document constants
Eric Wong [Fri, 5 Apr 2019 20:04:29 +0000 (20:04 +0000)]
viewdiff: document constants

We'll be building off of this for showing diffs in
the coderepo views.

5 years agodoc/config: update cgit.cgi scan location
Eric Wong [Fri, 5 Apr 2019 20:04:28 +0000 (20:04 +0000)]
doc/config: update cgit.cgi scan location

We account for the upstream default location as well as
the Debian-installed one.

5 years agoconfig: support more cgit directives for project lists
Eric Wong [Fri, 5 Apr 2019 20:04:27 +0000 (20:04 +0000)]
config: support more cgit directives for project lists

Hopefully this gets us closer to matching cgit upstream behavior
(which also lacks tests).  We'll still need to support macro
expansion at some point for compatibility...

5 years agocgit: serve static css, logo, favicon directly
Eric Wong [Fri, 5 Apr 2019 20:04:26 +0000 (20:04 +0000)]
cgit: serve static css, logo, favicon directly

We can reduce the configuration needed to run cgit by reusing
the static file handling logic of the dumb git HTTP protocol.

I hate logos and icons, so don't expect public-inbox.org or
80x24.org to ever have those to waste users' bandwidth with :P
But I expect other users to find this useful.

5 years agoconfig: support cgit scan-path and scan-hidden-path
Eric Wong [Fri, 5 Apr 2019 20:04:25 +0000 (20:04 +0000)]
config: support cgit scan-path and scan-hidden-path

project_list support still needs to be done

And tests need to be written... :<

5 years agodoc: fix a few minor typos
Kyle Meyer [Sun, 7 Apr 2019 03:31:40 +0000 (23:31 -0400)]
doc: fix a few minor typos

5 years agowww: fix missing cgit fallback after legacy redirects
Eric Wong [Tue, 2 Apr 2019 02:29:44 +0000 (02:29 +0000)]
www: fix missing cgit fallback after legacy redirects

We need to instate our cgit handler everywhere we use NewsWWW
to catch wildcard requests which our normal endpoints do not
handle.

5 years agospawn: require soft and hard entries in RLIMIT_* handling
Eric Wong [Tue, 12 Mar 2019 01:47:59 +0000 (01:47 +0000)]
spawn: require soft and hard entries in RLIMIT_* handling

Our high-level config already treats single limits as a
soft==hard limit for limiters; so stop handling that redundant
in the low-level spawn() sub.

5 years agocgit: use a dedicated named limiter
Eric Wong [Tue, 12 Mar 2019 01:18:30 +0000 (01:18 +0000)]
cgit: use a dedicated named limiter

I mainly need this to enforce RLIMIT_CPU (and RLIMIT_CORE)
when requests come which generate giant, unrealistic diffs.

Per-coderepo limiters may be added in the future.  But for now,
I need to prevent cgit from monopolizing resources on my dinky
server.

5 years agoqspawn: wire up RLIMIT_* handling to limiters
Eric Wong [Mon, 11 Mar 2019 23:53:02 +0000 (23:53 +0000)]
qspawn: wire up RLIMIT_* handling to limiters

This allows users to configure RLIMIT_{CORE,CPU,DATA} using
our "limiter" config directive when spawning external processes.

5 years agowww: wire up cgit as a 404 handler if cgitrc is configured
Eric Wong [Mon, 11 Mar 2019 05:32:07 +0000 (05:32 +0000)]
www: wire up cgit as a 404 handler if cgitrc is configured

Requests intended for cgit are unlikely to conflict with
requests to inboxes.  So we can safely hand those requests
off to cgit.cgi.

5 years agocgit: support running cgit as a standalone CGI
Eric Wong [Sun, 10 Mar 2019 22:02:29 +0000 (22:02 +0000)]
cgit: support running cgit as a standalone CGI

We depend on git-http-backend for smart HTTP clone support,
however; since cgit does not support smart clones natively.
WWW.pm will be able to cascade down to this as a 404 handler in
the future.

5 years agospawn: support absolute paths
Eric Wong [Sun, 10 Mar 2019 21:40:00 +0000 (21:40 +0000)]
spawn: support absolute paths

cgit (and most other CGI executables) is not typically installed
for use via $PATH, so we'll need to support absolute paths to
run it.

5 years agogithttpbackend: check for other errors and relax CRLF check
Eric Wong [Sun, 10 Mar 2019 10:32:33 +0000 (10:32 +0000)]
githttpbackend: check for other errors and relax CRLF check

Reads to git-http-backend(1) could fail or EOF prematurely,
so we must be ready for that case.

Furthermore, cgit (and possibly other CGI) uses LF instead
of CRLF, so support those programs, too.

5 years agogithttpbackend: move more psgi.input handling into subroutine
Eric Wong [Sun, 10 Mar 2019 07:51:45 +0000 (07:51 +0000)]
githttpbackend: move more psgi.input handling into subroutine

This will be useful for other CGI wrappers we make.
This also fixes a bug with some PSGI servers which did not
present a real IO::Handle in the psgi.input env field.

5 years agosupport publicinbox.cgitrc directive
Eric Wong [Fri, 8 Mar 2019 01:21:22 +0000 (01:21 +0000)]
support publicinbox.cgitrc directive

We can save admins the trouble of declaring [coderepo "..."]
sections in the public-inbox config by parsing the cgitrc
directly.

Macro expansion (e.g. $HTTP_HOST) expansion is not supported,
yet; but may be in the future.

5 years agospawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE
Eric Wong [Mon, 25 Feb 2019 05:14:10 +0000 (05:14 +0000)]
spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE

We'll be spawning cgit and git-diff, which can take gigantic
amounts of CPU time and/or heap given the right (ermm... wrong)
input.  Limit the damage that large/expensive diffs can cause.

5 years agoviewvcs: match 8000-byte lookup for git
Eric Wong [Sun, 24 Feb 2019 02:55:40 +0000 (02:55 +0000)]
viewvcs: match 8000-byte lookup for git

No need to scan the entire string, but prefer to match git
behavior.  This might be faster if/when Perl can create
substrings efficiently using CoW.

Fix a 80-column violation while we're at it.

5 years agoviewvcs: preliminary support for showing non-blobs
Eric Wong [Thu, 14 Feb 2019 02:29:10 +0000 (02:29 +0000)]
viewvcs: preliminary support for showing non-blobs

Eventually, we'll have special displays for various git objects
(commit, tree, tag).  But for now, we'll just use git-show
to spew whatever comes from git.

5 years agogit: add "commit_title" method
Eric Wong [Sun, 17 Feb 2019 05:09:50 +0000 (05:09 +0000)]
git: add "commit_title" method

This will be useful for extracting titles/subjects from
commit objects when displaying commits.

5 years agogithttpbackend: serve $GIT_DIR/info/attributes
Eric Wong [Mon, 1 Apr 2019 23:54:40 +0000 (23:54 +0000)]
githttpbackend: serve $GIT_DIR/info/attributes

This will be useful for reproducibility when mirroring
coderepos and generating diffs.

5 years agoviewvcs: check for premature EOF from git-cat-file
Eric Wong [Tue, 2 Apr 2019 00:45:48 +0000 (00:45 +0000)]
viewvcs: check for premature EOF from git-cat-file

Not entirely sure what is causing this, but it appears to
be causing infinite loops when attempting to display certain
blobs.

Fortunately, the fair scheduling of public-inbox-httpd prevented
this from becoming a real problem aside from increasing CPU
usage.

5 years agoconfig: ignore missing config files
Eric Wong [Fri, 8 Mar 2019 22:11:02 +0000 (22:11 +0000)]
config: ignore missing config files

There's no reason for us to have git-config(1) warn users when a
config file is entirely missing.

5 years agov2writable: index_sync adds new epochs to alternates
Eric Wong [Fri, 8 Mar 2019 22:52:17 +0000 (22:52 +0000)]
v2writable: index_sync adds new epochs to alternates

Newly-cloned epochs need to be in alternates file of
all.git for the web and NNTP interfaces to work.  So
allow invocations of "public-inbox-index" to idempotently
ensure the epoch is visible from the all.git repo.

5 years agov2writable: hoist out fill_alternates
Eric Wong [Fri, 8 Mar 2019 22:52:16 +0000 (22:52 +0000)]
v2writable: hoist out fill_alternates

We'll be using this sub to fill $GIT_DIR/objects/info/alternates
if somebody uses clone --mirror, too

5 years agov2writable: fix epoch rollover on incremental imports
Eric Wong [Wed, 27 Feb 2019 20:25:36 +0000 (20:25 +0000)]
v2writable: fix epoch rollover on incremental imports

All of our internal epoch rollover calculations are done using
the estimated unpacked (and uncompressed) size of the repo.  The
importer instance needs to check that unpacked size before
selecting an epoch when an epoch already has packed data.

This bug did not impact the initial mass imports since we only
initialize the Import instance once-per-epoch and did not need
to take existing epochs into account.

Tested manually with -mda on a local clone of LKML

Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
5 years agowww: prevent '!important' in BOFH-specified CSS
Eric Wong [Sat, 23 Feb 2019 10:56:38 +0000 (10:56 +0000)]
www: prevent '!important' in BOFH-specified CSS

CSS specified by the BOFH must never take precedence over
what a user sets in userContent.css.

5 years agousercontent.pm: set "!important" to override BOFH prefs
Eric Wong [Sat, 23 Feb 2019 10:56:37 +0000 (10:56 +0000)]
usercontent.pm: set "!important" to override BOFH prefs

The sample userContent.css needs a higher priority than what
the BOFH specifies.  In other words, user preference must
ALWAYS take precedence.

Reported-by: Dmitry Alexandrov <321942@gmail.com>
cf. https://public-inbox.org/meta/87mumn4kx8.fsf@gmail.com/

5 years agoensure bytes::length is available to callers
Eric Wong [Wed, 13 Feb 2019 22:44:42 +0000 (22:44 +0000)]
ensure bytes::length is available to callers

We were relying on Danga::Socket using the "bytes" pragma,
previously.  Nowadays, the "bytes" pragma is not recommended in
general, but bytes::length remains acceptable for getting the
byte-size of a scalar.

5 years agoMANIFEST: add newswww.psgi
Eric Wong [Tue, 12 Feb 2019 23:01:52 +0000 (23:01 +0000)]
MANIFEST: add newswww.psgi

Fixes: 285b9b4d7de53b0d ("examples/newswww.psgi: demonstrate standalone NewsWWW usage")
5 years agot/perf-msgview.t: fix broken performance test
Eric Wong [Wed, 6 Feb 2019 23:22:56 +0000 (23:22 +0000)]
t/perf-msgview.t: fix broken performance test

WwwStream started depending on the WWW::style method
for configurable CSS, so mock ::style so the benchmark
runs properly.

Fixes: f026dbdd392c9dd5 ('www: admin-configurable CSS via "publicinbox.css"')
5 years agot/perf-msgview: don't warn about --unordered if skipping
Eric Wong [Wed, 6 Feb 2019 23:12:10 +0000 (23:12 +0000)]
t/perf-msgview: don't warn about --unordered if skipping

No point in making noise about something that isn't used.

5 years agoexamples/newswww.psgi: demonstrate standalone NewsWWW usage
Eric Wong [Mon, 4 Feb 2019 11:00:26 +0000 (11:00 +0000)]
examples/newswww.psgi: demonstrate standalone NewsWWW usage

Plack::Builder allows "mounting" on with hostnames as well as
path names to enable virtual hosting.  This example demonstrates
how port 80/443 for "news.example.com" can redirect browser
requests when somebody attempts to use a "nntp://" URL and
the software assumes "http://"

5 years agonntp: get rid of long_response_limit constant
Eric Wong [Wed, 6 Feb 2019 01:15:55 +0000 (01:15 +0000)]
nntp: get rid of long_response_limit constant

Unused since commit b8c41362f2a5c8fcc6b1846a79c72bfa77565297
("nntp: simplify the long_response API")

5 years agohttp: cleanup partial-write handling on readonly values
Eric Wong [Wed, 6 Feb 2019 10:33:21 +0000 (10:33 +0000)]
http: cleanup partial-write handling on readonly values

Don't bother assigning to $_[1]; just let Danga::Socket
do its thing since $_[1] should be out-of-scope soon.

5 years agodaemon: favor Socket.pm in Perl 5.14+ for IPv6
Eric Wong [Wed, 6 Feb 2019 11:07:41 +0000 (11:07 +0000)]
daemon: favor Socket.pm in Perl 5.14+ for IPv6

Users on Perl 5.14+ are common, so we can try the bundled Socket
(not "Socket6") module before attempting Socket6 for IPv6.

5 years agodaemon: die properly when Socket6 is missing
Eric Wong [Wed, 6 Feb 2019 11:07:40 +0000 (11:07 +0000)]
daemon: die properly when Socket6 is missing

We need to keep Unix-socket-only httpd instances working
without Socket6.  This fixes t/httpd-unix.t with Socket6
uninstalled.

5 years agoMerge remote-tracking branch 'origin/help-color'
Eric Wong [Tue, 5 Feb 2019 22:24:19 +0000 (22:24 +0000)]
Merge remote-tracking branch 'origin/help-color'

* origin/help-color:
  wwwtext: inline sample CSS and use highlight
  hlmod: support "```$LANG" blocks in text
  hlmod: do_hl* performs src_escape immediately
  hlmod: make into a singleton
  hlmod: hoist out do_hl_lang sub
  viewvcs: cleanup utf8 handling

5 years agosolvergit: include the $oid_want tmpdir name
Eric Wong [Tue, 5 Feb 2019 21:33:16 +0000 (21:33 +0000)]
solvergit: include the $oid_want tmpdir name

This can help admins diagnose problems with SolverGit, since
qspawn logs the failed "git apply" command-line in stderr.
(or it can waste admins' time because sometimes there's crap
 mail clients which mangle patches)

5 years agoTODO: add item for "scraper" importers
Eric Wong [Tue, 5 Feb 2019 11:25:11 +0000 (11:25 +0000)]
TODO: add item for "scraper" importers

The git-users mailing list is on Google Groups with obfuscated
addresses and censored archives.  We should allow users to
import them soon, as obfuscated/censored archives are better
than not having archives at all when Google decides to shut down
yet-another-service.

There's also some mangling that (most) instances of Mailman do
(e.g. cgit), but being able to follow such groups over NNTP
and use our search functionality is still useful and better
than what typical Mailman installations provide.

5 years agowwwtext: inline sample CSS and use highlight
Eric Wong [Tue, 5 Feb 2019 05:28:03 +0000 (05:28 +0000)]
wwwtext: inline sample CSS and use highlight

For user documentation regarding CSS; showing users the sample
CSS with comments is probably more helpful than having
standalone documentation on CSS classes.

5 years agohlmod: support "```$LANG" blocks in text
Eric Wong [Tue, 5 Feb 2019 05:24:39 +0000 (05:24 +0000)]
hlmod: support "```$LANG" blocks in text

This is compatible with Markdown; but we still keep the WYSIWYG
nature of plain-text with this.  This is only intended for use
with our documentation.  Enabling any type of Markdown support
for emails can lead to incompatibilities or interopability
problems with alternative implementations.

5 years agohlmod: do_hl* performs src_escape immediately
Eric Wong [Sat, 2 Feb 2019 16:46:47 +0000 (16:46 +0000)]
hlmod: do_hl* performs src_escape immediately

We want to be able to take advantage of this in other modules

5 years agohlmod: make into a singleton
Eric Wong [Sat, 2 Feb 2019 07:35:06 +0000 (07:35 +0000)]
hlmod: make into a singleton

It turns out there's no point in having multiple instances of
this or having to worry about destruction or destruction
ordering.

This will make it easier to reuse the one instance we have
across different modules.

5 years agohlmod: hoist out do_hl_lang sub
Eric Wong [Sat, 2 Feb 2019 06:02:41 +0000 (06:02 +0000)]
hlmod: hoist out do_hl_lang sub

We'll want to use to support highlighting syntax used by
Markdown and possibly other markup languages (while retaining
the raw plain-text layout and formatting).

5 years agoviewvcs: cleanup utf8 handling
Eric Wong [Fri, 1 Feb 2019 22:12:52 +0000 (22:12 +0000)]
viewvcs: cleanup utf8 handling

Favor in-place utf8::decode since it's a bit faster without
method dispatch overhead; and don't care about validity just
yet.

HlMod->do_hl itself should return "utf8" strings, since other
parts of our code can use it, so it's not the job of ViewVCS to
post-process HlMod output.

5 years agoTODO: support migration to v2 without breaking v1 "git fetch"
Eric Wong [Mon, 4 Feb 2019 22:37:32 +0000 (22:37 +0000)]
TODO: support migration to v2 without breaking v1 "git fetch"

I'd like to move https://public-inbox.org/git/ to v2; but
cronjobs using "git fetch" to following should not break.

5 years agoviewdiff: group path match to not be confused by "/dev/null"
Eric Wong [Mon, 4 Feb 2019 11:40:13 +0000 (11:40 +0000)]
viewdiff: group path match to not be confused by "/dev/null"

Leaving out parentheses caused transitions to state="del" or
state="add" to be misidentified.

cf. https://public-inbox.org/meta/20190204105454.GG10587@szeder.dev/

Reported-by: SZEDER Gábor <szeder.dev@gmail.com>
5 years agonewswww: add /$MESSAGE_ID global redirector endpoint
Eric Wong [Wed, 9 Jan 2019 11:43:27 +0000 (11:43 +0000)]
newswww: add /$MESSAGE_ID global redirector endpoint

This is the fallback for the normal WWW endpoint.

Adding this to the top-level seems to be alright, since lynx and
w3m both understand nntp://<HOSTNAME>/<Message-ID> anyways.

If newsgroup and inbox names conflict, then consider it the
fault of the original sender.

Since NewsWWW is intended to support buggy linkifiers in mail clients,
they can interpret nntp:// URLs as http://<HOSTNAME>/<Message-ID>

Inbox ordering from the config file is preserved since
commit cfa8ff7c256e20f3240aed5f98d155c019788e3b
("config: each_inbox iteration preserves config order"),
so admins can rely on that to configure how scanning
works.

Requested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
cf. https://public-inbox.org/meta/20190107190719.GE9442@pure.paranoia.local/
    nntp://news.public-inbox.org/20190107190719.GE9442@pure.paranoia.local

5 years agoviewdiff: support renames and long paths in diffstat anchors
Eric Wong [Fri, 1 Feb 2019 07:36:02 +0000 (07:36 +0000)]
viewdiff: support renames and long paths in diffstat anchors

This is best-effort, but works well-enough in practice for
projects which use shell-friendly filenames as well as the
long path names for some Linux kernel selftests.

5 years agoviewdiff: escape HTML ampersand for renames
Eric Wong [Fri, 1 Feb 2019 06:30:17 +0000 (06:30 +0000)]
viewdiff: escape HTML ampersand for renames

For URLs we generate, we need to escape '&' in query parameters
for correctness.

5 years agoviewvcs: allow '0' as a valid filename for blob downloads
Eric Wong [Fri, 1 Feb 2019 06:06:04 +0000 (06:06 +0000)]
viewvcs: allow '0' as a valid filename for blob downloads

Only to be pedantic...

5 years agolinkify: support proto://hostname without trailing slash
Eric Wong [Fri, 1 Feb 2019 05:27:52 +0000 (05:27 +0000)]
linkify: support proto://hostname without trailing slash

Sometimes users will write "http://example.com" without the
trailing slash, which every browser and tool I've tested seems
to understand.

5 years agoview: simplify quote splitting
Eric Wong [Fri, 1 Feb 2019 03:06:47 +0000 (03:06 +0000)]
view: simplify quote splitting

Perl "split" can capture and group in the regexp itself,
so rely on that to shorten our code.

Comparing the /T/ HTML output of a thread from hell (on LKML with
1356 messages) reveals no difference in the rendered result.
Only the HTML source differs in newline placement before/after
the closing </span>

This allows a minor speedup on my X32 Thinkpad @ 1.6GHz with
the aforementioned LKML thread from hell:

before: 3.67s
 after: 3.55s

5 years agoview: fix broken hunk header hrefs in Atom feeds
Eric Wong [Fri, 1 Feb 2019 01:47:31 +0000 (01:47 +0000)]
view: fix broken hunk header hrefs in Atom feeds

We use absolute URLs in the Atom feeds (to ease
syndication/mirroring), so hunk headers need to point to the
solver URLs.

5 years agoview: diffstat anchors for multi-message/attachment views
Eric Wong [Thu, 31 Jan 2019 11:33:49 +0000 (11:33 +0000)]
view: diffstat anchors for multi-message/attachment views

diffstat <-> ^diff anchors work within the same attachment or
message while in HTML views which display multiple messages.

5 years agoviewdiff: diffstat links to diff anchors
Eric Wong [Thu, 31 Jan 2019 10:45:55 +0000 (10:45 +0000)]
viewdiff: diffstat links to diff anchors

This can be helpful for reviewing larger patches which span
across several files on the permalink (/$MESSAGE_ID/) HTML
page.

More work will be needed to get this working for the /T/ and /t/
pages which show multiple emails, as the filename-based anchors
will conflict at the moment.

5 years agohval: routines for attribute escaping
Eric Wong [Mon, 18 Jan 2016 21:16:14 +0000 (21:16 +0000)]
hval: routines for attribute escaping

We'll use HTML attributes + anchor links to link to filenames
in coming commits.

5 years agoMerge remote-tracking branch 'origin/purge'
Eric Wong [Thu, 31 Jan 2019 21:08:48 +0000 (21:08 +0000)]
Merge remote-tracking branch 'origin/purge'

* origin/purge:
  implement public-inbox-purge tool
  v2writable: read epoch on purge
  v2writable: cleanup processes when done
  v2writable: purge ignores non-existent git epoch directories
  v2writable: ->purge returns undef on no-op
  import: purge: reap fast-export process
  hoist out resolve_repo_dir from -index

5 years agodoc: remove completed TODO items
Eric Wong [Thu, 31 Jan 2019 09:04:12 +0000 (09:04 +0000)]
doc: remove completed TODO items

5 years agodoc/config: document "replyto" configuration knob
Eric Wong [Thu, 31 Jan 2019 09:17:45 +0000 (09:17 +0000)]
doc/config: document "replyto" configuration knob

I hate it, but it's necessary to support some mirrors.

5 years agodoc/config: user documentation for limiters
Eric Wong [Thu, 31 Jan 2019 08:52:49 +0000 (08:52 +0000)]
doc/config: user documentation for limiters

I've relied on this feature to keep the VPS behind
https://public-inbox.org/git/ from OOM-ing since 2016,
so document it to ensure others can make use of low-end
servers like I do.

More limiters may become configurable for viewvcs and
solver functionality (or we continue using the default
one).

5 years agoconfig: tiny cleanup to use _array() sub
Eric Wong [Thu, 31 Jan 2019 08:28:41 +0000 (08:28 +0000)]
config: tiny cleanup to use _array() sub

5 years agoqspawn: documentation updates
Eric Wong [Thu, 31 Jan 2019 08:17:15 +0000 (08:17 +0000)]
qspawn: documentation updates

This will become critical for future changes to display
git commits, diffs, and trees.

Use "qspawn.wcb" instead of "qspawn.response" to enhance
readability.

5 years agoinbox: drop psgi.url_scheme requirement from base_url
Eric Wong [Thu, 31 Jan 2019 06:00:22 +0000 (06:00 +0000)]
inbox: drop psgi.url_scheme requirement from base_url

This will make it easier to make command-line tools
from SolverGit.

5 years agoviewvcs: support streaming large blobs
Eric Wong [Thu, 31 Jan 2019 04:10:29 +0000 (04:10 +0000)]
viewvcs: support streaming large blobs

Forking off git-cat-file here for streaming large blobs is
reasonably efficient, at least no worse than using
git-http-backend for serving clones.  So let our limiter
framework deal with it.

git itself isn't great for large files, and AFAIK there's no
stable/widely-available mechanisms for reading smaller chunks
of giant blobs in git itself.

Tested with some giant GPU headers in the Linux kernel.

5 years agosolvergit: allow shorter-than-necessary OIDs from user
Eric Wong [Thu, 31 Jan 2019 02:46:13 +0000 (02:46 +0000)]
solvergit: allow shorter-than-necessary OIDs from user

We can rely on git to disambiguate, here; because sometimes
shorter OIDs can be unambiguous even if we only resolved the
longer one.

5 years agosolvergit: allow searching on longer-than-needed OIDs
Eric Wong [Thu, 31 Jan 2019 01:41:15 +0000 (01:41 +0000)]
solvergit: allow searching on longer-than-needed OIDs

public-inbox can only index the abbreviated object_ids in
emails, not the full or even longer-than-necessary object_ids.
So retry failed object_ids if they're longer than 7 hex
characters.

5 years agoinbox: perform cleanup of Git objects for coderepos
Eric Wong [Thu, 31 Jan 2019 00:32:42 +0000 (00:32 +0000)]
inbox: perform cleanup of Git objects for coderepos

Otherwise, long-running but idle git processes may keep unlinked
packs around indefinitely and waste disk space.

5 years agot/config.t: test PublicInbox::Git sharing between inboxes
Eric Wong [Wed, 30 Jan 2019 20:48:53 +0000 (20:48 +0000)]
t/config.t: test PublicInbox::Git sharing between inboxes

We need to ensure we don't introduce unnecessary processes
and memory usage for mapping multiple inboxes to the same
code repos.

5 years agodoc/config: document coderepo and css bits
Eric Wong [Wed, 30 Jan 2019 11:38:52 +0000 (11:38 +0000)]
doc/config: document coderepo and css bits

New features ought to be documented

5 years agosolvergit: don't confuse Xapian with ".." in filenames
Eric Wong [Wed, 30 Jan 2019 10:33:55 +0000 (10:33 +0000)]
solvergit: don't confuse Xapian with ".." in filenames

Xapian will interpret ".." as ranges, even quoted phrases.
So break up words on ".." since punctuation (AFAIK) is not
searchable, anyways.

5 years agogit: use "git rev-parse --git-path"
Eric Wong [Wed, 30 Jan 2019 08:47:43 +0000 (08:47 +0000)]
git: use "git rev-parse --git-path"

Using git worktrees was causing t/solver_git.t to fail on me.

5 years agoMerge remote-tracking branch 'origin/viewvcs' into master
Eric Wong [Wed, 30 Jan 2019 07:40:35 +0000 (07:40 +0000)]
Merge remote-tracking branch 'origin/viewvcs' into master

* origin/viewvcs: (66 commits)
  solvergit: deal with alternative diff prefixes
  solvergit: extract mode from diff headers properly
  solvergit: avoid "Wide character" warnings
  solvergit: do not show full path names to "git apply"
  css/216dark: add comments and tweak highlight colors
  viewvcs: avoid segfault with highlight.pm at shutdown
  solvergit: do not solve blobs twice
  t/check-www-inbox: disable history
  t/check-www-inbox: don't follow mboxes
  t/check-www-inbox: replace IPC::Run with PublicInbox::Spawn
  hval: add src_escape for highlight post-processing
  viewvcs: wire up syntax-highlighting for blobs
  hlmod: disable enclosing <pre> tag
  t/hl_mod: extra check to ensure we escape HTML
  wwwhighlight: read_in_full returns undef on errors
  solver: crank up max patches to 9999
  viewvcs: do not show final error message twice
  qspawn: decode $? for user-friendliness
  solver: reduce "git apply" invocations
  solver: hold patches in temporary directory
  ...

5 years agoview: remove unused _msg_date sub
Eric Wong [Wed, 30 Jan 2019 07:38:51 +0000 (07:38 +0000)]
view: remove unused _msg_date sub

Not needed since commit 956abe9ad5f13a0d1755262be412d6a54fda72e9
("view: depend on SearchMsg for Message-ID")

5 years agohttpd: a few comments about some fields we set
Eric Wong [Wed, 30 Jan 2019 07:39:02 +0000 (07:39 +0000)]
httpd: a few comments about some fields we set

Removing 'psgix.input.buffered' could be a possibility in
the future.

5 years agosolvergit: deal with alternative diff prefixes
Eric Wong [Wed, 30 Jan 2019 04:38:26 +0000 (04:38 +0000)]
solvergit: deal with alternative diff prefixes

At least, without extra directory levels, since
git-diff supports --src-prefix and --dst-prefix,
and /git/6aa8857a11/s/ uses it...

5 years agosolvergit: extract mode from diff headers properly
Eric Wong [Wed, 30 Jan 2019 04:29:24 +0000 (04:29 +0000)]
solvergit: extract mode from diff headers properly

grep() won't set $1, so use "=~", instead.