Eric Wong [Wed, 24 Apr 2019 19:52:14 +0000 (19:52 +0000)]
wwwhighlight: do linkification, too
Otherwise, there's no reason to use this API over highlight(1).
Maybe this can be an option in the future; but I'm struggling to
find a reason to not do it by default.
Eric Wong [Thu, 18 Apr 2019 00:58:06 +0000 (00:58 +0000)]
inbox: add `modified' sub
For inboxes with SQLite enabled (all v2, and probably most v1);
we can use the overview DB to get the timestamp of the latest
message. It's faster than scanning git branches for commit times,
but not always the same.
Eric Wong [Wed, 17 Apr 2019 10:49:15 +0000 (10:49 +0000)]
git: calculate modified time of repository
This will be used for generating an HTML listing for v1 inboxes,
at least. The logic for this follows that of grokmirror,
and we may dynamically generate manifest.js.gz natively...
Eric Wong [Mon, 15 Apr 2019 07:32:49 +0000 (07:32 +0000)]
HACKING: stop advertising mailing list subscription info
It's still available, but no point in advertising something
which increases the dependency on a centralized subscriber list.
In fact, it's actively against the goal of promoting resistance
against centralization.
Eric Wong [Tue, 16 Apr 2019 01:10:09 +0000 (01:10 +0000)]
cleanup: use '$ibx' consistently when referring to Inbox refs
'$inbox' is more human-readable, so that is for the more
human-readable name in most cases. Making our variable naming
more consistent should make the code easier-to-review and
harder to screw up.
Eric Wong [Fri, 5 Apr 2019 20:04:27 +0000 (20:04 +0000)]
config: support more cgit directives for project lists
Hopefully this gets us closer to matching cgit upstream behavior
(which also lacks tests). We'll still need to support macro
expansion at some point for compatibility...
Eric Wong [Fri, 5 Apr 2019 20:04:26 +0000 (20:04 +0000)]
cgit: serve static css, logo, favicon directly
We can reduce the configuration needed to run cgit by reusing
the static file handling logic of the dumb git HTTP protocol.
I hate logos and icons, so don't expect public-inbox.org or
80x24.org to ever have those to waste users' bandwidth with :P
But I expect other users to find this useful.
Eric Wong [Sun, 10 Mar 2019 22:02:29 +0000 (22:02 +0000)]
cgit: support running cgit as a standalone CGI
We depend on git-http-backend for smart HTTP clone support,
however; since cgit does not support smart clones natively.
WWW.pm will be able to cascade down to this as a 404 handler in
the future.
Eric Wong [Sun, 10 Mar 2019 07:51:45 +0000 (07:51 +0000)]
githttpbackend: move more psgi.input handling into subroutine
This will be useful for other CGI wrappers we make.
This also fixes a bug with some PSGI servers which did not
present a real IO::Handle in the psgi.input env field.
Eric Wong [Mon, 25 Feb 2019 05:14:10 +0000 (05:14 +0000)]
spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE
We'll be spawning cgit and git-diff, which can take gigantic
amounts of CPU time and/or heap given the right (ermm... wrong)
input. Limit the damage that large/expensive diffs can cause.
Eric Wong [Thu, 14 Feb 2019 02:29:10 +0000 (02:29 +0000)]
viewvcs: preliminary support for showing non-blobs
Eventually, we'll have special displays for various git objects
(commit, tree, tag). But for now, we'll just use git-show
to spew whatever comes from git.
Eric Wong [Fri, 8 Mar 2019 22:52:17 +0000 (22:52 +0000)]
v2writable: index_sync adds new epochs to alternates
Newly-cloned epochs need to be in alternates file of
all.git for the web and NNTP interfaces to work. So
allow invocations of "public-inbox-index" to idempotently
ensure the epoch is visible from the all.git repo.
Eric Wong [Wed, 27 Feb 2019 20:25:36 +0000 (20:25 +0000)]
v2writable: fix epoch rollover on incremental imports
All of our internal epoch rollover calculations are done using
the estimated unpacked (and uncompressed) size of the repo. The
importer instance needs to check that unpacked size before
selecting an epoch when an epoch already has packed data.
This bug did not impact the initial mass imports since we only
initialize the Import instance once-per-epoch and did not need
to take existing epochs into account.
Tested manually with -mda on a local clone of LKML
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Eric Wong [Wed, 13 Feb 2019 22:44:42 +0000 (22:44 +0000)]
ensure bytes::length is available to callers
We were relying on Danga::Socket using the "bytes" pragma,
previously. Nowadays, the "bytes" pragma is not recommended in
general, but bytes::length remains acceptable for getting the
byte-size of a scalar.
Plack::Builder allows "mounting" on with hostnames as well as
path names to enable virtual hosting. This example demonstrates
how port 80/443 for "news.example.com" can redirect browser
requests when somebody attempts to use a "nntp://" URL and
the software assumes "http://"
Eric Wong [Tue, 5 Feb 2019 22:24:19 +0000 (22:24 +0000)]
Merge remote-tracking branch 'origin/help-color'
* origin/help-color:
wwwtext: inline sample CSS and use highlight
hlmod: support "```$LANG" blocks in text
hlmod: do_hl* performs src_escape immediately
hlmod: make into a singleton
hlmod: hoist out do_hl_lang sub
viewvcs: cleanup utf8 handling
Eric Wong [Tue, 5 Feb 2019 21:33:16 +0000 (21:33 +0000)]
solvergit: include the $oid_want tmpdir name
This can help admins diagnose problems with SolverGit, since
qspawn logs the failed "git apply" command-line in stderr.
(or it can waste admins' time because sometimes there's crap
mail clients which mangle patches)
Eric Wong [Tue, 5 Feb 2019 11:25:11 +0000 (11:25 +0000)]
TODO: add item for "scraper" importers
The git-users mailing list is on Google Groups with obfuscated
addresses and censored archives. We should allow users to
import them soon, as obfuscated/censored archives are better
than not having archives at all when Google decides to shut down
yet-another-service.
There's also some mangling that (most) instances of Mailman do
(e.g. cgit), but being able to follow such groups over NNTP
and use our search functionality is still useful and better
than what typical Mailman installations provide.
Eric Wong [Tue, 5 Feb 2019 05:28:03 +0000 (05:28 +0000)]
wwwtext: inline sample CSS and use highlight
For user documentation regarding CSS; showing users the sample
CSS with comments is probably more helpful than having
standalone documentation on CSS classes.
Eric Wong [Tue, 5 Feb 2019 05:24:39 +0000 (05:24 +0000)]
hlmod: support "```$LANG" blocks in text
This is compatible with Markdown; but we still keep the WYSIWYG
nature of plain-text with this. This is only intended for use
with our documentation. Enabling any type of Markdown support
for emails can lead to incompatibilities or interopability
problems with alternative implementations.
Eric Wong [Sat, 2 Feb 2019 06:02:41 +0000 (06:02 +0000)]
hlmod: hoist out do_hl_lang sub
We'll want to use to support highlighting syntax used by
Markdown and possibly other markup languages (while retaining
the raw plain-text layout and formatting).
Eric Wong [Fri, 1 Feb 2019 22:12:52 +0000 (22:12 +0000)]
viewvcs: cleanup utf8 handling
Favor in-place utf8::decode since it's a bit faster without
method dispatch overhead; and don't care about validity just
yet.
HlMod->do_hl itself should return "utf8" strings, since other
parts of our code can use it, so it's not the job of ViewVCS to
post-process HlMod output.
Eric Wong [Wed, 9 Jan 2019 11:43:27 +0000 (11:43 +0000)]
newswww: add /$MESSAGE_ID global redirector endpoint
This is the fallback for the normal WWW endpoint.
Adding this to the top-level seems to be alright, since lynx and
w3m both understand nntp://<HOSTNAME>/<Message-ID> anyways.
If newsgroup and inbox names conflict, then consider it the
fault of the original sender.
Since NewsWWW is intended to support buggy linkifiers in mail clients,
they can interpret nntp:// URLs as http://<HOSTNAME>/<Message-ID>
Inbox ordering from the config file is preserved since
commit cfa8ff7c256e20f3240aed5f98d155c019788e3b
("config: each_inbox iteration preserves config order"),
so admins can rely on that to configure how scanning
works.
Requested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
cf. https://public-inbox.org/meta/20190107190719.GE9442@pure.paranoia.local/
nntp://news.public-inbox.org/20190107190719.GE9442@pure.paranoia.local
Eric Wong [Fri, 1 Feb 2019 07:36:02 +0000 (07:36 +0000)]
viewdiff: support renames and long paths in diffstat anchors
This is best-effort, but works well-enough in practice for
projects which use shell-friendly filenames as well as the
long path names for some Linux kernel selftests.
Eric Wong [Fri, 1 Feb 2019 03:06:47 +0000 (03:06 +0000)]
view: simplify quote splitting
Perl "split" can capture and group in the regexp itself,
so rely on that to shorten our code.
Comparing the /T/ HTML output of a thread from hell (on LKML with
1356 messages) reveals no difference in the rendered result.
Only the HTML source differs in newline placement before/after
the closing </span>
This allows a minor speedup on my X32 Thinkpad @ 1.6GHz with
the aforementioned LKML thread from hell:
Eric Wong [Thu, 31 Jan 2019 10:45:55 +0000 (10:45 +0000)]
viewdiff: diffstat links to diff anchors
This can be helpful for reviewing larger patches which span
across several files on the permalink (/$MESSAGE_ID/) HTML
page.
More work will be needed to get this working for the /T/ and /t/
pages which show multiple emails, as the filename-based anchors
will conflict at the moment.
Eric Wong [Thu, 31 Jan 2019 08:52:49 +0000 (08:52 +0000)]
doc/config: user documentation for limiters
I've relied on this feature to keep the VPS behind
https://public-inbox.org/git/ from OOM-ing since 2016,
so document it to ensure others can make use of low-end
servers like I do.
More limiters may become configurable for viewvcs and
solver functionality (or we continue using the default
one).
Eric Wong [Thu, 31 Jan 2019 04:10:29 +0000 (04:10 +0000)]
viewvcs: support streaming large blobs
Forking off git-cat-file here for streaming large blobs is
reasonably efficient, at least no worse than using
git-http-backend for serving clones. So let our limiter
framework deal with it.
git itself isn't great for large files, and AFAIK there's no
stable/widely-available mechanisms for reading smaller chunks
of giant blobs in git itself.
Tested with some giant GPU headers in the Linux kernel.
Eric Wong [Thu, 31 Jan 2019 01:41:15 +0000 (01:41 +0000)]
solvergit: allow searching on longer-than-needed OIDs
public-inbox can only index the abbreviated object_ids in
emails, not the full or even longer-than-necessary object_ids.
So retry failed object_ids if they're longer than 7 hex
characters.
Eric Wong [Wed, 30 Jan 2019 07:40:35 +0000 (07:40 +0000)]
Merge remote-tracking branch 'origin/viewvcs' into master
* origin/viewvcs: (66 commits)
solvergit: deal with alternative diff prefixes
solvergit: extract mode from diff headers properly
solvergit: avoid "Wide character" warnings
solvergit: do not show full path names to "git apply"
css/216dark: add comments and tweak highlight colors
viewvcs: avoid segfault with highlight.pm at shutdown
solvergit: do not solve blobs twice
t/check-www-inbox: disable history
t/check-www-inbox: don't follow mboxes
t/check-www-inbox: replace IPC::Run with PublicInbox::Spawn
hval: add src_escape for highlight post-processing
viewvcs: wire up syntax-highlighting for blobs
hlmod: disable enclosing <pre> tag
t/hl_mod: extra check to ensure we escape HTML
wwwhighlight: read_in_full returns undef on errors
solver: crank up max patches to 9999
viewvcs: do not show final error message twice
qspawn: decode $? for user-friendliness
solver: reduce "git apply" invocations
solver: hold patches in temporary directory
...