Eric Wong [Thu, 23 May 2019 09:36:56 +0000 (09:36 +0000)]
doc: xcpdb: update to reflect the current state
It is no longer a wrapper around copydatabase(1), since
copydatabase did not recover from DatabaseModifiedError.
Eric Wong [Thu, 23 May 2019 09:36:55 +0000 (09:36 +0000)]
xapcmd: avoid EXDEV when finalizing changes
By creating temporary directories as deep as possible,
we can allow v2 repositories to have `xap$SCHEMA_VERSION'
(e.g. `xap15') reside on a separate FS.
We also check st_dev ahead-of-time to avoid doing work which
will fail with EXDEV. Of course, another process may still
move/change things around.
Eric Wong [Thu, 23 May 2019 09:36:54 +0000 (09:36 +0000)]
xcpdb: cleanup error handling and diagnosis
Running a full "public-inbox-index --reindex" in parallel
with "public-inbox-xcpdb" on the same inbox can still cause
problems, though.
Eric Wong [Thu, 23 May 2019 09:36:53 +0000 (09:36 +0000)]
xcpdb: implement progress reporting
Copying an entire Xapian DB is horribly slow whether it's done
via Perl or copydatabase(1). So displaying some progress
indication is good for user experience.
While we're at it, prefix xapian-compact output, too; since
parallel processes end up clobbering each other.
Eric Wong [Thu, 23 May 2019 09:36:52 +0000 (09:36 +0000)]
xcpdb: use fine-grained locking
Copying an entire Xapian DB takes a long time, so update our
reindexing code to support partial reindexing, snapshot the
pre-copydatabase git revisions, perform the lengthy copy,
and do a partial reindex when the copy + renames are done.
Eric Wong [Thu, 23 May 2019 09:36:51 +0000 (09:36 +0000)]
v2writable: hoist out log_range sub for readability
This is preparation to to support partial reindexing
Eric Wong [Thu, 23 May 2019 09:36:50 +0000 (09:36 +0000)]
xapcmd: xcpdb supports compaction
To minimize the delay on active inboxes, it's actually ideal to
run xapian-compact at the end of the per-partition cpdb process;
since the new DB isn't accessible yet and so we don't have to
deal with lock contention with -mda or -watch processes. The
downside is temporary file overhead (3x instead of 2x) required.
Eric Wong [Thu, 23 May 2019 09:36:49 +0000 (09:36 +0000)]
xcpdb: implement using Perl bindings
By avoid copydatabase(1) entirely, we can make further changes
to avoid locking the entire inbox for a long operation and
switch to fine-grained locking.
Eric Wong [Thu, 23 May 2019 09:36:48 +0000 (09:36 +0000)]
admin: move index_inbox over
We will be reindexing after copydatabase
Eric Wong [Thu, 23 May 2019 09:36:47 +0000 (09:36 +0000)]
xapcmd: do not cleanup on errors
We move the old directory into the new directory, so avoid the
situation where a bug or error could cause the tempdir cleanup to run
and destroy both our old and new directories.
Eric Wong [Thu, 23 May 2019 09:36:46 +0000 (09:36 +0000)]
xcpdb: new tool which wraps Xapian's copydatabase(1)
copydatabase(1) is an existing Xapian tool which is the
recommended way to upgrade existing DBs to the latest Xapian
database format (currently "glass" for stable/released
versions). Our use of Xapian relies on preserving document IDs,
so we'll wrap it like we do xapian-compact(1) and use the
"--no-renumber" switch.
I could not name the tool "public-inbox-copydatabase" since it
would be ambiguous as to which DB it's actually copying. So, I
abbreviated the suffix to "xcpdb" (Xapian CoPy DataBase), which
I hope is acceptable and unambiguous.
Eric Wong [Thu, 23 May 2019 09:36:45 +0000 (09:36 +0000)]
xapcmd: support spawn options
copydatabase(1) is exceptionally noisy and it's output is
confusing when run in parallel. Support redirects at least, and
env while we're at it to give us future options.
We can also stuff a -jobs parameter into the options to limit
parallelism since it can be useful for low-priority upgrade
jobs.
Eric Wong [Thu, 23 May 2019 09:36:44 +0000 (09:36 +0000)]
admin: hoist out resolve_inboxes for -compact and -index
Both of these index-affecting commands should work similarly
on the command-line.
public-inbox-index no longer complains about unconfigured
~/.public-inbox/config; but often I found myself being
annoyed by that, anyways...
Eric Wong [Thu, 23 May 2019 09:36:43 +0000 (09:36 +0000)]
xapcmd: new module for wrapping Xapian commands
Port public-inbox-compact(1) over to using it, and we will need
to wrap copydatabase(1) to ease glass migrations, too.
Eric Wong [Thu, 23 May 2019 09:36:42 +0000 (09:36 +0000)]
search: reenable phrase search on non-chert Xapian
This is assuming nobody uses flint or earlier, anymore;
as flint predates the existence of this project.
Eric Wong [Thu, 23 May 2019 09:36:41 +0000 (09:36 +0000)]
doc: document the reason for --no-renumber
We're going to need copydatabase, too
Eric Wong [Thu, 23 May 2019 09:36:40 +0000 (09:36 +0000)]
v1writable: retire in favor of InboxWritable
In retrospect, introducing V1Writable was unnecessary and
InboxWritable->importer is in a better position to abstract
away differences between v1 and v2 writers.
So teach InboxWritable to initialize inboxes and get rid
of V1Writable.
Eric Wong [Thu, 23 May 2019 09:36:39 +0000 (09:36 +0000)]
t/convert-compact: skip on missing xapian-compact(1)
Can't run the test if the required Xapian tools are missing.
Eric Wong [Wed, 22 May 2019 21:54:21 +0000 (21:54 +0000)]
Merge branch 'ds-cleanup'
* ds-cleanup:
DS: warn on deprecations
DS: remove IPPROTO_TCP import
DS: drop $VERSION var
DS: remove support OtherFds code
DS: get rid of unused methods and aliases
Eric Wong [Wed, 22 May 2019 21:46:59 +0000 (21:46 +0000)]
DS: warn on deprecations
Enabling deprecation warnings didn't seem to have any
noticeable effects with "perl -w -c", so whatever reason
Danga had for it is long irrelevant.
Eric Wong [Wed, 22 May 2019 21:46:58 +0000 (21:46 +0000)]
DS: remove IPPROTO_TCP import
Unlike Danga::Socket, we do not support TCP_CORK, either
Eric Wong [Wed, 22 May 2019 21:46:57 +0000 (21:46 +0000)]
DS: drop $VERSION var
It was only relevant to Danga::Socket.
Eric Wong [Wed, 22 May 2019 21:46:56 +0000 (21:46 +0000)]
DS: remove support OtherFds code
It's easy enough to wrap FDs in classes that can use
all of the functionality of the event loop, not just
the read-only interface AddOtherFds provided.
Eric Wong [Wed, 22 May 2019 21:46:55 +0000 (21:46 +0000)]
DS: get rid of unused methods and aliases
"make syntax" is clean, now
Eric Wong [Wed, 22 May 2019 07:27:52 +0000 (07:27 +0000)]
usercontent: stop relying on autodie
It's a non-standard package on CentOS-7, actually; and we
shouldn't bloat the PSGI server by loading a module which
isn't strictly needed.
Eric Wong [Wed, 22 May 2019 02:14:02 +0000 (02:14 +0000)]
ci: support CentOS-7
Tested on an amd64 chroot built with rinse 3.4
Eric Wong [Wed, 22 May 2019 02:14:01 +0000 (02:14 +0000)]
t/search*: require DBI and DBD::SQLite, too
None of the Search::Xapian-dependent stuff works without DBI
and DBD::SQLite.
There are no plans to support Xapian w/o DBD::SQLite since
SQLite is more common and less resource-intensive than Xapian.
Eric Wong [Wed, 22 May 2019 02:14:00 +0000 (02:14 +0000)]
git: workaround old git-rev-parse(1) (--git-path)
git < 2.5.0 was missing --git-path support. This means any
users relying on some rare environment variables will need git
2.5.0+
Eric Wong [Wed, 22 May 2019 02:13:59 +0000 (02:13 +0000)]
t/watch_filter_rubylang: disable v2 test for git < 2.6
This test was not disabled properly for ancient versions of
git without get-mark support.
Eric Wong [Wed, 22 May 2019 02:13:58 +0000 (02:13 +0000)]
init: preserve permissions for git prior to 2.1.0
"git config" did not preserve permissions of the config file it
modifies prior to git 2.1.0, so workaround that.
Eric Wong [Tue, 21 May 2019 23:22:05 +0000 (23:22 +0000)]
Merge remote-tracking branch 'origin/xap-optional' into master
* origin/xap-optional:
admin: improve warnings and errors for missing modules
searchidx: do not create empty Xapian partitions for basic
lazy load Xapian and make it optional for v2
www: use Inbox->over where appropriate
nntp: use Inbox->over directly
inbox: add ->over method to ease access
Eric Wong [Tue, 21 May 2019 06:54:27 +0000 (06:54 +0000)]
searchidx: remove unused Compress::Zlib import
We only compress in OverIdx, now; since we no longer do overview
stuff in Xapian (and Xapian compresses document data, anyways).
Eric W. Biederman [Fri, 17 May 2019 01:53:06 +0000 (20:53 -0500)]
PublicInbox::Import::add: Consolidate subject handling
Consolidate subject handling in the add function to make it easier to
read and understand.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
Eric Wong [Thu, 16 May 2019 07:06:21 +0000 (07:06 +0000)]
www: unescape '+' => ' ' before general URI unescape
This allows searching for terms with "+" in them properly.
Eric Wong [Thu, 16 May 2019 06:54:08 +0000 (06:54 +0000)]
Revert "view: perform highlighting for space-prefixed diffs"
This was buggy and was causing non-diff text to have extra
leading spaces. The diff parsing code needs to be cleaned up,
so this will be fixed, later.
This reverts commit
1a67b91c1326efa372d1ec957e2494849d894f0b.
Eric Wong [Thu, 16 May 2019 03:40:47 +0000 (03:40 +0000)]
search: disable phrase searching, for now
There probably needs to be an option to enable this
independently of indexlevel; but for now this is
the safest option.
And, as I discovered during the development of the
indexlevel option, Xapian does a pretty good job of
finding phrases without position data, anyways.
Eric Wong [Thu, 16 May 2019 01:23:29 +0000 (01:23 +0000)]
view: perform highlighting for space-prefixed diffs
"git format-patch --interdiff" and similar can prefix diffs
with leading white space. Teach our diff parser to account
for it and set appropriate CSS classes for them.
Eric Wong [Wed, 15 May 2019 10:39:19 +0000 (10:39 +0000)]
remove hard Devel::Peek dependency and lazy load for daemons
It's only useful for a corner case in long-running daemons when
an admin decides to compact or vacuum a Xapian or SQLite DB.
As a result, other scripts should run slightly faster. For
instance, this saves about 80ms (2.710s => 2.630s) in t/mda.t
on my remote workstation.
While we're at it, make sure EvCleanup is properly require'd
in Daemon.pm and HTTP.pm and document our use of Devel::Peek.
Eric Wong [Wed, 15 May 2019 10:39:01 +0000 (10:39 +0000)]
inbox: remove POSIX strftime import
We no longer need it since Inbox->recent hits the overview
DB instead of Xapian
Eric Wong [Thu, 9 May 2019 09:57:37 +0000 (09:57 +0000)]
doc: add marketing guide
Preventative measures; since marketing is almost always annoying
to me. And trying to avoid unintended consequences.
Eric Wong [Wed, 15 May 2019 01:18:09 +0000 (01:18 +0000)]
admin: improve warnings and errors for missing modules
Since we lazy-load Xapian now, some errors may become
more cryptic or buried. Try to improve that by making
Admin show better errors.
Eric Wong [Wed, 15 May 2019 01:18:10 +0000 (01:18 +0000)]
searchidx: do not create empty Xapian partitions for basic
No point in leaving a mess of empty directories when Xapian
doesn't load.
Eric Wong [Wed, 15 May 2019 01:18:08 +0000 (01:18 +0000)]
lazy load Xapian and make it optional for v2
More tests work without Search::Xapian, now.
Usability issues still need to be fixed
Eric Wong [Wed, 15 May 2019 01:18:07 +0000 (01:18 +0000)]
www: use Inbox->over where appropriate
We don't need to rely on Xapian search functionality for the
majority of the WWW code, even. subject_normalized is moved to
SearchMsg, where it (probably) makes more sense, anyways.
Eric Wong [Wed, 15 May 2019 01:18:06 +0000 (01:18 +0000)]
nntp: use Inbox->over directly
None of the NNTP code actually relies on Xapian, anymore.
Eric Wong [Wed, 15 May 2019 01:18:05 +0000 (01:18 +0000)]
inbox: add ->over method to ease access
One small step towards making installing Xapian optional for v2
and providing more WWW and NNTP functionality without it.
Eric Wong [Tue, 14 May 2019 03:32:03 +0000 (03:32 +0000)]
tests: get rid of unnecessary Cwd module use
We only need it for tests that chdir, and maybe for ENV{PATH}
portability (dash seems fine, not sure about others).
v2: revert change to solver_git.t for FreeBSD 11.2 and document
Eric Wong [Tue, 14 May 2019 03:32:02 +0000 (03:32 +0000)]
t/nntp.t: skip if Data::Dumper is missing
We can revisit this, later; but Data::Dumper requires a separate
package for CentOS-7 users, at least.
Eric Wong [Tue, 14 May 2019 03:32:01 +0000 (03:32 +0000)]
t/config.t: remove Data::Dumper dependency
CentOS-7 needs the perl-Data-Dumper package, and the
test is small enough to roll our own escaping, here.
Eric Wong [Tue, 14 May 2019 03:32:00 +0000 (03:32 +0000)]
tests: remove unnecessary loading of ::DS and Socket
PublicInbox::DS works for every platform we we care about,
nowadays; so checking for it is a waste of time. Cleanup a
few POSIX and Socket imports while we're in the area.
Eric Wong [Sun, 5 May 2019 20:42:20 +0000 (20:42 +0000)]
httpd: get rid of Deflater warning
Deflating responses may be done by the reverse proxy (e.g. varnish
or nginx), so the warning for it could be invalid.
Eric Wong [Tue, 14 May 2019 02:04:43 +0000 (02:04 +0000)]
searchidx: fix incremental index with indexlevel=basic on v1
We were reindexing the full history every invocation of -index
when Xapian was not used because we were incorrectly relying on
'last_commit' metadata stored in Xapian.
Rewrite the indexing logic to be less confusing while we're
at it, since we rely on `git merge-base --is-ancestor' nowadays.
Furthermore, we need to handle message removals from the
overview index correctly when Xapian is not in use.
Co-authored-by: Eric W. Biederman <ebiederm@xmission.com>
Eric Wong [Tue, 14 May 2019 02:04:42 +0000 (02:04 +0000)]
v2writable: allow setting nproc via creat options
Avoiding reliance on environment variables is a bit cleaner
for writing tests
Eric Wong [Tue, 14 May 2019 02:04:41 +0000 (02:04 +0000)]
v1writable: new wrapper which is closer to v2writable
Import initialization is a little strange from history, but we
also can't change it too much because it's technically a public
API which external code may rely on...
And we may need to support v1 repos indefinitely. This should
make it easier to write tests for both formats.
Eric Wong [Tue, 7 May 2019 09:09:54 +0000 (09:09 +0000)]
ci: include some scripts to make CI easier
This should make it easier to test a bunch of package
installation profiles across whatever OS isolation
one chooses (chroots, containers, jails, VMs).
Eric Wong [Wed, 8 May 2019 08:34:36 +0000 (08:34 +0000)]
doc/include.mk: remove git use and redundant declarations
And document that we still have GNU-isms in that
include.mk Makefile (and may continue to do so).
Finally, take advantage of GNU-isms to warn users
to run "gmake" to build all manpages.
Eric Wong [Tue, 7 May 2019 23:43:42 +0000 (23:43 +0000)]
Makefile.PL: improve make(1) portability
The `shell' function appears missing, so we'll rely on Bourne
shell expansioN, instead.
Use "$?" instead of "$<" since the latter is only specified
for inference and .DEFAULT rules, not target rules.
Tested on FreeBSD make(1) and bmake(1) on Debian.
Eric Wong [Tue, 7 May 2019 20:40:28 +0000 (20:40 +0000)]
doc: give txt files proper titles
We can fix the redundant rule in include.mk which causes
make(1) on FreeBSD to complain; but HTML docs will likely
still require GNU make.
Eric Wong [Wed, 8 May 2019 22:32:43 +0000 (22:32 +0000)]
doc: use bullet list for wwwlisting options
Otherwise, pod2man complains about "=item 404" not starting
with a letter and thinking it's part of a numbered list.
Eric Wong [Wed, 8 May 2019 19:52:36 +0000 (19:52 +0000)]
t/purge.t: fix unreferenced variable
Eric Wong [Wed, 8 May 2019 20:03:33 +0000 (20:03 +0000)]
INSTALL: note xapian-compact(1) tool
The Perl Xapian bindings do not support compacting, yet,
so we rely on the xapian-compact tool.
Eric Wong [Wed, 8 May 2019 02:47:20 +0000 (02:47 +0000)]
INSTALL: depend on Encode instead of Encode::MIME::Header
Email::MIME uses Encode::MIME::Header and depends on that
appropriately; however we depend on other parts of the Encode
distribution, but that's bundled with Perl by upstream, anyways;
and should place no additional burden on users.
Eric Wong [Sun, 5 May 2019 01:37:00 +0000 (01:37 +0000)]
INSTALL: update with FreeBSD pkg names and reorder
Fortunately, there is a pattern to most of these package names
in all distros I've tested (and hopefully other BSDs have them,
too).
Then, reorder the INSTALL document to mention the top-level
modules, first, so users can avoid needing to type extra
dependencies. However, we still list some implicit dependencies
in case the upstream package drops dependencies independently of
us.
Finally, Devel::Peek is not a dependency worth making optional
since it's bundled by Perl upstream. Fedora/RH-based distros
are the only one which turn it into a non-standard package when
Perl5 is installed.
Eric Wong [Wed, 8 May 2019 19:24:07 +0000 (19:24 +0000)]
Merge remote-tracking branch 'origin/danga-bundle'
* origin/danga-bundle:
DS: epoll: fix misordered EPOLL_CTL_DEL call
DS: drop unused "_undef" sub
syscall: drop readahead wrapper
build: do not manify DS and Syscall pods
DS: handle EINTR in IO::Poll path, too
DS: workaround IO::Kqueue EINTR (mis-)handling
DS: drop profiling support
DS: remove unused fields and functions
listener: use EPOLLEXCLUSIVE for listen sockets
bundle Danga::Socket and Sys::Syscall
Eric Wong [Wed, 8 May 2019 19:04:17 +0000 (19:04 +0000)]
DS: epoll: fix misordered EPOLL_CTL_DEL call
Any operations on an fd after POSIX::close() are invalid, so
epoll_ctl will fail. Worse off, in a multi-threaded Perl, the
fd may be reused by another thread and EPOLL_CTL_DEL can hit the
wrong file description as a result.
cf. https://rt.cpan.org/Ticket/Display.html?id=129487
Eric Wong [Wed, 8 May 2019 18:58:58 +0000 (18:58 +0000)]
DS: drop unused "_undef" sub
No longer used since we removed the *_ip_string fields
Eric Wong [Wed, 8 May 2019 18:10:25 +0000 (18:10 +0000)]
syscall: drop readahead wrapper
No backwards compatibility to worry about for us; and fadvise
is superior anyways.
Eric Wong [Wed, 8 May 2019 18:15:40 +0000 (18:15 +0000)]
build: do not manify DS and Syscall pods
We don't need to increase our install footprint with
documentation from our internals (which will surely
change).
Eric Wong [Wed, 8 May 2019 09:07:36 +0000 (09:07 +0000)]
DS: handle EINTR in IO::Poll path, too
IO::Poll::_poll returns -1, which is "true" to Perl.
cf. https://rt.cpan.org/Ticket/Display.html?id=129484
Eric Wong [Wed, 8 May 2019 02:53:03 +0000 (02:53 +0000)]
nntp: avoid uninitialized variable from blank requests
We'll ignore blank lines from clients, since that's what innd
seems to do.
Eric Wong [Tue, 7 May 2019 08:27:12 +0000 (08:27 +0000)]
wwwstream: do not load URI.pm
It's unneeded since commit
e358bd7a3833f8c5 (2016-07-02)
("inbox: base_url method takes PSGI env hashref instead")
So we only depend on URI::Escape from the "URI" CPAN distribution,
at the moment.
Eric Wong [Mon, 6 May 2019 23:42:31 +0000 (23:42 +0000)]
spawn (Inline::C): fix off-by-one error
Noticed while testing on FreeBSD 11.2 amd64 with the optional
Inline::C extension using clang 6.0.0. The end result on
FreeBSD was spawning processes failed badly and things were
immediately unusable with this enabled.
av_len is a misleading API, and I failed to read the API
comments in perl:/av.c which state:
> Note that, unlike what the name implies, it returns
> the highest index in the array, so to get the size of
> the array you need to use "av_len(av) + 1".
> This is unlike "sv_len", which returns what you would expect.
If this bug affected anybody, it would've only affected users
using both the optional Inline::C module AND set the
PERL_INLINE_DIRECTORY environment variable.
That said, I've never seen any evidence of it on Debian
GNU/Linux + gcc on any x86 variant. That includes full 64-bit
systems, a full 32-bit system, a 64-bit system with 32-bit
userspace, across multiple gcc versions since 2016.
Eric Wong [Fri, 19 Apr 2019 07:41:22 +0000 (07:41 +0000)]
index: warn with info about the message as context
This can help users track down the source of warnings
when presented with imperfect emails.
While we're at it, make the __WARN__ callback in t/v2writable.t
a no-op since we don't check for warnings, there.
Eric Wong [Sun, 5 May 2019 23:28:02 +0000 (23:28 +0000)]
Merge remote-tracking branch 'origin/wwwlisting'
* origin/wwwlisting:
www: support listing of inboxes
start depending on Perl 5.10.1+
Eric Wong [Sun, 5 May 2019 04:56:14 +0000 (04:56 +0000)]
DS: workaround IO::Kqueue EINTR (mis-)handling
IO::Kqueue seems unmaintained, so workaround a long-standing
bug where it falls over on signals:
https://rt.cpan.org/Ticket/Display.html?id=116615
Eric Wong [Sun, 5 May 2019 21:33:40 +0000 (21:33 +0000)]
t/search.t: fix permissions check on FreeBSD
FreeBSD does not allow non-root users to set S_ISGID;
so git skips this bit on FreeBSD and Debian/kFreeBSD
platforms.
Eric Wong [Sun, 5 May 2019 00:27:10 +0000 (00:27 +0000)]
DS: drop profiling support
There's other ways to profile and we don't need to add runtime
branches to do this.
Eric Wong [Fri, 3 May 2019 10:34:10 +0000 (10:34 +0000)]
DS: remove unused fields and functions
More will likely be dropped in the future, but drop the obvious
ones we aren't using, for now; especially since some of them are
set at ->new time and unavoidable.
This saves 579 bytes per-client on my 64-bit Debian stable
system as measured by Devel::Size::total_size from
PublicInbox::HTTP::event_read. This adds up in C10K or C100K
situations.
Things we drop are:
* corked - MSG_MORE requires fewer syscalls
* read_push_back - tried to use it, ate CPU with slow clients
* IP/port fields - accept() already returns what we care about
Eric Wong [Fri, 3 May 2019 10:34:09 +0000 (10:34 +0000)]
listener: use EPOLLEXCLUSIVE for listen sockets
Since our listen sockets are non-blocking and we may run
multiple httpd|nntpd processes; we need a way to avoid
thundering herds when there are multiple httpd|nntpd worker
processes.
EPOLLEXCLUSIVE was added just for that in Linux 4.5
Eric Wong [Fri, 3 May 2019 10:34:08 +0000 (10:34 +0000)]
bundle Danga::Socket and Sys::Syscall
These modules are unmaintained upstream at the moment, but I'll
be able to help with the intended maintainer once/if CPAN
ownership is transferred. OTOH, we've been waiting for that
transfer for several years, now...
Changes I intend to make:
* EPOLLEXCLUSIVE for Linux
* remove unused fields wasting memory
* kqueue bugfixes e.g. https://rt.cpan.org/Ticket/Display.html?id=116615
* accept4 support
And some lower priority experiments:
* switch to EV_ONESHOT / EPOLLONESHOT (incompatible changes)
* nginx-style buffering to tmpfile instead of string array
* sendfile off tmpfile buffers
* io_uring maybe?
Eric Wong [Sat, 4 May 2019 18:42:39 +0000 (18:42 +0000)]
wwwstream: drop tor2web URL
It was a MITM hazard and been killed off by the DoS subsystem
last July: https://marc.info/?i=87d0vwwkbs.fsf@riseup.net
Eric Wong [Mon, 29 Apr 2019 01:33:12 +0000 (01:33 +0000)]
doc/standards: add a link to PSGI specs
Eric Wong [Sat, 27 Apr 2019 21:46:54 +0000 (21:46 +0000)]
extmsg: escape ampersands in @EXT_URL array
We already escape the user-provided Message-IDs (so there's no
security problem AFAIK), but the URL templates which exist in
our source code were not escaped properly.
This quiets down tidy(1).
Eric Wong [Fri, 26 Apr 2019 07:42:24 +0000 (07:42 +0000)]
viewdiff: do not break out of DSTATE_CTX on /^$/
It seems a common case for mangled patches is editors or MUAs
dropping trailing whitespace, and lines matching /^ $/ gets
the space dropped to only match /^$/.
Eric Wong [Thu, 25 Apr 2019 07:57:06 +0000 (07:57 +0000)]
cgit: improve handling of cgit data path
Document `publicinbox.cgitdata' config directive, but allow it
to be unspecified and/or missing for installations which do not
wish to serve static data at all.
For users installing cgit from source to their home directory,
we can usually infer the cgit data path based on the cgit.cgi
binary path, even.
Eric Wong [Wed, 24 Apr 2019 22:51:42 +0000 (22:51 +0000)]
examples/cgit-commit-filter.lua: some doc updates
It's been a while since I wrote this, and it needs to be kept
up-to-date with some advances in our Perl code.
Eric Wong [Wed, 24 Apr 2019 22:44:04 +0000 (22:44 +0000)]
examples: cgit filter for use with WwwHighlight
I'm using this as the cgit about-filter and source-filter
in https://80x24.org/public-inbox.git
Eric Wong [Wed, 24 Apr 2019 21:56:53 +0000 (21:56 +0000)]
wwwhighlight: deal with no-op highlight
Not all inputs are highlight-able, so reuse the original
input and just linkify it if it can't be highlighted.
Eric Wong [Wed, 24 Apr 2019 21:22:11 +0000 (21:22 +0000)]
examples/cgit-commit-filter: remove unused variable
Eric Wong [Wed, 24 Apr 2019 19:52:14 +0000 (19:52 +0000)]
wwwhighlight: do linkification, too
Otherwise, there's no reason to use this API over highlight(1).
Maybe this can be an option in the future; but I'm struggling to
find a reason to not do it by default.
Eric Wong [Tue, 23 Apr 2019 03:33:41 +0000 (03:33 +0000)]
view: avoid "1+ messages" in per-message footer of /t/ and /T/
Try to appear gramatically correct and state:
"only message in thread" when there's only one known (to us)
message in the thread.
Eric Wong [Sat, 20 Apr 2019 00:20:04 +0000 (00:20 +0000)]
doc: add standards reference
Incomplete at the moment, but this ought to be a handy reference
for both implementers and users alike.
Eric Wong [Thu, 18 Apr 2019 08:25:56 +0000 (08:25 +0000)]
www: support listing of inboxes
We will still return a 404 by default to '/' for compatibility
with users of Plack::App::Cascade or similar. Inboxes are
sorted by modification times to help users detect activity
(similar to the /$INBOX/ topic view).
New configuration options:
* publicinbox.wwwlisting - configure the listing type
* publicinbox.<name>.hide - hide a particular inbox from the listing
See changes to public-inbox-config.pod for full descriptions
of the new options.
Requested-by: Leah Neukirchen <leah@vuxu.org>
https://public-inbox.org/meta/871sdfzy80.fsf@gmail.com/
Eric Wong [Wed, 17 Apr 2019 22:05:38 +0000 (22:05 +0000)]
start depending on Perl 5.10.1+
I mainly want to start using the '//' (defined-or) operator to
simplify code, and Perl 5.10.1 is roughly a decade old at this
point.
"given/when" would've be nice, but it's future is in doubt AFAIK.
I also started using the 'parent' module in WwwHighlight, and
'autodie' in UserContent.pm, both of which were only distributed
with Perl since 5.10.1; and testing with ancient
versions/distros is time-consuming.
Anyways, I think this a small-enough jump to not break any
existing installations, given we already depend on fairly
recent versions of git and Xapian.
Maybe we can use more newish Perl features in the future...
Eric Wong [Fri, 19 Apr 2019 08:09:31 +0000 (08:09 +0000)]
t/hl_mod: workaround w3m not handling '
This fixes a test failure on my Debian buster system.
Bug report filed for w3m to handle "'":
https://bugs.debian.org/927409
and for "highlight" to favor "'" in case other browsers fail:
https://bugs.debian.org/927410
Eric Wong [Thu, 18 Apr 2019 23:49:42 +0000 (23:49 +0000)]
linkify: require parentheses pairs in URLs
Dangling parentheses with trailing punctuation usually means the
parentheses is not intended as part of the URL.
Eric Wong [Thu, 18 Apr 2019 23:31:46 +0000 (23:31 +0000)]
view: show "(no subject)" consistently in HTML
Empty subjects ("") and undefined Subjects: are now both
displayed as "(no subject)" for now.
Eric Wong [Thu, 18 Apr 2019 22:53:20 +0000 (22:53 +0000)]
doc: config: fix braino/typo :x
Eric Wong [Thu, 18 Apr 2019 22:47:01 +0000 (22:47 +0000)]
linkify: don't get confused by URLs in Perl code, at least
The URLs at the top of WwwStream.pm weren't getting linkified
correctly.
Eric Wong [Thu, 18 Apr 2019 00:58:06 +0000 (00:58 +0000)]
inbox: add `modified' sub
For inboxes with SQLite enabled (all v2, and probably most v1);
we can use the overview DB to get the timestamp of the latest
message. It's faster than scanning git branches for commit times,
but not always the same.