Eric Wong [Mon, 28 Oct 2019 10:45:26 +0000 (10:45 +0000)]
mda: support multiple List-ID matches
While it's not RFC2919-conformant, mail software can
theoretically set multiple List-ID headers. Deliver to all
inboxes which match a given List-ID since that's likely the
intended.
Eric Wong [Mon, 28 Oct 2019 10:45:17 +0000 (10:45 +0000)]
learn: update usage statement
Use <foo|bar> since that seems to be the favored notation
for required command args (taking a hint from git(1) manpage).
While we're at it, remove the space after '<' for the redirect
to match git.git coding style.
Eric Wong [Mon, 28 Oct 2019 10:45:16 +0000 (10:45 +0000)]
learn: only map recipient list on "ham" or "rm"
It's assumed that "spam" can end up anywhere due to Bcc:, so we
need to scan every single inbox. However, "rm" is usually more
targeted and and "ham" obviously only belongs in some inboxes.
Eric Wong [Wed, 30 Oct 2019 08:47:23 +0000 (08:47 +0000)]
Merge remote-tracking branch 'origin/multi-mid'
* origin/multi-mid:
view: show X-Alt-Message-ID in permalink view, too
index: allow search/lookups on X-Alt-Message-ID
linkify: support adding "(raw)" link for Message-IDs
view: improve warning for multiple Message-IDs
view: move '<' and '>' outside <a>
view: display redundant headers in permalink
search: support multiple From/To/Cc/Subject headers
Eric Wong [Thu, 24 Oct 2019 00:12:41 +0000 (00:12 +0000)]
view: show X-Alt-Message-ID in permalink view, too
Since we index X-Alt-Message-ID (because we need to placate some
NNTP clients), we now display it as well, since that Message-ID
could be the X-Alt-Message-ID that the reader is actually
interested in.
Eric Wong [Thu, 24 Oct 2019 00:12:40 +0000 (00:12 +0000)]
index: allow search/lookups on X-Alt-Message-ID
Since we replace extra Message-ID headers with X-Alt-Message-ID
to placate NNTP clients, we should allow searching and indexing
on X-Alt-Message-ID just like we do with Message-ID.
Eric Wong [Thu, 24 Oct 2019 00:12:38 +0000 (00:12 +0000)]
view: improve warning for multiple Message-IDs
"refer" is not the correct term, here; since that would mean
multiple messages have the current message in the "References:"
header, and that's a normal occurence.
Instead, we need to warn the reader that the given message
itself has multiple Message-IDs.
Eric Wong [Thu, 24 Oct 2019 00:12:37 +0000 (00:12 +0000)]
view: move '<' and '>' outside <a>
Browsers may underline '<' and '>' in links, which may be
confused with '≤' and '≥'. So have the Message-ID header
display follow what we do with In-Reply-To headers and move the
"<" and ">" outside of <a> in the HTML.
Eric Wong [Thu, 24 Oct 2019 00:12:36 +0000 (00:12 +0000)]
view: display redundant headers in permalink
Mail headers can contain multiple headers of any type, so ensure
we don't hide any information we're getting in the per-message
permalink views.
This means it's possible to have multiple From, Date, To, Cc,
Subject, and In-Reply-To headers displayed.
The thread indices are a special case, I guess, since we run
out of space on the line if the headers too long and tools like
mutt only show the first one.
Eric Wong [Tue, 22 Oct 2019 07:45:38 +0000 (07:45 +0000)]
Merge branch 'regen'
* regen:
v2writable: use msgmap as multi_mid queue
v2writable: move git->cleanup to the correct place
v2writable: reindex handles 3-headered monsters
v2writable: improve "num_for" API and disambiguate
v2writable: set unindexed article number
Eric Wong [Mon, 21 Oct 2019 22:57:50 +0000 (22:57 +0000)]
v2writable: use msgmap as multi_mid queue
Instead of storing Message-IDs in the Msgmap object, we can
store the blob OID.
For initial indexing of mirrors, this lets us preserve
$sync->{regen} by storing the intended article number in
the queue.
On --reindex, the article number we store in Msgmap is ignored
but only used for ordering purposes.
This also allows us to avoid ENOMEM errors if somebody abuses
our system by reusing Message-IDs; but we now risk ENOSPC
instead (but systems tend to have more FS storage than RAM).
Eric Wong [Sun, 20 Oct 2019 22:05:05 +0000 (22:05 +0000)]
v2writable: reindex handles 3-headered monsters
And maybe 8-headered ones, too...
I noticed --reindex failing on the linux-renesas-soc mirror due
one 3-headed monster of a message having 3 sets of headers;
while another normal message had a Message-ID that matched one
of the 3 IDs of the 3-headed monster.
We still try to do the majority of indexing backwards, but we
defer indexing multi-Message-ID'd messages until the end to
ensure we get all the "good" messages in before we process the
multi-headered ones.
We need to account for both the old ("mainrepo") and new
("inboxdir") names. But "dir" was just a search+replace
error and we don't use that outside of "coderepo.dir".
Eric Wong [Thu, 17 Oct 2019 08:19:11 +0000 (08:19 +0000)]
Merge remote-tracking branch 'origin/inboxdir'
* origin/inboxdir:
config: remove redundant inboxdir check
config: support "inboxdir" in addition to "mainrepo"
examples/grok-pull.post_update_hook: use "inbox_dir"
Eric Wong [Wed, 16 Oct 2019 11:14:51 +0000 (11:14 +0000)]
doc: avoid [<directory>] arg for git-clone(1)
While it is possible to host source code from the root of a URL
using git-http-backend(1), the lack of pathname in the URL can
also be confusing to users. So just add the path name of the
project into the URL itself so users can invoke "git clone"
with one command-line argument instead of two.
Of course, previously documented URLs continue to work as normal.
Eric Wong [Wed, 16 Oct 2019 08:59:55 +0000 (08:59 +0000)]
config: support "inboxdir" in addition to "mainrepo"
"mainrepo" ws a bad name and artifact from the early days when I
intended for there to be a "spamrepo" (now just the
ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be
especially confusing, since v2 needs at least two git
repositories (epoch + all.git) to function and we shouldn't
confuse users by having them point to a git repository for v2.
Much of our documentation already references "INBOX_DIR" for
command-line arguments, so use "inboxdir" as the
git-config(1)-friendly variant for that.
"mainrepo" remains supported indefinitely for compatibility.
Users may need to revert to old versions, or may be referring
to old documentation and must not be forced to change config
files to account for this change.
So if you're using "mainrepo" today, I do NOT recommend changing
it right away because other bugs can lurk.
Eric Wong [Wed, 16 Oct 2019 00:39:56 +0000 (00:39 +0000)]
mda: support --no-precheck option
Since -mda now supports List-ID to better support mirroring of
existing mailing lists, it probably makes sense to support
disabling the precheck function to provide more accurate (though
potentially spammier) mirrors of lists
Eric Wong [Wed, 16 Oct 2019 00:30:28 +0000 (00:30 +0000)]
Merge branch 'listid'
* listid:
wwwtext: show listid config directive(s)
mda, watch: wire up List-ID header support
config: allow "0" as a valid mainrepo path
config: avoid unnecessary '||' use
config: simplify lookup* methods
config: we always have {-section_order}
Config.pm: Add support for mailing list information
Eric Wong [Mon, 7 Oct 2019 20:52:31 +0000 (20:52 +0000)]
admin: show failing directory
Since public-inbox-index may be run against a large list of
(intended) inboxes from the command-line, it's helpful to show
which directory fails the resolution.
Eric Wong [Tue, 15 Oct 2019 01:11:58 +0000 (01:11 +0000)]
config: simplify lookup* methods
This ensures we always process inboxes in section order and
reduces the amount of code we have to maintain for each lookup.
Avoiding the cost of inboxes object creation is not worth the
code overhead; and we can implement a config cache via Storable
easily for large configs and -mda users.
Eric Wong [Tue, 15 Oct 2019 00:38:06 +0000 (00:38 +0000)]
config: we always have {-section_order}
Rewrite a bunch of tests to use ordered input (emulating
"git config -l" output) so we can always walk sections in
the order they were given in the config file.
Eric W. Biederman [Fri, 17 May 2019 00:22:46 +0000 (19:22 -0500)]
Config.pm: Add support for mailing list information
The world has turned since I first started following mailing lists and
to my surprise every mailing list that I am subscribed to properly
sets the "List-ID:" mailing list header. So instead of doing
something clever and flexible I am adding support for looking up
public inbox mailing lists by their mailing list name.
That makes the work needed for each email trivial and easy to understand.
- Parse the "List-ID:" header.
- Lookup in the configuration which mailbox is connected to that
"List-ID:"
- Deliver the mail to that mailbox.
To that end this change enhances PublicInbox to have an additional
mailbox configuration parameter "listid" that holds the mailing list
name.
A method is added to the PublicInbox config object called
lookup_list_id that given a mailing list name will return the
PublicInbox in the configuration that is configured to handle that
mailing list.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
[ew: avoid autovivification of $ibx->{listid} for t/config.t]
Eric W. Biederman [Tue, 15 Jan 2019 22:36:42 +0000 (16:36 -0600)]
PublicInbox::Import Smuggle a raw message into add
I don't trust the MIME type to not munge my email messages in horrible
ways upon occasion. Therefore allow for passing in the raw message
value instead of trusting the mime object to preserve it.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
[ew: use "//" from Perl 5.10+ for defined check]
Eric Wong [Tue, 15 Oct 2019 03:59:26 +0000 (03:59 +0000)]
INSTALL: recommend inotify|kqueue modules for -watch
Jan Kiszka reported high polling frequency when using -watch.
It turns out OS-specific packages for Filesys::Notify::Simple do
not pull in interfaces to use kqueue or inotify, which are
required to perform power-efficient event-based wakeups on
Maildir writes.
Fix the name of the Filesys::Notify::Simple for FreeBSD while
we're at it.
Eric Wong [Fri, 13 Sep 2019 08:57:02 +0000 (08:57 +0000)]
t/git-http-backend: disable worker processes
We want to ensure we run lsof(8) on the worker (if needed),
and not the master, which doesn't serve requests.
This was originally on top of a test-only patch in
https://public-inbox.org/meta/20190913015043.17149-1-e@80x24.org/
In any case, no point in spawning extra processes for this test.
Alyssa Ross [Wed, 9 Oct 2019 13:21:14 +0000 (13:21 +0000)]
doc: explain publicinbox.<name>.watchheader
It wasn't clear to me exactly what this does -- in particular, what
happens if it isn't specified? Does it support multiple values? A
very brief explanation can answer both of these questions without
making somebody look at the code.
Eric Wong [Tue, 8 Oct 2019 08:58:06 +0000 (08:58 +0000)]
doc: PublicInbox::SaPlugin::ListMirror manpage
This is a plugin for SpamAssassin that happens to be quite
useful in keeping spam off lists I mirror. Hopefully more
people can find it useful now that it has a manpage.
Eric Wong [Sun, 6 Oct 2019 23:56:51 +0000 (23:56 +0000)]
examples: add grok-pull post_update_hook example
This requires the latest (to be in 1.2) -init changes for
synchronization and has no dependencies on GNU or bash-isms
so it should run on *BSD systems without GNU tools.
It does attempt to use curl on <$INBOX_URL/_/text/config/raw>,
but curl is fairly standard nowadays, and falls back to using
an invalid address to initialize.
Eric Wong [Sun, 6 Oct 2019 23:31:36 +0000 (23:31 +0000)]
doc: generate NEWS, NEWS.atom, and NEWS.html
We'll use our Documentation/RelNotes directory and internal APIs
to generate these files for website use (the website should be
completely reproducible).
Eric Wong [Thu, 3 Oct 2019 07:21:59 +0000 (07:21 +0000)]
init: implement locking
First, we use flock(2) to wait on parallel public-inbox-init(1)
invocations while we make multiple changes using git-config(1).
This flock allows -init processes to wait on each other if using
reasonable POSIX filesystems.
Then, we also need a git-config(1)-compatible lock to prevent
user-invoked git-config(1) processes from clobbering our
changes while we're holding the flock.
Eric Wong [Thu, 3 Oct 2019 07:57:29 +0000 (07:57 +0000)]
t/search: bail out on `git init --shared' failures
We can save future testers some time if we bail out early
on "git init --shared" failures, since things like seccomp
or non-POSIX FSes would trigger failures.
BAIL_OUT has been in Test::Simple since Perl v5.10.0, so it's
old-enough to call for our purposes.
Eric Wong [Tue, 1 Oct 2019 23:18:02 +0000 (23:18 +0000)]
listener: warn on some accept()/accept4() errors
We need to warn when hitting file or socket memory limits
or misconfigurations which clear O_NONBLOCK to make it easier
to diagnose configuration problems. We'll also warn on
some other errors in case bugs creep in...
Eric Wong [Tue, 1 Oct 2019 07:01:30 +0000 (07:01 +0000)]
www: fix absolute URLs when mounted under a subdir
While we avoid generating absolute URLs in most cases, our
"git clone" instructions and URL headers in mboxrd files
contain full URLs.
So do the same thing we do for WwwAtomStream and pre-generate
the full URL before Plack::App::URLMap changes $env->{PATH_INFO}
and $env->{SCRIPT_NAME} back to their original values.
Eric Wong [Tue, 1 Oct 2019 01:03:55 +0000 (01:03 +0000)]
TODO: update with "git cat-file" items
Millions of inboxes in an instance is probably not feasible, but
dozens or even hundreds could happen and
/proc/sys/fs/pipe-user-pages-soft is only 16384 on my system,
with each "cat-file --batch" process using 16+1 pages worth
of pipes.
The httpd-supplied write callback is the leak culprit under Perl
5.16.3. undef-ing it immediately after use keeps a repeated
"git fetch" loop from monotonically increasing memory and FD use
on the Perl shipped with RHEL/CentOS 7.x.
Other endpoints tested showed no increase in memory use under
constant load with "ab -HAccept-Encoding:gzip -k", including the
async psgi_qx code path used by $INBOX_URL/$OBJECT_ID/s/ via
SolverGit module.
Eric Wong [Fri, 27 Sep 2019 10:48:25 +0000 (10:48 +0000)]
wwwtext: support $INBOX_URL/_/text/config/raw
This returns a git-config(1)-compatible file to make it easier
to get started on mirroring an existing public-inbox. Omitting
the "raw" from the URL works, as well, but I'm not sure if
it's very useful.
Alyssa Ross [Tue, 24 Sep 2019 20:05:55 +0000 (20:05 +0000)]
hlmod: update for highlight 3.51 API change
Quoting Amitai Schleier, who made this same change in ikiwiki[1],
where lots of the public-inbox highlight code comes from:
> As of 3.51, searchFile() is no longer provided in highlight's Perl
> bindings (at least on NetBSD and OS X, as built from pkgsrc). This
> leaves us falling through to getConfDir(), which has been gone
> rather longer.
>
> From highlight git, it appears searchFile() and getFiletypesConfPath()
> both originated in the 3.14 release. The latter is still available in
> 3.51, and returns the same result searchFile() used to. Switch to it.
So, this should still be compatible with the version of highlight.pm in
Debian, but add support for newer versions as well.
Eric Wong [Thu, 26 Sep 2019 01:50:38 +0000 (01:50 +0000)]
httpd: disable Deflater middleware by default on Perl <5.18
Testing with perl-5.16.3-294.el7_6 RPM package on RHEL/CentOS 7,
the Deflater middleware triggers a leak when used in conjunction
with our push-based responses from PublicInbox::Qspawn.
I could not find another solution to workaround the memory leak
in this case, and I could not find a specific leak fix in
the perl5180delta manpage[1] which looked like it would
solve our problem.
Attempting to workaround the issue proved futile. Using
internal Deflater-specific keys to prevent deflating in
GitHTTPBackend and Qspawn did not solve the problem:
So this appears to be a problem with Plack::Util::response_cb
somewhere.
This does NOT appear to be a problem with ref() leaking as in
DS::next_tick[2], since I couldn't find where
Plack::Middleware::Deflater or Plack::Util::response_cb would be
calling ref() on a blessed reference to trigger a leak.
Also, oddly enough, the ref() use for backwards compatibility at
the top of PublicInbox::GitHTTPBackend::serve does NOT seem to
trigger a leak on 5.16.3 due to [2]:
# XXX compatibility... ugh, can we stop supporting this?
$git = PublicInbox::Git->new($git) unless ref($git);
Eric Wong [Thu, 26 Sep 2019 01:50:37 +0000 (01:50 +0000)]
ds: workaround a memory leak in Perl 5.16.x
The perl-5.16.3-294.el7_6 RPM package on RHEL/CentOS 7 is
affected by a memory leak in Perl when calling `ref' on
blessed references. This resulted in a very slow leak that
manifests more quickly with a nonstop "git fetch" loop.
Use Scalar::Util::blessed to work around the issue.
Tested overnight on a CentOS 7 VM.
Eric Wong [Tue, 24 Sep 2019 03:39:03 +0000 (03:39 +0000)]
spawnpp: use absolute path for exec
We support "-env" to clear the environment with spawn(),
which causes test failures but no runtime failures
(since "-env" isn't used anywhere in our real code)
Reported-and-tested-by: Alyssa Ross <hi@alyssa.is>
Eric Wong [Sat, 21 Sep 2019 00:06:42 +0000 (00:06 +0000)]
doc: update HACKING and TODO with a few items
Inline::C seems alright, so we might use it more since it still
allows end users to quickly make changes. Our performance on
rotational disks is also terrible, and could be improved...