Eric Wong [Sat, 9 Apr 2016 00:28:07 +0000 (00:28 +0000)]
import: initial module + test case
This will allow us to write fast importers for existing
archives as well as eventually removing the ssoma dependency
for performance and ease-of-installation.
Eric Wong [Wed, 6 Apr 2016 08:23:15 +0000 (08:23 +0000)]
view: account for threads lacking a common parent
In the per-message view, we still need to account for threads
lacking a common parent. This can happen when threads are
broken by some broken clients or if somebody sends the same
message twice to the same inbox with a different Message-ID.
Eric Wong [Wed, 6 Apr 2016 07:21:12 +0000 (07:21 +0000)]
view: do not prune ghosts from threads
Keeping readers informed of ghost messages is important,
so do not ever prune them. Previously, ghosts could get
pruned and sole children would get promoted as the new
root.
Eric Wong [Wed, 6 Apr 2016 06:30:28 +0000 (06:30 +0000)]
examples/public-inbox.psgi: add note for our httpd
Default to maximizing compatibility in the example, but document the
potential improvement if possible. Of course, using
public-inbox-httpd out-of-the-box without a user-specified config
file already enables chunked encoding by default.
Eric Wong [Wed, 6 Apr 2016 05:38:53 +0000 (05:38 +0000)]
http: clarify intent for persistence
We don't actually need to know if a response is chunked or
what the actual Content-Length is; we just need to know if
the PSGI app properly terminated the response so we can
handle persistent connections.
Eric Wong [Tue, 5 Apr 2016 06:26:35 +0000 (06:26 +0000)]
view: link restructuring for index view
The "next/prev" links seem a bit awkward and I don't use them as
much as I expected to. However, move the "raw" message link
near the top since it's most useful for checking or reinforcing
the validity of the message via GPG or just reading headers.
Turn the Subject line into a permalink to the message, since
that's probably the common behavior anyways for other messaging
systems. Make the "[threaded|flat]" view links to always
visible for bookmark-ability despite the lack of a "permalink"
label.
Eric Wong [Mon, 4 Apr 2016 21:15:26 +0000 (21:15 +0000)]
http: fix condition for detecting persistence
Oops, we need to watch out for how we handle operator
precedence and ensure responses without a Content-Length
or "Transfer-Encoding: chunked" header will always
disconnect after writing.
Eric Wong [Thu, 17 Mar 2016 01:50:07 +0000 (01:50 +0000)]
daemon: expand @ARGV paths for running in '/'
We also require --stdout/--stderr/--pid-file to be absolute
paths for USR2 usage. However, allow PSGI files for -httpd
to be relative paths for ease-of-use.
Eric Wong [Sat, 12 Mar 2016 06:51:22 +0000 (06:51 +0000)]
searchmsg: preserve hard tabs, but drop CR (\r)
Hard tabs *may* be searchable, so preserve them since they do
not take up any more space than a normal space. However, CR
(carriage return) is worthless and likely a sign of a buggy mail
(or spam) client anyways.
Eric Wong [Sat, 12 Mar 2016 03:14:26 +0000 (03:14 +0000)]
examples: disable Chunked response in PSGI example
It seems incompatible with Starman and probably confuses other
HTTP/1.0-only servers, too. Our -httpd will respect it and
requires it for persistent connections.
Eric Wong [Sat, 12 Mar 2016 00:20:12 +0000 (00:20 +0000)]
http: prevent zero-byte writes
Plack::Middleware::Deflater (and perhaps other middleware)
triggers zero-byte writes which wastes syscalls when
they get passed to Danga::Socket. This may also trigger
problems when we introduce TLS support in the future.
Eric Wong [Fri, 11 Mar 2016 21:59:42 +0000 (21:59 +0000)]
daemon: fixup usage of the '-l' switch with IP/INET6 sockets
We need to ensure $sock_pkg is preserved outside of the loop.
The variable passed to "for" or "foreach" is implicitly local
and restores the previous value when the loop exits. This is
documented in the perlsyn manpage in the "Foreach Loops"
section.
Fixes: ea1b6cbd422b ("daemon: allow using IO::Socket::IP over INET6")
Eric Wong [Mon, 7 Mar 2016 17:43:19 +0000 (17:43 +0000)]
daemon: allow using IO::Socket::IP over INET6
IO::Socket::IP is bundled with newer versions of Perl,
so it is more likely to be available. There should
be no differences between these with our use cases.
Eric Wong [Sun, 6 Mar 2016 02:09:21 +0000 (02:09 +0000)]
http: ensure errors are printable before PSGI env
We cannot rely on a client socket having a PSGI env before headers
are fully-parsed as we seek to avoid storing hashes for idle
clients. Sso print errors to the psgi.errors value which belongs to
the httpd listener, instead.
Eric Wong [Sun, 6 Mar 2016 02:09:20 +0000 (02:09 +0000)]
http: reject excessive headers
HTTP::Parser::XS::PP does not reject excessively large
headers like the XS version. Ensure we reject headers
over 16K since public-inbox should never need such large
request headers.
Eric Wong [Sat, 5 Mar 2016 07:35:22 +0000 (07:35 +0000)]
t/httpd-corner: avoid clobbering existing FDs after fork
Due to the deterministic way reference counting works,
we do not want to drop references to existing FDs
even if we no longer need the glob reference; the actual
FD is all we can pass through on exec.
Eric Wong [Sat, 5 Mar 2016 05:44:16 +0000 (05:44 +0000)]
daemon: drop listener sockets ASAP on termination
We do not want to be accepting connections during graceful
shutdown because another new process is likely taking over.
This also allows us to free up the listener case another
(independent) process wants to claim it.
Eric Wong [Thu, 3 Mar 2016 10:33:02 +0000 (10:33 +0000)]
daemon: support listening on Unix domain sockets
Listening on Unix domain sockets can be convenient for running
behind reverse proxies, avoiding port conflicts, limiting access,
or avoiding the overhead (if any) of TCP over loopback.
Eric Wong [Thu, 3 Mar 2016 05:14:31 +0000 (05:14 +0000)]
daemon: introduce host_with_port for identifying sockets
This allows us to share more code between daemons and avoids
having to make additional syscalls for preparing REMOTE_HOST
and REMOTE_PORT in the PSGI env in -httpd.
This will also make supporting HTTP (and NNTP) over Unix sockets
easier in a future commit.
Eric Wong [Thu, 3 Mar 2016 05:14:30 +0000 (05:14 +0000)]
daemon: avoid polluting the main package
We've distilled the daemon code into one public function ("run"),
so avoid polluting the main namespace and just have users
prefix with the full package name for this rarely-used class.
Eric Wong [Thu, 3 Mar 2016 03:16:58 +0000 (03:16 +0000)]
use raw header for Message-ID
Message-IDs should not be MIME encoded, but in case they are,
use the raw form for compatibility with ssoma and possibly
other tools. This prevents a potential problem where a
malicious client could confuse our storage layer into indexing
incorrect contents.
Eric Wong [Tue, 1 Mar 2016 08:19:12 +0000 (08:19 +0000)]
http: better error handling for EMFILE/ENFILE
Better to throw the error back to the client ASAP if we're
out-of-descriptors. We will need to implement idle client
expiration for long-lived HTTP connections.
Eric Wong [Tue, 1 Mar 2016 03:44:04 +0000 (03:44 +0000)]
linkify: do not capture trailing '.' or ';' in URLs
It seems common for users to end statements with URLs,
while it is rare for a URL itself to end with a '.' or ';'.
So make a guess and assume the URL was intended to not
include the trailing '.' or ';'
Eric Wong [Tue, 1 Mar 2016 02:45:34 +0000 (02:45 +0000)]
view: consolidate whitespace stripping from messages
We now keep intermediate blank lines in messages, since it
could be used to denote logical gaps in the message
(such as giving readers a chance to opt out of "spoiler"
information).
However leading blank lines, trailing blank lines, and
trailing whitespace have no useful value we can discern;
so drop those entirely to prevent clients from eating up
vertical whitespace.
Eric Wong [Mon, 29 Feb 2016 02:48:45 +0000 (02:48 +0000)]
t/search.t: use transactions to reduce I/O load
In case folks do not use eatmydata or tmpfs for testing,
use transactions to reduce the number of fsync calls
made and hopefully prevent drives from wearing out.
Eric Wong [Mon, 29 Feb 2016 00:41:02 +0000 (00:41 +0000)]
distinguish error messages intended for users vs developers
For error messages intended to show user error (e.g. giving
invalid options), we add a newline ("\n") at the end to
polluting the output with location information.
However, for diagnosing non-user-triggered errors, we should
show the location of where the error occured.
Eric Wong [Sun, 28 Feb 2016 23:06:31 +0000 (23:06 +0000)]
examples/public-inbox.psgi: relax license to GPL-3.0+
Using the AGPL for server config files is probably overkill.
GPL-3.0+ still requires appliance vendors to disclose
configurations which seems desirable for end users.
Eric Wong [Sun, 28 Feb 2016 22:28:50 +0000 (22:28 +0000)]
examples/: PSGI example updates
Users wanting to customize their installation should know
to about the usability of STDOUT for logging.
(and we still need manpages for -nntpd and -httpd)
Eric Wong [Sun, 28 Feb 2016 11:28:33 +0000 (11:28 +0000)]
reduce calls to close unless error checks are needed
We can rely on timely auto-destruction based on reference
counting; reducing the chance of redundant close(2) calls
which may hit the wront FD.
We do care about certain close calls (e.g. writing to a buffered
IO handle) if we require error-checking for write-integrity. In
other cases, let things go out-of-scope so it can be freed
automatically after use.
Eric Wong [Sun, 28 Feb 2016 00:57:11 +0000 (00:57 +0000)]
httpd: allow running if ReverseProxy is missing
Not everybody will be running this behind a ReverseProxy;
but it's probably the likely configuration. Anyways,
warn about this and also about Deflater being missing.
Eric Wong [Sun, 28 Feb 2016 04:27:11 +0000 (04:27 +0000)]
spawn: disable popen optimization for non-vfork
This is necessary since we want to be able to do arbitrary redirects
via the popen interface. Oh well, we'll be a little slower for now
for users without vfork. vfork users will get all the performance
benefits.
Eric Wong [Sat, 27 Feb 2016 22:36:32 +0000 (22:36 +0000)]
daemon: refresh before forking
This means we always load the PSGI server code early for
-httpd. This may make things less compatible with existing
PSGI/Plack apps, but we prioritize our httpd for the uses
of public-inbox itself, first.
And any existing PSGI/Plack app which wants to may adapt
themselves to being preload-friendly.
Eric Wong [Sat, 27 Feb 2016 21:57:57 +0000 (21:57 +0000)]
move executables to script/ directory
This seems to match more closely with what is expected of Perl
packages based on how blib is used. Hopefully makes the top-level
source tree less cluttered and things easier-to-find.
Eric Wong [Sat, 27 Feb 2016 02:14:23 +0000 (02:14 +0000)]
initial spawn implementation using vfork
Under Linux, vfork maintains constant performance as
parent process size increases. fork needs to prepare pages
for copy-on-write, requiring a linear scan of the address
space.
Eric Wong [Fri, 26 Feb 2016 09:15:36 +0000 (09:15 +0000)]
psgi: enable ReverseProxy middleware by default
ReverseProxy is the common way to run Perl applications,
so enable it by default and don't care too much about fake
requests because we don't handle any sensitive information
or rely on authentication (everything is read-only from
the WWW interface and will remain so).
Eric Wong [Fri, 26 Feb 2016 01:57:57 +0000 (01:57 +0000)]
www: workaround for malformed NNTP links
Some linkifiers to create invalid HTTP links when it sees a
link intended for NNTP services. This means we may see links
to news.public-inbox.org/inbox.comp.mail.public-inbox.meta
point to "http://" on port 80 instead of 119. Try to
redirect users to http://public-inbox.org/meta/ in this case.