]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
8 years agoimport: use bytes::length for true data length in bytes
Eric Wong [Mon, 11 Apr 2016 04:44:53 +0000 (04:44 +0000)]
import: use bytes::length for true data length in bytes

git is byte-oriented and fast-import will not tolerate
miscalculations.  This is necessary for wide characters
in commit messages (email Subjects).

8 years agoimport: set binmode before printing author names
Eric Wong [Sat, 9 Apr 2016 09:07:16 +0000 (09:07 +0000)]
import: set binmode before printing author names

Author names may have wide characters in them, so avoid warnings
as git favors UTF-8 for names and fast-import even requires them
for commit messages

8 years agoimport: initial module + test case
Eric Wong [Sat, 9 Apr 2016 00:28:07 +0000 (00:28 +0000)]
import: initial module + test case

This will allow us to write fast importers for existing
archives as well as eventually removing the ssoma dependency
for performance and ease-of-installation.

8 years agogit: add support for qx wrapper
Eric Wong [Thu, 31 Dec 2015 21:16:39 +0000 (21:16 +0000)]
git: add support for qx wrapper

This lets us one-line git commands easily like ``, but without
having to remember --git-dir or escape arguments.

8 years agombox: unconditionally add trailing newline
Eric Wong [Mon, 11 Apr 2016 04:51:40 +0000 (04:51 +0000)]
mbox: unconditionally add trailing newline

This may be necessary for compatibility with non-mboxrd aware
parsers which expect "\nFrom " for everything but the first
record.

8 years agopublic-inbox-learn: drop leading "From " line from mboxes
Eric Wong [Sat, 9 Apr 2016 01:27:37 +0000 (01:27 +0000)]
public-inbox-learn: drop leading "From " line from mboxes

It can confuse Email::MIME if we have it.

8 years agofilter: remove out dated comments
Eric Wong [Sat, 9 Apr 2016 01:21:59 +0000 (01:21 +0000)]
filter: remove out dated comments

Followup-to commit 5a590bcb6813
("filter: preserve Mail-Followup-To and Mail-Reply-To")

8 years agofilter: preserve Mail-Followup-To and Mail-Reply-To
Eric Wong [Sat, 9 Apr 2016 00:57:26 +0000 (00:57 +0000)]
filter: preserve Mail-Followup-To and Mail-Reply-To

Allow users to do wacky things here if they really wish...
It's bad practice, but at least allow other readers to
mock users of these headers :P

8 years agoview: account for threads lacking a common parent
Eric Wong [Wed, 6 Apr 2016 08:23:15 +0000 (08:23 +0000)]
view: account for threads lacking a common parent

In the per-message view, we still need to account for threads
lacking a common parent.  This can happen when threads are
broken by some broken clients or if somebody sends the same
message twice to the same inbox with a different Message-ID.

8 years agoview: shorter link for ghosts in per-message view
Eric Wong [Wed, 6 Apr 2016 07:37:46 +0000 (07:37 +0000)]
view: shorter link for ghosts in per-message view

Shorten lines used for long Message-IDs in the
inline thread view for per-message views for readability.

8 years agoview: do not prune ghosts from threads
Eric Wong [Wed, 6 Apr 2016 07:21:12 +0000 (07:21 +0000)]
view: do not prune ghosts from threads

Keeping readers informed of ghost messages is important,
so do not ever prune them.  Previously, ghosts could get
pruned and sole children would get promoted as the new
root.

8 years agoview: eliminate dead code and hash fields
Eric Wong [Wed, 6 Apr 2016 06:55:39 +0000 (06:55 +0000)]
view: eliminate dead code and hash fields

These were the vestigial remains of our previous use of
of Message-ID compression.

8 years agoexamples/public-inbox.psgi: add note for our httpd
Eric Wong [Wed, 6 Apr 2016 06:30:28 +0000 (06:30 +0000)]
examples/public-inbox.psgi: add note for our httpd

Default to maximizing compatibility in the example, but document the
potential improvement if possible.  Of course, using
public-inbox-httpd out-of-the-box without a user-specified config
file already enables chunked encoding by default.

8 years agohttp: clarify intent for persistence
Eric Wong [Wed, 6 Apr 2016 05:38:53 +0000 (05:38 +0000)]
http: clarify intent for persistence

We don't actually need to know if a response is chunked or
what the actual Content-Length is; we just need to know if
the PSGI app properly terminated the response so we can
handle persistent connections.

8 years agoview: link restructuring for index view
Eric Wong [Tue, 5 Apr 2016 06:26:35 +0000 (06:26 +0000)]
view: link restructuring for index view

The "next/prev" links seem a bit awkward and I don't use them as
much as I expected to.  However, move the "raw" message link
near the top since it's most useful for checking or reinforcing
the validity of the message via GPG or just reading headers.

Turn the Subject line into a permalink to the message, since
that's probably the common behavior anyways for other messaging
systems.  Make the "[threaded|flat]" view links to always
visible for bookmark-ability despite the lack of a "permalink"
label.

8 years agohttp: fix condition for detecting persistence
Eric Wong [Mon, 4 Apr 2016 21:15:26 +0000 (21:15 +0000)]
http: fix condition for detecting persistence

Oops, we need to watch out for how we handle operator
precedence and ensure responses without a Content-Length
or "Transfer-Encoding: chunked" header will always
disconnect after writing.

8 years agowww: more explicit "git clone" usage
Eric Wong [Sat, 2 Apr 2016 22:32:13 +0000 (22:32 +0000)]
www: more explicit "git clone" usage

Little harm in having the entire command-line for users and
avoiding the cognitive overhead of figuring out $URL.

8 years agowww: various style changes and comment updates
Eric Wong [Sat, 2 Apr 2016 22:32:01 +0000 (22:32 +0000)]
www: various style changes and comment updates

Reduce stack depth of arguments and rely more on state hashref
to store response state.  We may end up shoving everything
in ctx eventually.

8 years agohttpd: remove reference to callback during close
Eric Wong [Thu, 31 Mar 2016 03:33:59 +0000 (03:33 +0000)]
httpd: remove reference to callback during close

Avoid wasting memory and the risk of a potential reference
cycles by dropping the callback ASAP.

8 years agodaemon: expand @ARGV paths for running in '/'
Eric Wong [Thu, 17 Mar 2016 01:50:07 +0000 (01:50 +0000)]
daemon: expand @ARGV paths for running in '/'

We also require --stdout/--stderr/--pid-file to be absolute
paths for USR2 usage.  However, allow PSGI files for -httpd
to be relative paths for ease-of-use.

8 years agofeed: fix brain farts in new_oneline removal
Eric Wong [Sat, 12 Mar 2016 07:34:20 +0000 (07:34 +0000)]
feed: fix brain farts in new_oneline removal

Ugh...

Fixes: 476fc666c223 (reduce "PublicInbox::Hval->new_oneline" use)
8 years agosearchmsg: preserve hard tabs, but drop CR (\r)
Eric Wong [Sat, 12 Mar 2016 06:51:22 +0000 (06:51 +0000)]
searchmsg: preserve hard tabs, but drop CR (\r)

Hard tabs *may* be searchable, so preserve them since they do
not take up any more space than a normal space.  However, CR
(carriage return) is worthless and likely a sign of a buggy mail
(or spam) client anyways.

8 years agoreduce "PublicInbox::Hval->new_oneline" use
Eric Wong [Sat, 12 Mar 2016 06:42:04 +0000 (06:42 +0000)]
reduce "PublicInbox::Hval->new_oneline" use

It's probably a bad idea to strip extraneous whitespace
from some headers as an extra space may convey useful
information.

Newlines don't seem to be preserved by Email::MIME or
Email::Simple anyways, so there's no danger in breaking
formatting.

8 years agohttp: use Plack::HTTPParser for HTTP parsing
Eric Wong [Sat, 12 Mar 2016 03:55:20 +0000 (03:55 +0000)]
http: use Plack::HTTPParser for HTTP parsing

This allows us to reduce installation dependencies while
retaining performance as it favors HTTP::Parser::XS when
it is installed and available.

PLACK_HTTP_PARSER_PP may be set to 1 to force a pure Perl
parser for testing.

8 years agoexamples: disable Chunked response in PSGI example
Eric Wong [Sat, 12 Mar 2016 03:14:26 +0000 (03:14 +0000)]
examples: disable Chunked response in PSGI example

It seems incompatible with Starman and probably confuses other
HTTP/1.0-only servers, too.  Our -httpd will respect it and
requires it for persistent connections.

8 years agohttp: prevent zero-byte writes
Eric Wong [Sat, 12 Mar 2016 00:20:12 +0000 (00:20 +0000)]
http: prevent zero-byte writes

Plack::Middleware::Deflater (and perhaps other middleware)
triggers zero-byte writes which wastes syscalls when
they get passed to Danga::Socket.  This may also trigger
problems when we introduce TLS support in the future.

8 years agodaemon: fixup usage of the '-l' switch with IP/INET6 sockets
Eric Wong [Fri, 11 Mar 2016 21:59:42 +0000 (21:59 +0000)]
daemon: fixup usage of the '-l' switch with IP/INET6 sockets

We need to ensure $sock_pkg is preserved outside of the loop.
The variable passed to "for" or "foreach" is implicitly local
and restores the previous value when the loop exits.  This is
documented in the perlsyn manpage in the "Foreach Loops"
section.

Fixes: ea1b6cbd422b ("daemon: allow using IO::Socket::IP over INET6")
8 years agodaemon: allow using IO::Socket::IP over INET6
Eric Wong [Mon, 7 Mar 2016 17:43:19 +0000 (17:43 +0000)]
daemon: allow using IO::Socket::IP over INET6

IO::Socket::IP is bundled with newer versions of Perl,
so it is more likely to be available.  There should
be no differences between these with our use cases.

8 years agohttp: reject excessively large HTTP request bodies
Eric Wong [Sun, 6 Mar 2016 02:09:22 +0000 (02:09 +0000)]
http: reject excessively large HTTP request bodies

We cannot risk using all of a users' disk space buffering
gigantic requests.  Use the defaults git gives us since
we primarily host git repositories.

8 years agohttp: ensure errors are printable before PSGI env
Eric Wong [Sun, 6 Mar 2016 02:09:21 +0000 (02:09 +0000)]
http: ensure errors are printable before PSGI env

We cannot rely on a client socket having a PSGI env before headers
are fully-parsed as we seek to avoid storing hashes for idle
clients.  Sso print errors to the psgi.errors value which belongs to
the httpd listener, instead.

8 years agohttp: reject excessive headers
Eric Wong [Sun, 6 Mar 2016 02:09:20 +0000 (02:09 +0000)]
http: reject excessive headers

HTTP::Parser::XS::PP does not reject excessively large
headers like the XS version.  Ensure we reject headers
over 16K since public-inbox should never need such large
request headers.

8 years agodaemon: sockname detects listeners correctly
Eric Wong [Sat, 5 Mar 2016 22:42:16 +0000 (22:42 +0000)]
daemon: sockname detects listeners correctly

This means we can avoid false-positives when inheriting multiple
Unix domain sockets.

8 years agodaemon: document optional Net::Server dependency
Eric Wong [Sat, 5 Mar 2016 22:42:12 +0000 (22:42 +0000)]
daemon: document optional Net::Server dependency

Non-socket activation users will want to install Net::Server
for daemonization, pid file writing, and user/group switching.

8 years agodoc: add contact/see-also/copyright sections to mda manpage
Eric Wong [Sat, 5 Mar 2016 22:07:53 +0000 (22:07 +0000)]
doc: add contact/see-also/copyright sections to mda manpage

We need manpages before we can expect people to install this.

8 years agohttpd: remove unnecessary eval
Eric Wong [Sat, 5 Mar 2016 20:53:25 +0000 (20:53 +0000)]
httpd: remove unnecessary eval

We have per-middleware evals to deal with them being missing;
no need to put an eval around the whole thing and use an
extra level of indentation.

8 years agot/httpd-corner: avoid clobbering existing FDs after fork
Eric Wong [Sat, 5 Mar 2016 07:35:22 +0000 (07:35 +0000)]
t/httpd-corner: avoid clobbering existing FDs after fork

Due to the deterministic way reference counting works,
we do not want to drop references to existing FDs
even if we no longer need the glob reference; the actual
FD is all we can pass through on exec.

8 years agodoc: language-neutral client-side endpoints
Eric Wong [Sat, 5 Mar 2016 07:08:12 +0000 (07:08 +0000)]
doc: language-neutral client-side endpoints

Be less specific, client-side code can be written in any
language (and I do not care for JS runtimes implemented in
C++ :P).

8 years agodoc: varyus speling fickses
Eric Wong [Sat, 5 Mar 2016 07:00:41 +0000 (07:00 +0000)]
doc: varyus speling fickses

Letz trie 2 uphear liter8

8 years agofeed: remove unnecessary encoding lookup
Eric Wong [Sat, 5 Mar 2016 06:45:41 +0000 (06:45 +0000)]
feed: remove unnecessary encoding lookup

We handle encoding-related things elsewhere.

8 years agodaemon: simplify parent death handling
Eric Wong [Sat, 5 Mar 2016 06:00:59 +0000 (06:00 +0000)]
daemon: simplify parent death handling

No need to create a new sub which kill ourselves $$ when we can
invoke worker_quit directly.

8 years agodaemon: avoid cyclic references for once-used callbacks
Eric Wong [Sat, 5 Mar 2016 05:52:14 +0000 (05:52 +0000)]
daemon: avoid cyclic references for once-used callbacks

Not that these subs are repeatedly created, but this makes
the code easier-to-review and these callbacks are idempotent
anyways.

8 years agodaemon: drop listener sockets ASAP on termination
Eric Wong [Sat, 5 Mar 2016 05:44:16 +0000 (05:44 +0000)]
daemon: drop listener sockets ASAP on termination

We do not want to be accepting connections during graceful
shutdown because another new process is likely taking over.
This also allows us to free up the listener case another
(independent) process wants to claim it.

8 years agot/httpd-corner: additional callback test
Eric Wong [Sat, 5 Mar 2016 05:41:12 +0000 (05:41 +0000)]
t/httpd-corner: additional callback test

Just to ensure we hit the code path independently of
WWW code.

8 years agogit-http-backend: favor sysread for regular files
Eric Wong [Sat, 5 Mar 2016 00:24:16 +0000 (00:24 +0000)]
git-http-backend: favor sysread for regular files

We do not need line buffering, here; so favor sysread to
bypass extra copies which may be done by normal read.

8 years agodaemon: simplify socket inheriting, slightly
Eric Wong [Fri, 4 Mar 2016 01:00:26 +0000 (01:00 +0000)]
daemon: simplify socket inheriting, slightly

IO::Handle->new_from_fd has existed since at least 1996,
so it should be safe to depend on at this point.

8 years agodaemon: support listening on Unix domain sockets
Eric Wong [Thu, 3 Mar 2016 10:33:02 +0000 (10:33 +0000)]
daemon: support listening on Unix domain sockets

Listening on Unix domain sockets can be convenient for running
behind reverse proxies, avoiding port conflicts, limiting access,
or avoiding the overhead (if any) of TCP over loopback.

8 years agodaemon: introduce host_with_port for identifying sockets
Eric Wong [Thu, 3 Mar 2016 05:14:31 +0000 (05:14 +0000)]
daemon: introduce host_with_port for identifying sockets

This allows us to share more code between daemons and avoids
having to make additional syscalls for preparing REMOTE_HOST
and REMOTE_PORT in the PSGI env in -httpd.

This will also make supporting HTTP (and NNTP) over Unix sockets
easier in a future commit.

8 years agodaemon: avoid polluting the main package
Eric Wong [Thu, 3 Mar 2016 05:14:30 +0000 (05:14 +0000)]
daemon: avoid polluting the main package

We've distilled the daemon code into one public function ("run"),
so avoid polluting the main namespace and just have users
prefix with the full package name for this rarely-used class.

8 years agot/*.t: use identifiable tempdir names
Eric Wong [Thu, 3 Mar 2016 09:07:40 +0000 (09:07 +0000)]
t/*.t: use identifiable tempdir names

This should make identifiying leftover directories
due to SIGKILL-ed tests easier.

8 years agoview: fix stupid typo in inline_dump
Eric Wong [Thu, 3 Mar 2016 07:35:34 +0000 (07:35 +0000)]
view: fix stupid typo in inline_dump

Ugh, this enabled-iff-xapian-is-available code really
needs better testing...

8 years agouse raw header for Message-ID
Eric Wong [Thu, 3 Mar 2016 03:16:58 +0000 (03:16 +0000)]
use raw header for Message-ID

Message-IDs should not be MIME encoded, but in case they are,
use the raw form for compatibility with ssoma and possibly
other tools.  This prevents a potential problem where a
malicious client could confuse our storage layer into indexing
incorrect contents.

8 years agohttp: better error handling for EMFILE/ENFILE
Eric Wong [Tue, 1 Mar 2016 08:19:12 +0000 (08:19 +0000)]
http: better error handling for EMFILE/ENFILE

Better to throw the error back to the client ASAP if we're
out-of-descriptors.  We will need to implement idle client
expiration for long-lived HTTP connections.

8 years agohttpd: remove unneeded err and out fields from class
Eric Wong [Tue, 1 Mar 2016 07:52:49 +0000 (07:52 +0000)]
httpd: remove unneeded err and out fields from class

Vestigial pieces from the nntpd code which aren't needed because
the psgi env already has the "psgi.errors" key.

8 years agohttpd: document pi-httpd.async as totally unstable
Eric Wong [Tue, 1 Mar 2016 07:48:53 +0000 (07:48 +0000)]
httpd: document pi-httpd.async as totally unstable

We'll have to use it some more before deciding it is a public
interface.  I do hope for it to be a usable public interface
one day for other users.

8 years agoprocesspipe: preserve native close behavior
Eric Wong [Tue, 1 Mar 2016 04:15:59 +0000 (04:15 +0000)]
processpipe: preserve native close behavior

We need to ensure close on handles tied to this class
get the same errors a normal "close" in Perl gets.

8 years agolinkify: do not capture trailing '.' or ';' in URLs
Eric Wong [Tue, 1 Mar 2016 03:44:04 +0000 (03:44 +0000)]
linkify: do not capture trailing '.' or ';' in URLs

It seems common for users to end statements with URLs,
while it is rare for a URL itself to end with a '.' or ';'.
So make a guess and assume the URL was intended to not
include the trailing '.' or ';'

8 years agoextract linkification code to a separate package
Eric Wong [Tue, 1 Mar 2016 03:44:03 +0000 (03:44 +0000)]
extract linkification code to a separate package

This will allow us to more easily reuse it elsewhere.

8 years agoMANIFEST: add examples/apache2_perl_old.conf
Eric Wong [Tue, 1 Mar 2016 03:44:02 +0000 (03:44 +0000)]
MANIFEST: add examples/apache2_perl_old.conf

Ugh, I wonder if we can/should generate this automatically...

8 years agoview: consolidate whitespace stripping from messages
Eric Wong [Tue, 1 Mar 2016 02:45:34 +0000 (02:45 +0000)]
view: consolidate whitespace stripping from messages

We now keep intermediate blank lines in messages, since it
could be used to denote logical gaps in the message
(such as giving readers a chance to opt out of "spoiler"
information).

However leading blank lines, trailing blank lines, and
trailing whitespace have no useful value we can discern;
so drop those entirely to prevent clients from eating up
vertical whitespace.

8 years agoview: do not hide patches or signatures
Eric Wong [Tue, 1 Mar 2016 02:08:38 +0000 (02:08 +0000)]
view: do not hide patches or signatures

It's often not that much information and may be useful
to reduce HTTP requests a reader will want to make.

8 years agofixup Plack-related requires
Eric Wong [Mon, 29 Feb 2016 10:58:39 +0000 (10:58 +0000)]
fixup Plack-related requires

We do not need to load Plack::Request outside of WWW anymore.

8 years agot/init.t: avoid spewing directory names in output
Eric Wong [Mon, 29 Feb 2016 08:21:40 +0000 (08:21 +0000)]
t/init.t: avoid spewing directory names in output

This is a step towards having consistent, reproducible
test output. (ugh, but each %hash usage screws that up).

8 years agot/search.t: use transactions to reduce I/O load
Eric Wong [Mon, 29 Feb 2016 02:48:45 +0000 (02:48 +0000)]
t/search.t: use transactions to reduce I/O load

In case folks do not use eatmydata or tmpfs for testing,
use transactions to reduce the number of fsync calls
made and hopefully prevent drives from wearing out.

8 years agogit-http-backend: fixes for mod_perl
Eric Wong [Mon, 29 Feb 2016 01:34:33 +0000 (01:34 +0000)]
git-http-backend: fixes for mod_perl

Apache2 mod_perl does not give us a real file handle, so
we must translate that before giving that to git-http-backend(1).

Also, parse the Status: correctly for errors since we failed to
set %ENV properly before the previous fix for SpawnPP

8 years agospawnpp: use env(1) for mod_perl compatibility
Eric Wong [Mon, 29 Feb 2016 01:32:24 +0000 (01:32 +0000)]
spawnpp: use env(1) for mod_perl compatibility

We cannot modify %ENV directly under mod_perl (even after forking!),
so use env(1) instead to pass the environment.

8 years agogit-http-backend: stricter parsing of CRLF
Eric Wong [Mon, 29 Feb 2016 01:05:16 +0000 (01:05 +0000)]
git-http-backend: stricter parsing of CRLF

It is not needed as we know git uses CRLF termination.

8 years agofavor procedural calls for most private functions
Eric Wong [Mon, 29 Feb 2016 00:56:20 +0000 (00:56 +0000)]
favor procedural calls for most private functions

This makes for better compile-time checking and also helps
document which calls are private for HTTP and NNTP.

While we're at it, use IO::Handle::* functions procedurally,
too, since we know we're working with native glob handles.

8 years agodistinguish error messages intended for users vs developers
Eric Wong [Mon, 29 Feb 2016 00:41:02 +0000 (00:41 +0000)]
distinguish error messages intended for users vs developers

For error messages intended to show user error (e.g. giving
invalid options), we add a newline ("\n") at the end to
polluting the output with location information.

However, for diagnosing non-user-triggered errors, we should
show the location of where the error occured.

8 years agohttp: avoid needless time2str calls
Eric Wong [Mon, 29 Feb 2016 00:29:03 +0000 (00:29 +0000)]
http: avoid needless time2str calls

Checking the time is nearly free on modern systems with
vDSO/vsyscall/similar while sprintf is always expensive.

8 years agohttp: document event_write usage
Eric Wong [Mon, 29 Feb 2016 00:13:43 +0000 (00:13 +0000)]
http: document event_write usage

It may not be obvious where we are when we enter the event_write
callback.  Hopefully this clarifies things.

8 years agohttp: error check for sysseek on input
Eric Wong [Mon, 29 Feb 2016 00:11:23 +0000 (00:11 +0000)]
http: error check for sysseek on input

Just in case we screwed up somewhere, we need to match up
syswrite to sysseek and we also favor procedural calls for
native types.

8 years agoexamples/public-inbox.psgi: relax license to GPL-3.0+
Eric Wong [Sun, 28 Feb 2016 23:06:31 +0000 (23:06 +0000)]
examples/public-inbox.psgi: relax license to GPL-3.0+

Using the AGPL for server config files is probably overkill.
GPL-3.0+ still requires appliance vendors to disclose
configurations which seems desirable for end users.

8 years agoexamples: various Apache-related doc updates
Eric Wong [Sun, 28 Feb 2016 23:03:52 +0000 (23:03 +0000)]
examples: various Apache-related doc updates

Plack::Handler::Apache2 exists and seems to work very well.

8 years agoexamples/cgi-webrick.rb: set CGIPathEnv, update comments
Eric Wong [Sun, 28 Feb 2016 22:40:22 +0000 (22:40 +0000)]
examples/cgi-webrick.rb: set CGIPathEnv, update comments

webrick clears PATH otherwise, and we rely on git commands.

8 years agoexamples/: PSGI example updates
Eric Wong [Sun, 28 Feb 2016 22:28:50 +0000 (22:28 +0000)]
examples/: PSGI example updates

Users wanting to customize their installation should know
to about the usability of STDOUT for logging.
(and we still need manpages for -nntpd and -httpd)

8 years agospawnpp: die instead of exit on exec failure
Eric Wong [Sun, 28 Feb 2016 23:39:12 +0000 (23:39 +0000)]
spawnpp: die instead of exit on exec failure

Perl may complain about exit not being executed, but not die.

8 years agonewswww: set Content-Type properly
Eric Wong [Sun, 28 Feb 2016 23:36:45 +0000 (23:36 +0000)]
newswww: set Content-Type properly

Oops :x

8 years agohttp: support graceful shutdown like nntp
Eric Wong [Sun, 28 Feb 2016 20:57:57 +0000 (20:57 +0000)]
http: support graceful shutdown like nntp

HTTP responses may be long-running or requests may be slow or
pipelined.  Ensure we don't kill them off prematurely.

8 years agoreduce calls to close unless error checks are needed
Eric Wong [Sun, 28 Feb 2016 11:28:33 +0000 (11:28 +0000)]
reduce calls to close unless error checks are needed

We can rely on timely auto-destruction based on reference
counting; reducing the chance of redundant close(2) calls
which may hit the wront FD.

We do care about certain close calls (e.g. writing to a buffered
IO handle) if we require error-checking for write-integrity.  In
other cases, let things go out-of-scope so it can be freed
automatically after use.

8 years agosearchidx: use defined for checking EOF behavior
Eric Wong [Sun, 28 Feb 2016 11:28:32 +0000 (11:28 +0000)]
searchidx: use defined for checking EOF behavior

While empty or "0" should never appear, this allows the
reviewer to think and know less about the context in which
this check is done.

8 years agot/: remove unnecessary Dumper use
Eric Wong [Sun, 28 Feb 2016 11:28:31 +0000 (11:28 +0000)]
t/: remove unnecessary Dumper use

No point in loading Data::Dumper if we do not use it
in the tests.

8 years agosearchmsg: update + fix license header
Eric Wong [Sun, 28 Feb 2016 05:26:38 +0000 (05:26 +0000)]
searchmsg: update + fix license header

Not sure how, but this should've always been AGPL-3.0+ like
the rest of the code, not GPL-3.0+

8 years agoMANIFEST: update (generate via "git ls-files")
Eric Wong [Sun, 28 Feb 2016 05:22:07 +0000 (05:22 +0000)]
MANIFEST: update (generate via "git ls-files")

It's been a while...

8 years agohttpd: allow running if ReverseProxy is missing
Eric Wong [Sun, 28 Feb 2016 00:57:11 +0000 (00:57 +0000)]
httpd: allow running if ReverseProxy is missing

Not everybody will be running this behind a ReverseProxy;
but it's probably the likely configuration.  Anyways,
warn about this and also about Deflater being missing.

8 years agospawn: disable popen optimization for non-vfork
Eric Wong [Sun, 28 Feb 2016 04:27:11 +0000 (04:27 +0000)]
spawn: disable popen optimization for non-vfork

This is necessary since we want to be able to do arbitrary redirects
via the popen interface.  Oh well, we'll be a little slower for now
for users without vfork.  vfork users will get all the performance
benefits.

8 years agospawnpp: fix error message for stderr redirect failing
Eric Wong [Sun, 28 Feb 2016 04:27:10 +0000 (04:27 +0000)]
spawnpp: fix error message for stderr redirect failing

Oops :x

8 years agodaemon: refresh before forking
Eric Wong [Sat, 27 Feb 2016 22:36:32 +0000 (22:36 +0000)]
daemon: refresh before forking

This means we always load the PSGI server code early for
-httpd.  This may make things less compatible with existing
PSGI/Plack apps, but we prioritize our httpd for the uses
of public-inbox itself, first.

And any existing PSGI/Plack app which wants to may adapt
themselves to being preload-friendly.

8 years agospawn: fail properly if Inline fails
Eric Wong [Sat, 27 Feb 2016 22:18:04 +0000 (22:18 +0000)]
spawn: fail properly if Inline fails

We must stash the error correctly when nesting evals, oops :x

8 years agomove executables to script/ directory
Eric Wong [Sat, 27 Feb 2016 21:57:57 +0000 (21:57 +0000)]
move executables to script/ directory

This seems to match more closely with what is expected of Perl
packages based on how blib is used.  Hopefully makes the top-level
source tree less cluttered and things easier-to-find.

8 years agogit: use built-in spawn implementation for vfork
Eric Wong [Sat, 27 Feb 2016 21:31:24 +0000 (21:31 +0000)]
git: use built-in spawn implementation for vfork

This should reduce overhead of spawning git processes
from our long-running httpd and nntpd servers.

8 years agoinitial spawn implementation using vfork
Eric Wong [Sat, 27 Feb 2016 02:14:23 +0000 (02:14 +0000)]
initial spawn implementation using vfork

Under Linux, vfork maintains constant performance as
parent process size increases.  fork needs to prepare pages
for copy-on-write, requiring a linear scan of the address
space.

8 years agohttp: simplify offset calculation slightly
Eric Wong [Sat, 27 Feb 2016 02:14:21 +0000 (02:14 +0000)]
http: simplify offset calculation slightly

No point in comparing an empty string; length() is only
potentially expensive on big strings.

8 years agopsgi: enable ReverseProxy middleware by default
Eric Wong [Fri, 26 Feb 2016 09:15:36 +0000 (09:15 +0000)]
psgi: enable ReverseProxy middleware by default

ReverseProxy is the common way to run Perl applications,
so enable it by default and don't care too much about fake
requests because we don't handle any sensitive information
or rely on authentication (everything is read-only from
the WWW interface and will remain so).

8 years agodaemon: update comment about usage in httpd
Eric Wong [Fri, 26 Feb 2016 09:14:21 +0000 (09:14 +0000)]
daemon: update comment about usage in httpd

Writing a read-only IMAP server isn't out-of-scope, either,
but I've never studiied the IMAP protocol, much, unlike HTTP/1.x
or even NNTP.

8 years agohttpd: avoid loading our WWW-specific code early
Eric Wong [Fri, 26 Feb 2016 03:51:47 +0000 (03:51 +0000)]
httpd: avoid loading our WWW-specific code early

We want this to be usable as a generic httpd for other Free Software
projects, so do not force users to load our WWW code at
compile-time.

8 years agowww: add News* wrappers to preload
Eric Wong [Fri, 26 Feb 2016 03:44:23 +0000 (03:44 +0000)]
www: add News* wrappers to preload

We want to preload as much as possible in -httpd when forking
to save memory via CoW.

8 years agowww: workaround for malformed NNTP links
Eric Wong [Fri, 26 Feb 2016 01:57:57 +0000 (01:57 +0000)]
www: workaround for malformed NNTP links

Some linkifiers to create invalid HTTP links when it sees a
link intended for NNTP services.  This means we may see links
to news.public-inbox.org/inbox.comp.mail.public-inbox.meta
point to "http://" on port 80 instead of 119.  Try to
redirect users to http://public-inbox.org/meta/ in this case.

8 years agosupport protocol-relative URLs in publicinbox.$LISTNAME.url
Eric Wong [Fri, 26 Feb 2016 01:57:56 +0000 (01:57 +0000)]
support protocol-relative URLs in publicinbox.$LISTNAME.url

All URL generation in dynamic HTTP pages should be capable of
generating "https" or "http" URLs depending on the user's
preference.

8 years agogit-http-backend: extract input_to_file function
Eric Wong [Fri, 26 Feb 2016 01:57:54 +0000 (01:57 +0000)]
git-http-backend: extract input_to_file function

This will allow us to more easily read and test later.

8 years agoextmsg: do not modify shared array via prurl
Eric Wong [Fri, 26 Feb 2016 02:35:04 +0000 (02:35 +0000)]
extmsg: do not modify shared array via prurl

We cannot modify elements in any shared data strucutures
shared between requests.  Oops!