=head1 SYNOPSIS
+ public-inbox-netd
public-inbox-httpd
public-inbox-imapd
public-inbox-nntpd
+ public-inbox-pop3d
=head1 DESCRIPTION
This manual describes common options and behavior for
public-inbox network daemons. Network daemons for public-inbox
-provide read-only NNTP, IMAP and HTTP access to public-inboxes. Write
-access to a public-inbox will never be required to run these.
+provide read-only IMAP, HTTP, NNTP and POP3 access to public-inboxes.
+Write access to a public-inbox will never be required to run these.
These daemons are implemented with a common core using
non-blocking sockets and optimized for fairness; even with
=over
-=item -l ADDRESS
+=item -l [PROTO://]ADDRESS[?opt1=val1,opt2=val2]
-=item --listen ADDRESS
+=item --listen [PROTO://]ADDRESS[?opt1=val1,opt2=val2]
This takes an absolute path to a Unix socket or HOST:PORT
to listen on. For example, to listen to TCP connections on
May be specified multiple times to allow listening on multiple
sockets.
-This does not need to be specified at all if relying on
-L<systemd.socket(5)> or similar
+Unless per-listener options are used (required for
+L<public-inbox-netd(1)>), this does not need to be specified at
+all if relying on L<systemd.socket(5)> or similar,
+
+Per-listener options may be specified after C<?> as C<KEY=VALUE>
+pairs delimited by C<,>. See L<public-inbox-netd(1)> for
+documentation on the C<cert=>, C<key=>, C<env.NAME=VALUE>,
+C<out=>, C<err=>, and C<psgi=> options available.
Default: server-dependent unless socket activation is used with
L<systemd(1)> or similar (see L<systemd.socket(5)>).
(e.g. E<gt>E<gt>/path/to/log in shell) since it allows
SIGUSR1 to be handled (see L<SIGNALS/SIGNALS> below).
-Default: /dev/null
+C<out=> may also be specified on a per-listener basis.
+
+Default: /dev/null with C<--daemonize>, inherited otherwise
=item -2 PATH
Like C<--stdout>, but for the stderr descriptor (2).
+C<err=> may also be specified on a per-listener basis.
+
+Default: /dev/null with C<--daemonize>, inherited otherwise
+
=item -W
=item --worker-processes
Default: 1
+=item --cert /path/to/cert
+
+The default TLS certificate for HTTPS, IMAPS, NNTPS, POP3S and/or STARTTLS
+support if the C<cert> option is not given with C<--listen>.
+
+Well-known TCP ports automatically get TLS or STARTTLS support
+If using systemd-compatible socket activation and a TCP listener
+on port well-known ports (563 is inherited, it is automatically
+NNTPS when this option is given. When a listener on port 119 is
+inherited and this option is given, it automatically gets
+STARTTLS support.
+
+=item --key /path/to/key
+
+The default TLS certificate key for the default C<--cert> or
+per-listener C<cert=> option. The private key may be
+concatenated into the path used by the cert, in which case this
+option is not needed.
+
=back
=head1 SIGNALS
=head1 COPYRIGHT
-Copyright 2013-2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
L<public-inbox-httpd(1)>, L<public-inbox-imapd(1)>,
-L<public-inbox-nntpd(1)>
+L<public-inbox-nntpd(1)>, L<public-inbox-pop3d(1)>, L<public-inbox-netd(1)>
=head1 DESCRIPTION
-public-inbox-netd provides a read-only HTTP/IMAP/NNTP/POP3 daemon for
-public-inbox. It uses options and environment variables common
-to all L<public-inbox-daemon(8)> implementations.
+public-inbox-netd provides a read-only multi-protocol
+(HTTP/IMAP/NNTP/POP3) daemon for public-inbox. It uses options
+and environment variables common to all
+L<public-inbox-daemon(8)> implementations.
The default configuration will never require write access
to the directory where the public-inbox is stored, so it
=item --listen PROTO://ADDRESS/?cert=/path/to/cert,key=/path/to/key
+=item -l http://ADDRESS/?env.PI_CONFIG=/path/to/cfg,psgi=/path/to/app.psgi
+
In addition to the normal C<-l>/C<--listen> switch described in
L<public-inbox-daemon(8)>, the protocol prefix (e.g. C<nntp://> or
C<nntps://>) may be specified to force a given protocol.
+Environment variable overrides in effect during loading and
+reloading (SIGHUP) can be specified as C<env.NAME=VALUE> for
+all protocols.
+
+HTTP(S) listeners may also specify C<psgi=> to use a different
+C<.psgi> file for each listener.
+
+C<err=/path/to/errors.log> may be used to isolate error/debug output
+for a particular listener away from C<--stderr>.
+
+Non-HTTP(S) listeners may also specify C<out=> for logging to
+C<stdout>. HTTP(S) users are encouraged to configure
+L<Plack::Middleware::AccessLog> or
+L<Plack::Middleware::AccessLog::Timed>, instead.
+
=item --cert /path/to/cert
-The default TLS certificate for optional TLS support
-if the C<cert> option is not given with C<--listen>.
+See L<public-inbox-daemon(1)>.
=item --key /path/to/key
-The default private TLS certificate key for optional TLS support
-if the C<key> option is not given with C<--listen>. The private
-key may be concatenated into the path used by C<--cert>, in which case this
-option is not needed.
+See L<public-inbox-daemon(1)>.
=back
=item publicinbox.nntpserver
+=item publicinbox.pop3state
+
=back
See L<public-inbox-config(5)> for documentation on them.
t/.gitconfig
t/address.t
t/admin.t
+t/alt.psgi
t/altid.t
t/altid_v2.t
t/cgi.t
use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev);
use IO::Handle; # ->autoflush
use IO::Socket;
+use File::Spec;
use POSIX qw(WNOHANG :signal_h);
use Socket qw(IPPROTO_TCP SOL_SOCKET);
STDOUT->autoflush(1);
my (@cfg_listen, $stdout, $stderr, $group, $user, $pid_file, $daemonize);
my $worker_processes = 1;
my @listeners;
-my %pids;
+my (%pids, %logs);
my %tls_opt; # scheme://sockname => args for IO::Socket::SSL->start_SSL
my $reexec_pid;
my ($uid, $gid);
my %KNOWN_TLS = (443 => 'https', 563 => 'nntps', 993 => 'imaps', 995 =>'pop3s');
my %KNOWN_STARTTLS = (110 => 'pop3', 119 => 'nntp', 143 => 'imap');
-sub accept_tls_opt ($) {
- my ($opt_str) = @_;
- # opt_str: opt1=val1,opt2=val2 (opt may repeat for multi-value)
- require PublicInbox::TLS;
+sub listener_opt ($) {
+ my ($str) = @_; # opt1=val1,opt2=val2 (opt may repeat for multi-value)
my $o = {};
# allow ',' as delimiter since '&' is shell-unfriendly
- foreach (split(/[,&]/, $opt_str)) {
+ for (split(/[,&]/, $str)) {
my ($k, $v) = split(/=/, $_, 2);
- push @{$o->{$k} ||= []}, $v;
+ push @{$o->{$k}}, $v;
}
# key may be a part of cert. At least
# p5-io-socket-ssl/example/ssl_server.pl has this fallback:
- $o->{cert} //= [ $default_cert ];
+ $o->{cert} //= [ $default_cert ] if defined($default_cert);
$o->{key} //= defined($default_key) ? [ $default_key ] : $o->{cert};
+ $o;
+}
+
+sub accept_tls_opt ($) {
+ my ($opt) = @_;
+ my $o = ref($opt) eq 'HASH' ? $opt : listener_opt($opt);
+ return if !defined($o->{cert});
+ require PublicInbox::TLS;
my %ctx_opt = (SSL_server => 1);
# parse out hostname:/path/to/ mappings:
- foreach my $k (qw(cert key)) {
+ for my $k (qw(cert key)) {
+ $o->{$k} // next;
my $x = $ctx_opt{'SSL_'.$k.'_file'} = {};
foreach my $path (@{$o->{$k}}) {
my $host = '';
{ SSL_server => 1, SSL_startHandshake => 0, SSL_reuse_ctx => $ctx };
}
-sub load_mod ($) {
- my ($scheme) = @_;
+sub check_absolute ($$) {
+ my ($var, $val) = @_;
+ die <<EOM if index($val // '/', '/') != 0;
+$var must be an absolute path when using --daemonize: $val
+EOM
+}
+
+sub do_chown ($) {
+ $uid // return;
+ my ($path) = @_;
+ chown($uid, $gid, $path) or warn "chown $path: $!\n";
+}
+
+sub open_log_path ($$) { # my ($fh, $path) = @_; # $_[0] is modified
+ open $_[0], '>>', $_[1] or die "open(>> $_[1]): $!";
+ $_[0]->autoflush(1);
+ do_chown($_[1]);
+}
+
+sub load_mod ($;$) {
+ my ($scheme, $opt) = @_;
my $modc = "PublicInbox::\U$scheme";
my $mod = $modc.'D';
eval "require $mod"; # IMAPD|HTTPD|NNTPD|POP3D
die $@ if $@;
my %xn;
my $tlsd = $xn{tlsd} = $mod->new;
- $xn{refresh} = sub { $tlsd->refresh_groups(@_) };
+ my %env = map {
+ substr($_, length('env.')) => $opt->{$_}->[-1];
+ } grep(/\Aenv\./, keys %$opt);
+ $xn{refresh} = sub {
+ my ($sig) = @_;
+ local @ENV{keys %env} = values %env;
+ $tlsd->refresh_groups($sig);
+ };
$xn{post_accept} = $tlsd->can('post_accept_cb') ?
$tlsd->post_accept_cb : sub { $modc->new($_[0], $tlsd) };
- $xn{af_default} = 'httpready' if $modc eq 'PublicInbox::HTTP';
+ my @paths = qw(out err);
+ if ($modc eq 'PublicInbox::HTTP') {
+ @paths = qw(err);
+ $xn{af_default} = 'httpready';
+ if (my $p = $opt->{psgi}) {
+ die "multiple psgi= options specified\n" if @$p > 1;
+ check_absolute('psgi=', $p->[0]) if $daemonize;
+ $tlsd->{psgi} = $p->[0];
+ }
+ }
+ for my $f (@paths) {
+ my $p = $opt->{$f} or next;
+ die "multiple $f= options specified\n" if @$p > 1;
+ check_absolute("$f=", $p->[0]) if $daemonize;
+ $p = File::Spec->canonpath($p->[0]);
+ open_log_path(my $fh, $p);
+ $tlsd->{$f} = $logs{$p} = $fh;
+ }
\%xn;
}
GetOptions(%opt) or die $help;
if ($show_help) { print $help; exit 0 };
+ $_ = File::Spec->canonpath($_ // next) for ($stdout, $stderr);
if (defined $pid_file && $pid_file =~ /\.oldbin\z/) {
die "--pid-file cannot end with '.oldbin'\n";
}
my $s = $KNOWN_TLS{$1} // $KNOWN_STARTTLS{$1};
$scheme = $s if defined $s;
}
+ my $opt; # non-TLS options
if ($l =~ s!/?\?(.+)\z!!) {
- $tls_opt{"$scheme://$l"} = accept_tls_opt($1);
+ $opt = listener_opt($1);
+ $tls_opt{"$scheme://$l"} = accept_tls_opt($opt);
} elsif (defined($default_cert)) {
$tls_opt{"$scheme://$l"} = accept_tls_opt('');
} elsif ($scheme =~ /\A(?:https|imaps|nntps|pop3s)\z/) {
die "$orig specified w/o cert=\n";
}
$scheme =~ /\A(http|imap|nntp|pop3)/ and
- $xnetd->{$l} = load_mod($1);
+ $xnetd->{$l} = load_mod($1, $opt);
next if $listener_names->{$l}; # already inherited
my (%o, $sock_pkg);
$tls_opt{''} ||= accept_tls_opt('');
}
}
-
+ my @d;
+ while (my ($k, $v) = each %tls_opt) { push(@d, $k) if !defined($v) }
+ delete @tls_opt{@d};
die "No listeners bound\n" unless @listeners;
}
-sub check_absolute ($$) {
- my ($var, $val) = @_;
- if (defined $val && index($val, '/') != 0) {
- die
-"--$var must be an absolute path when using --daemonize: $val\n";
- }
-}
-
sub daemonize () {
if ($daemonize) {
require Cwd;
next unless -e $arg;
$ARGV[$i] = Cwd::abs_path($arg);
}
- check_absolute('stdout', $stdout);
- check_absolute('stderr', $stderr);
- check_absolute('pid-file', $pid_file);
+ check_absolute('--stdout', $stdout);
+ check_absolute('--stderr', $stderr);
+ check_absolute('--pid-file', $pid_file);
chdir '/' or die "chdir failed: $!";
}
}
sub reopen_logs {
- if ($stdout) {
- open STDOUT, '>>', $stdout or
- warn "failed to redirect stdout to $stdout: $!\n";
- STDOUT->autoflush(1);
- do_chown($stdout);
- }
- if ($stderr) {
- open STDERR, '>>', $stderr or
- warn "failed to redirect stderr to $stderr: $!\n";
- STDERR->autoflush(1);
- do_chown($stderr);
- }
+ $logs{$stdout} //= \*STDOUT if defined $stdout;
+ $logs{$stderr} //= \*STDERR if defined $stderr;
+ while (my ($p, $fh) = each %logs) { open_log_path($fh, $p) }
}
sub sockname ($) {
# ->DESTROY runs when $for_destroy goes out-of-scope
}
-sub do_chown ($) {
- my ($path) = @_;
- if (defined $uid and !chown($uid, $gid, $path)) {
- warn "could not chown $path: $!\n";
- }
-}
-
sub write_pid ($) {
my ($path) = @_;
Net::Server::Daemonize::create_pid_file($path);
# we have a different env for ever listener socket for
# SERVER_NAME, SERVER_PORT and psgi.url_scheme
# envs: listener FD => PSGI env
-sub new { bless { envs => {} }, __PACKAGE__ }
+sub new { bless { envs => {}, err => \*STDERR }, __PACKAGE__ }
# this becomes {srv_env} in PublicInbox::HTTP
sub env_for ($$$) {
SERVER_PORT => $port,
SCRIPT_NAME => '',
'psgi.version' => [ 1, 1 ],
- 'psgi.errors' => \*STDERR,
+ 'psgi.errors' => $self->{err},
'psgi.url_scheme' => $client->can('accept_SSL') ?
'https' : 'http',
'psgi.nonblocking' => Plack::Util::TRUE,
sub refresh_groups {
my ($self) = @_;
my $app;
- if (@main::ARGV) {
- eval { $app = Plack::Util::load_psgi(@ARGV) };
+ $self->{psgi} //= $main::ARGV[0] if @main::ARGV;
+ if ($self->{psgi}) {
+ eval { $app = Plack::Util::load_psgi($self->{psgi}) };
die $@, <<EOM if $@;
$0 runs in /, command-line paths must be absolute
EOM
# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# represents an IMAPD (currently a singleton),
-# see script/public-inbox-imapd for how it is used
+# represents an IMAPD, see script/public-inbox-imapd for how it is used
package PublicInbox::IMAPD;
use strict;
use v5.10.1;
sub new {
my ($class) = @_;
- my $pi_cfg = PublicInbox::Config->new;
- my $name = $pi_cfg->{'publicinbox.nntpserver'};
- if (!defined($name) or $name eq '') {
- $name = hostname;
- } elsif (ref($name) eq 'ARRAY') {
- $name = $name->[0];
- }
-
bless {
- groups => {},
err => \*STDERR,
out => \*STDOUT,
- pi_cfg => $pi_cfg,
- servername => $name,
- greet => \"201 $name ready - post via email\r\n",
+ # pi_cfg => $pi_cfg,
# accept_tls => { SSL_server => 1, ..., SSL_reuse_ctx => ... }
# idler => PublicInbox::InboxIdle
}, $class;
sub refresh_groups {
my ($self, $sig) = @_;
- my $pi_cfg = $sig ? PublicInbox::Config->new : $self->{pi_cfg};
+ my $pi_cfg = PublicInbox::Config->new;
+ my $name = $pi_cfg->{'publicinbox.nntpserver'};
+ if (!defined($name) or $name eq '') {
+ $name = hostname;
+ } elsif (ref($name) eq 'ARRAY') {
+ $name = $name->[0];
+ }
+ if ($name ne ($self->{servername} // '')) {
+ $self->{servername} = $name;
+ $self->{greet} = \"201 $name ready - post via email\r\n";
+ }
my $groups = $pi_cfg->{-by_newsgroup}; # filled during each_inbox
my $cache = eval { $pi_cfg->ALL->misc->nntpd_cache_load } // {};
$pi_cfg->each_inbox(sub {
# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# represents an POP3D (currently a singleton)
+# represents an POP3D
package PublicInbox::POP3D;
use v5.12;
use parent qw(PublicInbox::Lock);
die "File::FcntlLock required for POP3 on $^O: $@\n";
sub new {
- my ($cls, $pi_cfg) = @_;
- $pi_cfg //= PublicInbox::Config->new;
- my $d = $pi_cfg->{'publicinbox.pop3state'} //
- die "publicinbox.pop3state undefined\n";
- -d $d or do {
- require File::Path;
- File::Path::make_path($d, { mode => 0700 });
- PublicInbox::Syscall::nodatacow_dir($d);
- };
+ my ($cls) = @_;
bless {
err => \*STDERR,
out => \*STDOUT,
- pi_cfg => $pi_cfg,
- lock_path => "$d/db.lock", # PublicInbox::Lock to protect SQLite
+ # pi_cfg => PublicInbox::Config
+ # lock_path => ...
# interprocess lock is the $pop3state/txn.locks file
# txn_locks => {}, # intraworker locks
# accept_tls => { SSL_server => 1, ..., SSL_reuse_ctx => ... }
my ($self, $sig) = @_;
# TODO share pi_cfg with nntpd/imapd inside -netd
my $new = PublicInbox::Config->new;
- my $old = $self->{pi_cfg};
- my $s = 'publicinbox.pop3state';
- $new->{$s} //= $old->{$s};
- if ($new->{$s} ne $old->{$s}) {
- warn <<EOM;
+ my $d = $new->{'publicinbox.pop3state'} //
+ die "publicinbox.pop3state undefined ($new->{-f})\n";
+ -d $d or do {
+ require File::Path;
+ File::Path::make_path($d, { mode => 0700 });
+ PublicInbox::Syscall::nodatacow_dir($d);
+ };
+ $self->{lock_path} //= "$d/db.lock";
+ if (my $old = $self->{pi_cfg}) {
+ my $s = 'publicinbox.pop3state';
+ $new->{$s} //= $old->{$s};
+ return warn <<EOM if $new->{$s} ne $old->{$s};
$s changed: `$old->{$s}' => `$new->{$s}', config reload ignored
EOM
- } else {
- $self->{pi_cfg} = $new;
}
+ $self->{pi_cfg} = $new;
}
# persistent tables
--- /dev/null
+# Copyright (C) all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use v5.12;
+use warnings;
+use Plack::Builder;
+my $pi_config = $ENV{PI_CONFIG} // 'unset'; # capture ASAP
+my $app = sub {
+ my ($env) = @_;
+ $env->{'psgi.errors'}->print("ALT\n");
+ [ 200, ['Content-Type', 'text/plain'], [ $pi_config ] ]
+};
+
+builder {
+ enable 'ContentLength';
+ enable 'Head';
+ $app;
+}
-# Copyright (C) 2016-2021 all contributors <meta@public-inbox.org>
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# corner case tests for the generic PSGI server
# Usage: plackup [OPTIONS] /path/to/this/file
-use strict;
+use v5.12;
use warnings;
use Plack::Builder;
require Digest::SHA;
+my $pi_config = $ENV{PI_CONFIG} // 'unset'; # capture ASAP
my $app = sub {
my ($env) = @_;
my $path = $env->{PATH_INFO};
} elsif ($path eq '/url_scheme') {
$code = 200;
push @$body, $env->{'psgi.url_scheme'}
+ } elsif ($path eq '/PI_CONFIG') {
+ $code = 200;
+ push @$body, $pi_config; # show value at ->refresh_groups
}
[ $code, $h, $body ]
};
-# Copyright (C) 2016-2021 all contributors <meta@public-inbox.org>
+#!perl -w
+# Copyright (C) all contributors <meta@public-inbox.org>
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# note: our HTTP server should be standalone and capable of running
# generic PSGI/Plack apps.
my $err = "$tmpdir/stderr.log";
my $out = "$tmpdir/stdout.log";
my $psgi = "./t/httpd-corner.psgi";
-my $sock = tcp_server() or die;
+my $sock = tcp_server();
my @zmods = qw(PublicInbox::GzipFilter IO::Uncompress::Gunzip);
# Make sure we don't clobber socket options set by systemd or similar
my $upath = "$tmpdir/s";
my $unix = unix_server($upath);
+my $alt = tcp_server();
my $td;
my $spawn_httpd = sub {
my (@args) = @_;
- my $cmd = [ '-httpd', @args, "--stdout=$out", "--stderr=$err", $psgi ];
- $td = start_script($cmd, undef, { 3 => $sock, 4 => $unix });
+ my $x = tcp_host_port($alt);
+ my $cmd = [ '-httpd', @args, "--stdout=$out", "--stderr=$err", $psgi,
+ '-l', "http://$x/?psgi=t/alt.psgi,env.PI_CONFIG=/path/to/alt".
+ ",err=$tmpdir/alt.err" ];
+ my $env = { PI_CONFIG => '/dev/null' };
+ $td = start_script($cmd, $env, { 3 => $sock, 4 => $unix, 5 => $alt });
};
$spawn_httpd->();
+{
+ my $conn = conn_for($alt, 'alt PSGI path');
+ $conn->write("GET / HTTP/1.0\r\n\r\n");
+ $conn->read(my $buf, 4096);
+ like($buf, qr!^/path/to/alt\z!sm,
+ 'alt.psgi loaded on alt socket with correct env');
+
+ $conn = conn_for($sock, 'default PSGI path');
+ $conn->write("GET /PI_CONFIG HTTP/1.0\r\n\r\n");
+ $conn->read($buf, 4096);
+ like($buf, qr!^/dev/null\z!sm,
+ 'default PSGI on original socket');
+ my $log = capture("$tmpdir/alt.err");
+ ok(grep(/ALT/, @$log), 'alt psgi.errors written to');
+ $log = capture($err);
+ ok(!grep(/ALT/, @$log), 'STDERR not written to');
+ is(unlink($err, "$tmpdir/alt.err"), 2, 'unlinked stderr and alt.err');
+
+ $td->kill('USR1'); # trigger reopen_logs
+}
+
if ('test worker death') {
my $conn = conn_for($sock, 'killed worker');
$conn->write("GET /pid HTTP/1.1\r\nHost:example.com\r\n\r\n");
like($body, qr/\A[0-9]+\z/, '/pid response');
isnt($body, $pid, 'respawned worker');
}
+{ # check on prior USR1 signal
+ ok(-e $err, 'stderr recreated after USR1');
+ ok(-e "$tmpdir/alt.err", 'alt.err recreated after USR1');
+}
{
my $conn = conn_for($sock, 'Header spaces bogus');
$conn->write("GET /empty HTTP/1.1\r\nSpaced-Out : 3\r\n\r\n");