]> Sergey Matveev's repositories - public-inbox.git/log
public-inbox.git
8 years agogit: allow cloning from the URL root, too
Eric Wong [Fri, 1 Jul 2016 02:09:45 +0000 (02:09 +0000)]
git: allow cloning from the URL root, too

This means we can still show non-git users a somewhat browseable
URL with a link to the README.html file while allowing git users
to type less when cloning.

All of the following are supported:

git clone https://public-inbox.org/ public-inbox
git clone https://public-inbox.org/public-inbox
git clone https://public-inbox.org/public-inbox.git
torsocks git clone http://ou63pmih66umazou.onion/public-inbox

8 years agoupdate git repo location to https:// using GitHTTPBackend.pm
Eric Wong [Fri, 1 Jul 2016 01:03:31 +0000 (01:03 +0000)]
update git repo location to https:// using GitHTTPBackend.pm

Might as well eat our own dogfood...

8 years agogithttpbackend: allow git to be a regular scalar string
Eric Wong [Fri, 1 Jul 2016 00:42:44 +0000 (00:42 +0000)]
githttpbackend: allow git to be a regular scalar string

No point in forcing users to pass a hashref/object to
get a single git directory.

8 years agosearchview: add missing newline in search results
Eric Wong [Fri, 1 Jul 2016 00:20:20 +0000 (00:20 +0000)]
searchview: add missing newline in search results

Hrm... is there a more obvious way to do an internal API for
this while still being streamable?

8 years agowww_stream: fix search for new.html endpoint
Eric Wong [Fri, 1 Jul 2016 00:20:19 +0000 (00:20 +0000)]
www_stream: fix search for new.html endpoint

We want to avoid the bare './' wherever possible, but it
doesn't seem possible here.

8 years agoview: fix permalink and raw links at the top
Eric Wong [Thu, 30 Jun 2016 18:59:12 +0000 (18:59 +0000)]
view: fix permalink and raw links at the top

Oops :x  I really need to whip check-inbox.perl into
shape or at least start running it, again.

Fixes: e29518088b3f ("view: fix up some HTML injection via Message-ID vectors")
8 years agoview: move per-message anchor to top in conversation view
Eric Wong [Thu, 30 Jun 2016 18:53:14 +0000 (18:53 +0000)]
view: move per-message anchor to top in conversation view

This fixes the '^' (up) link in the $INBOX/new.html endpoint
for search-less displays.

8 years agosearchview: show result count in thread index, for now
Eric Wong [Thu, 30 Jun 2016 10:34:00 +0000 (10:34 +0000)]
searchview: show result count in thread index, for now

I'm not sure what to show here, actually; but it's better
than triggering an uninitialized variable warning.

8 years agoview: reference posting style article on Wikipedia
Eric Wong [Thu, 30 Jun 2016 09:51:11 +0000 (09:51 +0000)]
view: reference posting style article on Wikipedia

Storage is precious when it is forever and distributed.

And public-inbox aims to not only end Eternal September(*),
but to build a world less-centralized than Usenet ever was:
forkable discussion groups

(*) https://en.wikipedia.org/wiki/Eternal_September

8 years agoview: improve readability of msg_reply with here-doc
Eric Wong [Thu, 30 Jun 2016 09:41:52 +0000 (09:41 +0000)]
view: improve readability of msg_reply with here-doc

This should make formatting more apparent since we can rely
on <pre> semantics.

8 years agowww_stream: add response wrapper sub
Eric Wong [Thu, 30 Jun 2016 09:13:21 +0000 (09:13 +0000)]
www_stream: add response wrapper sub

This encapsulates an entire PSGI response array, hopefully
making it easier to generate responses and avoid typos when
setting the Content-Type.

8 years agoview: fixup bad reference to new_msgid
Eric Wong [Thu, 30 Jun 2016 09:06:05 +0000 (09:06 +0000)]
view: fixup bad reference to new_msgid

Oops, this endpoint needs testing :x

8 years agoMerge branch 'thread-view-skel'
Eric Wong [Thu, 30 Jun 2016 08:57:49 +0000 (08:57 +0000)]
Merge branch 'thread-view-skel'

* thread-view-skel:
  view: show thread size when linking to summary
  view: default to flat/hybrid thread display
  view: fix up some HTML injection via Message-ID vectors
  www: reinstate old thread view as an option
  view: show more nearby messages in flat thread view
  view: tweak thread/index header slightly
  feed: add $INBOX/new.html endpoint
  view: merge $state hash with existing $ctx
  view: show thread context in the thread-aware flat view
  www: use WwwStream for dumping thread and search views
  www: implement hybrid flat+thread conversation view

8 years agoview: show thread size when linking to summary
Eric Wong [Thu, 30 Jun 2016 08:39:35 +0000 (08:39 +0000)]
view: show thread size when linking to summary

This should give readers a better idea of what to expect.

8 years agoview: default to flat/hybrid thread display
Eric Wong [Thu, 30 Jun 2016 08:01:39 +0000 (08:01 +0000)]
view: default to flat/hybrid thread display

This is friendlier for people on small screens and usually
eliminates the need to scroll horizontally.

8 years agoview: fix up some HTML injection via Message-ID vectors
Eric Wong [Thu, 30 Jun 2016 07:41:41 +0000 (07:41 +0000)]
view: fix up some HTML injection via Message-ID vectors

Oops, these were only introduced during the hybrid flat thread
view reworking and never deployed.

8 years agowww: reinstate old thread view as an option
Eric Wong [Thu, 30 Jun 2016 02:35:21 +0000 (02:35 +0000)]
www: reinstate old thread view as an option

This hybrid view is better than the old flat, but can
still fall down compared to the old threaded view in
some cases.

8 years agoRevert "view: update git-send-email URL"
Eric Wong [Thu, 30 Jun 2016 07:46:18 +0000 (07:46 +0000)]
Revert "view: update git-send-email URL"

This reverts commit a391cf5aaf7181b5e5e20eb240c7ce50cbdf8fa2
as kernel.org is updating again.  Ref:
http://mid.gmane.org/xmqqpor18a2n.fsf@gitster.mtv.corp.google.com

8 years agoview: show more nearby messages in flat thread view
Eric Wong [Thu, 30 Jun 2016 02:35:20 +0000 (02:35 +0000)]
view: show more nearby messages in flat thread view

Context is important, but so is conserving precious screen
space.  Decisions :<

8 years agoview: tweak thread/index header slightly
Eric Wong [Thu, 30 Jun 2016 02:35:19 +0000 (02:35 +0000)]
view: tweak thread/index header slightly

This makes the top permalink/raw as well as the In-Reply-To
show up without search.  While we're at it, try to make
the links on the thread index from the "X siblings, Y replies"
more obvious.

8 years agofeed: add $INBOX/new.html endpoint
Eric Wong [Thu, 30 Jun 2016 02:35:18 +0000 (02:35 +0000)]
feed: add $INBOX/new.html endpoint

This acts like the Atom feed; but should be viewable directly
from browsers.

8 years agoview: merge $state hash with existing $ctx
Eric Wong [Thu, 30 Jun 2016 02:35:17 +0000 (02:35 +0000)]
view: merge $state hash with existing $ctx

This reduces the level of indirection to reach certain objects
within the hash and there are no namespace or lifetime conflicts
anyways.

8 years agoview: show thread context in the thread-aware flat view
Eric Wong [Thu, 30 Jun 2016 02:35:15 +0000 (02:35 +0000)]
view: show thread context in the thread-aware flat view

This lets user have a small window of the context of
the current message relative to other threads.

8 years agowww: use WwwStream for dumping thread and search views
Eric Wong [Thu, 30 Jun 2016 02:35:14 +0000 (02:35 +0000)]
www: use WwwStream for dumping thread and search views

This allows us the HTTP server to react to backpressure
from slow clients when writing.  As a side effect, this
also makes it easier for us to maintain a consistent
header/footer across our HTML.

8 years agowww: implement hybrid flat+thread conversation view
Eric Wong [Thu, 30 Jun 2016 02:35:13 +0000 (02:35 +0000)]
www: implement hybrid flat+thread conversation view

This should be more accessible to readers on narrow terminals
(or giant fonts) while providing a chronological view which
is also aware of message threading relationships.

8 years agoview: HTML escape mailto: link
Eric Wong [Mon, 27 Jun 2016 21:13:41 +0000 (21:13 +0000)]
view: HTML escape mailto: link

Oops, we cannot have bare '&' in mailto: links, either.

8 years agoview: attribute in parent link for permalink message
Eric Wong [Mon, 27 Jun 2016 21:07:51 +0000 (21:07 +0000)]
view: attribute in parent link for permalink message

Oops, but apparently this does not trigger errors?

8 years agoinbox: avoid trying s// on undef
Eric Wong [Sun, 26 Jun 2016 23:41:00 +0000 (23:41 +0000)]
inbox: avoid trying s// on undef

Oops, I guess I'm trigger-happy today.

Fixes: 31a6ff1221fe ("inbox: ensure we do not show leading "From " lines")
8 years agoinbox: ensure we do not show leading "From " lines
Eric Wong [Sun, 26 Jun 2016 23:36:36 +0000 (23:36 +0000)]
inbox: ensure we do not show leading "From " lines

Some messages will be misimported due to an old bug,
clean them up and ensure we do not propagate the mistake.

Followup-to: a0c07cba0e5d ("mda: drop leading "From " lines again")
8 years agomda: drop leading "From " lines again
Eric Wong [Sun, 26 Jun 2016 23:06:48 +0000 (23:06 +0000)]
mda: drop leading "From " lines again

Oops...

While we're at it, drop blank lines before the "From ", too,
since it could happen.

8 years agospamcheck/spamc: fix compatibility with Perl 5.14.2
Eric Wong [Sun, 26 Jun 2016 07:58:43 +0000 (07:58 +0000)]
spamcheck/spamc: fix compatibility with Perl 5.14.2

This is necessary for Debian 7.x (wheezy), since
GLOB objects do not seem responsive to the can("fileno")
check (we do a similar check in GitHTTPBackend).

8 years agowatch_maildir: warn on spam check failures
Eric Wong [Sun, 26 Jun 2016 06:24:25 +0000 (06:24 +0000)]
watch_maildir: warn on spam check failures

It would be nice to know about spamcheck failures.

8 years agowww: unescape '+' in query parameter to space
Eric Wong [Sun, 26 Jun 2016 03:46:56 +0000 (03:46 +0000)]
www: unescape '+' in query parameter to space

Fixes: fbcb7de93884b ("www: remove a few more Plack::Request dependencies")
8 years agoTODO: add a few Xapian-related items
Eric Wong [Sat, 25 Jun 2016 22:27:55 +0000 (22:27 +0000)]
TODO: add a few Xapian-related items

"git cat-file --batch" seems expensive for big repos and
loading 70K+ tree objects in git isn't all that fast.

Ideas are cheap, time, code, and testing are not :P

8 years agowww_stream: linkify cloneurl entries if they're HTTP/HTTPS
Eric Wong [Sat, 25 Jun 2016 22:05:35 +0000 (22:05 +0000)]
www_stream: linkify cloneurl entries if they're HTTP/HTTPS

They may be other public-inbox instances which are browseable,
so provide a link to them to encourage their use as clones.

8 years agoview: safer and optional quoting for --in-reply-to arg
Eric Wong [Sat, 25 Jun 2016 21:29:43 +0000 (21:29 +0000)]
view: safer and optional quoting for --in-reply-to arg

Angle brackets around the --in-reply-to= arg for git send-email
has been optional since git v1.5.3.2, so strip them and make the
command-line argument easier-to-type.

8 years agoaddress: remove Address::from_name
Eric Wong [Sat, 25 Jun 2016 10:56:59 +0000 (10:56 +0000)]
address: remove Address::from_name

Address::names is sufficient to handle what from_name did.

8 years agoview: show To/Cc destinations in conversation view
Eric Wong [Sat, 25 Jun 2016 09:27:16 +0000 (09:27 +0000)]
view: show To/Cc destinations in conversation view

It is important to show the decentralized nature of
communication in our web views.

8 years agoaddress: beef up the module with name list extaction
Eric Wong [Sat, 25 Jun 2016 07:19:10 +0000 (07:19 +0000)]
address: beef up the module with name list extaction

We may remove from_name in the future.

...And disallow quotes in email addresses.
Technically I believe they're allowed, but they're definitely
uncommon and unlikely to show up in legitimate mail.

8 years agoinbox: do not weaken already-weak refs
Eric Wong [Fri, 24 Jun 2016 23:40:01 +0000 (23:40 +0000)]
inbox: do not weaken already-weak refs

This quiets a (hopefully harmless) warning when a ref remains
alive through several expiry timeouts.

8 years agohttp: cork chunked responses for small savings
Eric Wong [Fri, 24 Jun 2016 23:40:03 +0000 (23:40 +0000)]
http: cork chunked responses for small savings

This only affects Linux users with MSG_MORE support.

We can avoid extra TCP overhead for sub-optimal chunk sizes
by using MSG_MORE even with chunk trailers under Linux.

This breaks real-time apps which require <= 200ms latency for
streaming small packets (e.g. implementing "tail -F"), but the
public-inbox WWW code does not (and will never) do such things.

8 years agombox: reduce small packets for gzipped mboxes
Eric Wong [Fri, 24 Jun 2016 23:40:00 +0000 (23:40 +0000)]
mbox: reduce small packets for gzipped mboxes

We want to avoid sending 10 or 20-byte gzip headers as
separate TCP packets to reduce syscalls and avoid wasting
bandwidth.

8 years agogithttpbackend: shallow clone workaround
Eric Wong [Fri, 24 Jun 2016 23:39:59 +0000 (23:39 +0000)]
githttpbackend: shallow clone workaround

Apparently git-http-backend exits with a non-zero
status on shallow clones (due to git-upload-pack),
so there is a to-be-fixed bug in git.git

http://mid.gmane.org/20160621112303.GA21973@dcvr.yhbt.net
http://mid.gmane.org/20160621121041.GA29156@sigill.intra.peff.net

8 years agoevcleanup: micro-optimize asap function
Eric Wong [Fri, 24 Jun 2016 23:39:58 +0000 (23:39 +0000)]
evcleanup: micro-optimize asap function

Instead of relying on a timer with immediate callback,
arm a pipe to watch for writability, ensuring the callback
always fires.

8 years agohttp: always yield on getline/body
Eric Wong [Fri, 24 Jun 2016 23:39:57 +0000 (23:39 +0000)]
http: always yield on getline/body

We want to maximize fairness for large responses which may
download the entire mbox.

8 years agowatch_maildir: ignore Trash and Drafts, support Dovecot
Eric Wong [Fri, 24 Jun 2016 20:37:52 +0000 (20:37 +0000)]
watch_maildir: ignore Trash and Drafts, support Dovecot

Trashed messages and drafts are probably not intended for
importing, so do not import them.  Dovecot uses extra flags via
lowercase letters, so we must support those (as that's the
server I use).

8 years agowatch_maildir: implement optional spam checking
Eric Wong [Fri, 24 Jun 2016 01:15:16 +0000 (01:15 +0000)]
watch_maildir: implement optional spam checking

Mailing lists I watch and mirror may not have the best spam
filtering, and an extra layer should not hurt.

8 years agowatch_maildir: rename _check_spam => _remove_spam
Eric Wong [Fri, 24 Jun 2016 01:15:15 +0000 (01:15 +0000)]
watch_maildir: rename _check_spam => _remove_spam

We do not actually do spam checking, here; but will
do spam checking before adding a message in the future.

8 years agodocument Filesys::Notify::Simple dependency
Eric Wong [Fri, 24 Jun 2016 01:15:14 +0000 (01:15 +0000)]
document Filesys::Notify::Simple dependency

And improve documentation for existing dependencies, too.

8 years agosplit out spamcheck/spamc to its own module.
Eric Wong [Fri, 24 Jun 2016 01:15:13 +0000 (01:15 +0000)]
split out spamcheck/spamc to its own module.

This should hopefully make it easier to try other anti-spam
systems (or none at all) in the future.

8 years agoimplement ListMirror SpamAssassin plugin
Eric Wong [Fri, 24 Jun 2016 01:15:12 +0000 (01:15 +0000)]
implement ListMirror SpamAssassin plugin

When running mailing list mirrors, one needs to be careful
spammers do not try to sidestep the list server we want to
mirror from and inject email into our mail directly by setting
the appropriate list headers (e.g. "X-Mailing-List" or
"List-Id").  We trust the top-most Received: header is
the one our own mail server got the mail from.

Bcc:-ing a public mailing list is a very likely indicator of
spam in my experience, so throw in an extra rule mark it.
While public-inbox-mda rejects Bcc: entirely, public-inbox-watch
needs to mirror lists which allow Bcc.

==> list_mirror.cf <==
loadplugin PublicInbox::SaPlugin::ListMirror

ifplugin PublicInbox::SaPlugin::ListMirror
  header LIST_MIRROR_RECEIVED eval:check_list_mirror_received()
  describe LIST_MIRROR_RECEIVED Received does not match trusted list server
  score LIST_MIRROR_RECEIVED 10

  header LIST_MIRROR_BCC eval:check_list_mirror_bcc()
  describe LIST_MIRROR_BCC Mailing list was Bcc-ed
  score LIST_MIRROR_BCC 1
endif

==> ~/.spamassassin/user_prefs <==
ifplugin PublicInbox::SaPlugin::ListMirror
  list_mirror X-Mailing-List git@vger.kernel.org *.kernel.org git@vger.kernel.org
endif

8 years agoview: update git-send-email URL
Eric Wong [Wed, 22 Jun 2016 23:08:40 +0000 (23:08 +0000)]
view: update git-send-email URL

I've setup my own mirror on https://git-htmldocs.bogomips.org/
since kernel.org is no longer updated, and I don't believe
in endorsing commercial hosting sites or interests.

Unfortunately, it still contains JavaScript and CSS, but it's
generated by AsciiDoc.  Oh well, it's all Free Software, at
least.

Junio C Hamano writes:
> On Wed, Jun 22, 2016 at 12:00 PM, Eric Wong <e@80x24.org> wrote:
> > Just wondering, who updates
> > https://kernel.org/pub/software/scm/git/docs/
> > and why hasn't it been updated in a while?
> > (currently it says Last updated 2015-06-06 at the bottom)
>
> Nobody. It is too cumbersome to use their upload tool to update many
> files (it is geared towards updating a handful of tarballs at a time).

ref:
http://mid.gmane.org/20160622190018.GA786@dcvr.yhbt.net/
http://mid.gmane.org/CAPc5daUiUv-EEv7ouQ=K+Q8S64QVV5wn4H6+TuF0wLeo123K5Q@mail.gmail.com
http://mid.gmane.org/20160622214811.GA19633@dcvr.yhbt.net

8 years agoview: allow thread generation to EOF on errors
Eric Wong [Wed, 22 Jun 2016 03:57:33 +0000 (03:57 +0000)]
view: allow thread generation to EOF on errors

We want to ensure the actual message gets shown, and less
important info does not destroy things.

8 years agoview: fix topic threading when ghosts are present
Eric Wong [Wed, 22 Jun 2016 06:01:07 +0000 (06:01 +0000)]
view: fix topic threading when ghosts are present

This fixes a bug where a message replying to a ghost would
accidentally be added to the wrong topic in the index/topic
view.

Before commit 76d8f68dc273e54809ad69cfe49e141003f790ef ("view:
avoid recursion in topic index"), we would refuse to indent a
topic which started with a ghost which hid the problem.  Now we
inform the user the thread started elsewhere.

8 years agoview: support non-existent Subjects for permalink titles
Eric Wong [Tue, 21 Jun 2016 10:35:37 +0000 (10:35 +0000)]
view: support non-existent Subjects for permalink titles

Showing "(no subject)" seems like a common fallback for
messages without a Subject header.

8 years agosearch: support Subject:-less messages
Eric Wong [Tue, 21 Jun 2016 10:35:36 +0000 (10:35 +0000)]
search: support Subject:-less messages

Some mailing lists allow empty Subject headers and we shall support
searching and indexing them.

8 years agodaemon: disable SIGWINCH unless explicitly daemonized
Eric Wong [Tue, 21 Jun 2016 10:03:10 +0000 (10:03 +0000)]
daemon: disable SIGWINCH unless explicitly daemonized

Checking stdin/stdout/stderr is not sufficient as the daemon
without setsid can still be under the control of a terminal.

Unfortunately this means systemd users cannot use SIGWINCH,
either.

8 years agosearchidx: merge old thread id from ghosts
Eric Wong [Tue, 21 Jun 2016 03:16:53 +0000 (03:16 +0000)]
searchidx: merge old thread id from ghosts

We failed to discard old thread IDs when vivifying ghosts
due to out-of-order message arrival.  This rectifies the
failure and will trigger a re-index.

8 years agosearchidx: simplify ghost creation
Eric Wong [Tue, 21 Jun 2016 03:16:52 +0000 (03:16 +0000)]
searchidx: simplify ghost creation

Remove some worthless parameters and redundant no-ops
to make the next (important) patch easier-to-review.

8 years agoview: common thread walking interface
Eric Wong [Tue, 21 Jun 2016 02:58:58 +0000 (02:58 +0000)]
view: common thread walking interface

Since we have a common pattern, for walking threads,
extract it into a function and reduce the amount of code
we haev.

This will make it easier to switch to an event-driven interface
for getline, too.

8 years agoview: avoid recursion in topic index
Eric Wong [Tue, 21 Jun 2016 02:58:57 +0000 (02:58 +0000)]
view: avoid recursion in topic index

Recursion can cause problems, so do our best to avoid it
even in the topic index.

8 years agosearchview: remove recursion from thread view
Eric Wong [Tue, 21 Jun 2016 02:58:56 +0000 (02:58 +0000)]
searchview: remove recursion from thread view

As before, recursion can cause problems sooner than unshifting
objects into the head of a queue.

8 years agoview: remove recursion from expanded thread view
Eric Wong [Tue, 21 Jun 2016 02:58:55 +0000 (02:58 +0000)]
view: remove recursion from expanded thread view

This should let us generate HTML for arbitrarily deep
threads without blowing the stack.

How it renders on the client side is another matter...

8 years agoview: remove recursion from thread skeleton dump
Eric Wong [Tue, 21 Jun 2016 02:58:54 +0000 (02:58 +0000)]
view: remove recursion from thread skeleton dump

This should help prevent OOM errors from arbitrarily
deep threads and will make our streaming interface
easier-to-implement.

8 years agoview: remove dst parameter from thread skeleton dump
Eric Wong [Tue, 21 Jun 2016 02:58:53 +0000 (02:58 +0000)]
view: remove dst parameter from thread skeleton dump

We can stuff this into the state hash to reduce stack size and
hopefully improve readability.

8 years agoview: remove upfx parameter from thread skeleton dump
Eric Wong [Tue, 21 Jun 2016 02:58:52 +0000 (02:58 +0000)]
view: remove upfx parameter from thread skeleton dump

This makes the string creation somewhat simpler hopefully
makes the code easier-to-reason with.

8 years agospawn: improve error checking for fork failures
Eric Wong [Tue, 21 Jun 2016 02:39:27 +0000 (02:39 +0000)]
spawn: improve error checking for fork failures

fork failures are unfortunately common when Xapian has
gigabytes and gigabytes mmapped.

8 years agosearchview: use inbox->msg_by_mid
Eric Wong [Mon, 20 Jun 2016 23:05:02 +0000 (23:05 +0000)]
searchview: use inbox->msg_by_mid

This abstracts out the path lookup logic and and allow us
potentially allow different heads in the same repository.

We may also bypass slow tree name lookups in the future
by storing the raw blob ID in the Xapian document.

Followup-to: 4b313dc74bc9 ("feed: various object-orientation cleanups")
8 years agosearchview: fix Atom dump
Eric Wong [Mon, 20 Jun 2016 20:24:35 +0000 (20:24 +0000)]
searchview: fix Atom dump

Ugh, and I will still need to write better tests for this
(and a billion other things :x)

Fixes: 4b313dc74bc9 ("feed: various object-orientation cleanups")
8 years agonntp: use lookup_mail instead of lookup_message
Eric Wong [Mon, 20 Jun 2016 07:22:24 +0000 (07:22 +0000)]
nntp: use lookup_mail instead of lookup_message

lookup_mail is safer since it won't inadvertently load ghosts.

8 years agowww: improve topic view by scanning for ghosts
Eric Wong [Mon, 20 Jun 2016 07:22:23 +0000 (07:22 +0000)]
www: improve topic view by scanning for ghosts

This should help avoid having too many fake top-level
messages in the topic view since we only have a partial
window for threading results.

8 years agoaddress: remove quotes around names
Eric Wong [Mon, 20 Jun 2016 03:17:11 +0000 (03:17 +0000)]
address: remove quotes around names

They're needless for actual display once outside of email
headers.  But we will still show them when displaying mock
headers in the permalink view.

8 years agoinbox: move field population logic to initializer
Eric Wong [Mon, 20 Jun 2016 00:57:17 +0000 (00:57 +0000)]
inbox: move field population logic to initializer

Inboxes are normally created by Config, but having the
population logic in Inbox should make it easier to mock
for testing.

8 years agofeed: various object-orientation cleanups
Eric Wong [Mon, 20 Jun 2016 00:57:16 +0000 (00:57 +0000)]
feed: various object-orientation cleanups

Favor Inbox objects as our primary source of truth to simplify
our code.  This increases our coupling with PSGI to make it
easier to write tests in the future.

A lot of this code was originally designed to be usable
standalone without PSGI or CGI at all; but that might increase
development effort.

8 years agombox: avoid write dependency for streaming
Eric Wong [Mon, 20 Jun 2016 00:57:15 +0000 (00:57 +0000)]
mbox: avoid write dependency for streaming

Prefer to return strings instead, so Content-Length can be
calculated for caching and such.

8 years agombox: remove feed dependency
Eric Wong [Mon, 20 Jun 2016 00:57:14 +0000 (00:57 +0000)]
mbox: remove feed dependency

We do not need feed options there (or anywhere, hopefully).

8 years agofeed: remove dependence on fh->write for streaming
Eric Wong [Mon, 20 Jun 2016 00:57:13 +0000 (00:57 +0000)]
feed: remove dependence on fh->write for streaming

We'll be switching to a getline/close response body
to give the HTTP server more control when dealing
with slow clients.

8 years agofeed: avoid needless method dispatches on 404
Eric Wong [Mon, 20 Jun 2016 00:57:12 +0000 (00:57 +0000)]
feed: avoid needless method dispatches on 404

We overuse streaming, here.  Allow Content-Length to be
calculated in this case.

8 years agoMANIFEST: update with recent changes
Eric Wong [Mon, 20 Jun 2016 00:57:11 +0000 (00:57 +0000)]
MANIFEST: update with recent changes

And add a check-manifest target to the Makefile to
ensure we're up-to-date with git (but do not depend on
git).

8 years agoexamples/*@.service: wait one day for graceful shutdown
Eric Wong [Sun, 19 Jun 2016 09:59:31 +0000 (09:59 +0000)]
examples/*@.service: wait one day for graceful shutdown

Because sometimes folks will want to download gigantic mboxes
or make large clones over Tor which are not resume-friendly.

Note: the timeout logic in nntpd is somewhat over-aggressive
and can break some large slrnpulls.  This ought to be easily
recoverable on the client-side, though, since it's based on
per-message fetches.

8 years agosearch: reopen and retry on updated databases
Eric Wong [Sun, 19 Jun 2016 09:05:00 +0000 (09:05 +0000)]
search: reopen and retry on updated databases

This seems like a nasty thing which breaks downloads of
large mailboxes.

8 years agohttp: constrain getline/close responses by time
Eric Wong [Sun, 19 Jun 2016 06:55:42 +0000 (06:55 +0000)]
http: constrain getline/close responses by time

This allows us to yield control to other clients gracefully if
getline takes too long to generate a chunk.  This is more
expensive but should not cost a syscall on modern 64-bit systems.

8 years agohttp: avoid recursion when hitting write count limit
Eric Wong [Sun, 19 Jun 2016 06:32:41 +0000 (06:32 +0000)]
http: avoid recursion when hitting write count limit

Use the EvCleanup::asap handler to reschedule our writes
after yielding to other clients.

8 years agombox: set gzip timestamp to the Unix epoch
Eric Wong [Sun, 19 Jun 2016 04:50:40 +0000 (04:50 +0000)]
mbox: set gzip timestamp to the Unix epoch

This allows consistency between different invocations from
roughly the same period and is no worse for caching any any of
our existing HTML and Atom feeds.

We cannot set the timestamp to the end date since messages
may be added to the repository while we are iterating
(and this streaming mechanism will pick them up).

8 years agowatch_maildir: tighten up path checks
Eric Wong [Sun, 19 Jun 2016 02:13:52 +0000 (02:13 +0000)]
watch_maildir: tighten up path checks

Only mark seen messages as spam, otherwise it could be
too aggressive and cause problems or over training.
We wouldn't want a wayward FIFO ruining our day, either :)

8 years agoimport: allow messages without subject
Eric Wong [Sun, 19 Jun 2016 00:23:27 +0000 (00:23 +0000)]
import: allow messages without subject

Because our WatchMaildir module is liberal about what
it accepts, we can potentially have messages without a
subject.

8 years agowatch_maildir: spam removal support
Eric Wong [Sat, 18 Jun 2016 23:25:20 +0000 (23:25 +0000)]
watch_maildir: spam removal support

We can support spam removal by watching a special "spam"
Maildir, too.  We can run public-inbox-learn as a separate
step, and that command will be improved to support
auto-learning, too.

8 years agowatch_maildir: add scan test
Eric Wong [Sat, 18 Jun 2016 22:23:52 +0000 (22:23 +0000)]
watch_maildir: add scan test

This should be portable despite the intended use of this
directory being non-portable.

8 years agoemergency: avoid needless mkpath dependency
Eric Wong [Sat, 18 Jun 2016 22:11:01 +0000 (22:11 +0000)]
emergency: avoid needless mkpath dependency

Be more explicit and slightly speed up tests.

8 years agodaemon: be less misleading about graceful shutdown
Eric Wong [Sat, 18 Jun 2016 10:51:37 +0000 (10:51 +0000)]
daemon: be less misleading about graceful shutdown

We do not need to count the httpd.async object
against our running client count, that is tied to
the socket of the actual client.

This prevents misleading sysadmins about connected
clients during shutdown.

8 years agospawn: try to keep signals blocked in spawned child
Eric Wong [Sat, 18 Jun 2016 10:53:32 +0000 (10:53 +0000)]
spawn: try to keep signals blocked in spawned child

While we only want to stop our daemons and gracefully destroy
subprocesses, it is common for 'Ctrl-C' from a terminal to kill
the entire pgroup.

Killing an entire pgroup nukes subprocesses like git-upload-pack
breaks graceful shutdown on long clones.  Make a best effort to
ensure git-upload-pack processes are not broken when somebody
signals an entire process group.

Followup-to: commit 37bf2db81bbbe114d7fc5a00e30d3d5a6fa74de5
("doc: systemd examples should only kill one process")

8 years agoview: consolidate per-message newline handling
Eric Wong [Sat, 18 Jun 2016 08:49:05 +0000 (08:49 +0000)]
view: consolidate per-message newline handling

We don't want to blindly append a trailing newline
if the message ends in quoted text leading to a <span>,
as a newline is already added to a <span>...

8 years agoview: minor tweaks to reduce long lines
Eric Wong [Sat, 18 Jun 2016 00:22:34 +0000 (00:22 +0000)]
view: minor tweaks to reduce long lines

Fold addressee fields to better delimit destinations,
reduce horizontal rule <hr /> to reduce scrolling,
and use spaces to indent "git send-email" example.

8 years agoview: introduce WwwStream interface
Eric Wong [Fri, 17 Jun 2016 21:32:59 +0000 (21:32 +0000)]
view: introduce WwwStream interface

This will allow us to commonalize HTML generation in the future
and is the start of moving existing HTML generation to a "pull"
streaming model (from the existing "push" one).

Using the getline/close pull model is superior to the existing
$fh->write streaming as it allows us to throttle response
generation based on backpressure from slow clients.

8 years agofeed: split out top-of-page generation
Eric Wong [Fri, 17 Jun 2016 19:26:45 +0000 (19:26 +0000)]
feed: split out top-of-page generation

This will eventually allow us to reuse code to generate a common
header.

8 years agowww: undefined query string values are empty strings
Eric Wong [Fri, 17 Jun 2016 21:06:38 +0000 (21:06 +0000)]
www: undefined query string values are empty strings

We use very short query parameters for search, so "&r"
without a '=' implies truth for 'r' (relevance).

8 years agofilter/base: reject more types by default
Eric Wong [Fri, 17 Jun 2016 17:47:59 +0000 (17:47 +0000)]
filter/base: reject more types by default

Try to be descriptive for some of these.

8 years agowww: escape HTML in footer description
Eric Wong [Fri, 17 Jun 2016 18:56:02 +0000 (18:56 +0000)]
www: escape HTML in footer description

This isn't a security vulnerability since $GIT_DIR/description
is controlled by the admin; but it causes the footer to
misrender.

8 years agoremove dependency on IPC::Run
Eric Wong [Fri, 17 Jun 2016 02:44:59 +0000 (02:44 +0000)]
remove dependency on IPC::Run

We no longer depend on it for the core code, and tests
are optional for users.  Hopefully this makes this
easier-to-install.

8 years agoimport: auto-update index when done
Eric Wong [Fri, 17 Jun 2016 01:56:05 +0000 (01:56 +0000)]
import: auto-update index when done

This prevents multiple update processes from stepping over
each other while called under the lock, and also allows the
new -watch process to update the index iff indexing was
desired.