net_reader: restart on first UID when UIDVALIDITY changes

In other words, treat the same IMAP folder with a different
UIDVALIDITY as a completely different folder.  If the UIDVALIDITY
changes, we can start from UID=1 without falling behind or
losing data.  If the UIDVALIDITY gets reset to a previously
known-good message, we can still resume where we left off
before the first UIDVALIDITY change.

This affects public-inbox-watch and "lei import"

One potential downside of this is for rare altid users, but
that's mainly intended for NNTP article numbers which are/were
often publicized; not IMAP UIDs which are rarely publicized.

The other potential downside is bandwidth waste in in the rare
case UIDVALIDITY changes while IMAP folder contents remain
unchanged.  There's no extra storage used due to existing
(v1|v2|lei/store) deduplication mechanisms.

Before this change, we were matching offlineimap behavior and
stopped synching an IMAP folder when its UIDVALIDITY changed.
offlineimap behavior made sense for IMAP <=> Maildir
synchronization since Maildirs had no sense of UIDVALIDITY and
could only rely on name mapping.

diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
index bcf7af2e11b5de2e63a5a30bd13dfc7fb159bda8..fe8135823840de7b3f6099658ddbf12307f3ee6d 100644 (file)
--- a/lib/PublicInbox/IMAPTracker.pm
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -39,12 +39,20 @@ sub dbh_new ($) {
        $dbh;
 }
 
-sub get_last ($) {
-       my ($self) = @_;
-       my $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+sub get_last ($;$) {
+       my ($self, $validity) = @_;
+       my $sth;
+       if (defined $validity) {
+               $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+SELECT uid_validity, uid FROM imap_last WHERE url = ? AND uid_validity = ?
+
+               $sth->execute($self->{url}, $validity);
+       } else {
+               $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
 SELECT uid_validity, uid FROM imap_last WHERE url = ?
 
-       $sth->execute($self->{url});
+               $sth->execute($self->{url});
+       }
        $sth->fetchrow_array;
 }
 

diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c7b43f014f6bd31d5c2e435f8ef3b71640cbfa17..39129b342f0134f59e42033d1748a798ac271aac 100644 (file)
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -397,17 +397,12 @@ sub _imap_fetch_all ($$$) {
        $r_uidnext //= $mic->uidnext($mbx) //
                return "E: $uri cannot get UIDNEXT";
        my $itrk = _itrk($self, $uri);
-       my ($l_uidval, $l_uid) = $itrk ? $itrk->get_last : ();
-       $l_uidval //= $r_uidval; # first time
+       my $l_uid;
+       $l_uid = $itrk->get_last($r_uidval) if $itrk;
        $l_uid //= 0;
-       if ($l_uidval != $r_uidval) {
-               return "E: $uri UIDVALIDITY mismatch\n".
-                       "E: local=$l_uidval != remote=$r_uidval";
-       }
        my $r_uid = $r_uidnext - 1;
        if ($l_uid > $r_uid) {
-               return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n".
-                       "E: $uri strangely, UIDVALIDLITY matches ($l_uidval)\n";
+               return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n";
        }
        return if $l_uid >= $r_uid; # nothing to do
        $l_uid ||= 1;