1 From e@80x24.org Wed May 9 20:23:03 2018
2 Date: Wed, 9 May 2018 20:23:03 +0000
3 From: Eric Wong <e@80x24.org>
4 To: meta@public-inbox.org
5 Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
6 Subject: [ANNOUNCE] public-inbox 1.1.0-pre1
7 Message-ID: <20180509202303.GA15156@dcvr>
9 Pre-release for v2 repository support.
10 Thanks to The Linux Foundation for supporting this work!
12 https://public-inbox.org/releases/public-inbox-1.1.0-pre1.tar.gz
14 SHA-256: d0023770a63ca109e6fe2c58b04c58987d4f81572ac69d18f95d6af0915fa009
15 (only intended to guard against accidental file corruption)
20 nntp: improve fairness during XOVER and similar commands
21 nntp: do not drain rbuf if there is a command pending
22 extmsg: use news.gmane.org for Message-ID lookups
23 searchview: fix non-numeric comparison
24 mbox: do not barf on queries which return no results
25 nntp: allow and ignore empty commands
26 ensure SQLite and Xapian files respect core.sharedRepository
27 TODO: a few more updates
28 filter/rubylang: do not set altid on spam training
29 import: cleanup git cat-file processes when ->done
30 disallow "\t" and "\n" in OVER headers
31 searchidx: release lock again during v1 batch callback
32 searchidx: remove leftover debugging code
33 convert: copy description and git config from v1 repo
34 view: untangle loop when showing message headers
35 view: wrap To: and Cc: headers in HTML display
36 view: drop redundant References: display code
37 TODO: add EPOLLEXCLUSIVE item
38 searchview: do not blindly append "l" parameter to URL
39 search: avoid repeated mbox results from search
40 msgmap: add limit to response for NNTP
41 thread: prevent hidden threads in /$INBOX/ landing page
42 thread: sort incoming messages by Date
43 searchidx: preserve umask when starting/committing transactions
44 scripts/import_slrnspool: support v2 repos
45 scripts/import_slrnspool: cleanup progress messages
46 public-inbox 1.1.0-pre1
48 Eric Wong (Contractor, The Linux Foundation) (239):
49 AUTHORS: add The Linux Foundation
50 watch_maildir: allow '-' in mail filename
51 scripts/import_vger_from_mbox: relax From_ line match slightly
52 import: stop writing legacy ssoma.index by default
53 import: begin supporting this without ssoma.lock
54 import: initial handling for v2
55 t/import: test for last_object_id insertion
56 content_id: add test case
57 searchmsg: add mid_mime import for _extract_mid
58 scripts/import_vger_from_mbox: support --dry-run option
59 import: APIs to support v2 use
60 search: free up 'Q' prefix for a real unique identifier
61 searchidx: fix comment around next_thread_id
62 address: extract more characters from email addresses
63 import: pass "raw" dates to git-fast-import(1)
64 scripts/import_vger_from_mbox: use v2 layout for import
65 import: quiet down warnings from bogus From: lines
66 import: allow the epoch (0s) as a valid time
67 extmsg: fix broken Xapian MID lookup
68 search: stop assuming Message-ID is unique
69 www: stop assuming mainrepo == git_dir
70 v2writable: initial cut for repo-rotation
71 git: reload alternates file on missing blob
72 v2: support Xapian + SQLite indexing
73 import_vger_from_inbox: allow "-V" option
74 import_vger_from_mbox: use PublicInbox::MIME and avoid clobbering
75 v2: parallelize Xapian indexing
76 v2writable: round-robin to partitions based on article number
77 searchidxpart: increase pipe size for partitions
78 v2writable: warn on duplicate Message-IDs
79 searchidx: do not modify Xapian DB while iterating
80 v2/ui: some hacky things to get the PSGI UI to show up
81 v2/ui: retry DB reopens in a few more places
82 v2writable: cleanup unused pipes in partitions
83 searchidxpart: binmode
84 use PublicInbox::MIME consistently
85 searchidxpart: chomp line before splitting
86 searchidx*: name child subprocesses
87 searchidx: get rid of pointless index_blob wrapper
88 view: remove X-PI-TS reference
89 searchidxthread: load doc data for references
90 searchidxpart: force integers into add_message
91 search: reopen skeleton DB as well
92 searchidx: index values in the threader
93 search: use different Enquire object for skeleton queries
94 rename SearchIdxThread to SearchIdxSkeleton
95 v2writable: commit to skeleton via remote partitions
96 searchidxskeleton: extra error checking
97 searchidx: do not modify Xapian DB while iterating
98 search: query_xover uses skeleton DB iff available
99 v2/ui: get nntpd and init tests running on v2
100 v2writable: delete ::Import obj when ->done
101 search: remove informational "warning" message
102 searchidx: add PID to error message when die-ing
103 content_id: special treatment for Message-Id headers
104 evcleanup: disable outside of daemon
105 v2writable: deduplicate detection on add
106 evcleanup: do not create event loop if nothing was registered
107 mid: add `mids' and `references' methods for extraction
108 content_id: use `mids' and `references' for MID extraction
109 searchidx: use new `references' method for parsing References
110 content_id: no need to be human-friendly
111 v2writable: inject new Message-IDs on true duplicates
112 search: revert to using 'Q' as a uniQue id per-Xapian conventions
113 searchidx: support indexing multiple MIDs
114 mid: be strict with References, but loose on Message-Id
115 searchidx: avoid excessive XNQ indexing with diffs
116 searchidxskeleton: add a note about locking
117 v2writable: generated Message-ID goes first
118 searchidx: use add_boolean_term for internal terms
119 searchidx: add NNTP article number as a searchable term
120 mid: truncate excessively long MIDs early
121 nntp: use NNTP article numbers for lookups
122 nntp: fix NEWNEWS command
123 searchidx: store the primary MID in doc data for NNTP
124 import: consolidate object info for v2 imports
125 v2: avoid redundant/repeated configs for git partition repos
126 INSTALL: document more optional dependencies
127 search: favor skeleton DB for lookup_mail
128 search: each_smsg_by_mid uses skeleton if available
129 v2writable: remove unnecessary skeleton commit
130 favor Received: date over Date: header globally
131 import: fall back to Sender for extracting name and email
132 scripts/import_vger_from_mbox: perform mboxrd or mboxo escaping
133 v2writable: detect and use previous partition count
134 extmsg: rework partial MID matching to favor current inbox
135 extmsg: rework partial MID matching to favor current inbox
136 content_id: use Sender header if From is not available
137 v2writable: support "barrier" operation to avoid reforking
138 use string ref for Email::Simple->new
139 v2writable: remove unnecessary idx_init call
140 searchidx: do not delete documents while iterating
141 search: allow ->reopen to be chainable
142 v2writable: implement remove correctly
143 skeleton: barrier init requires a lock
144 import: (v2) delete writes the blob into history in subdir
145 import: (v2): write deletes to a separate '_' subdirectory
146 import: implement barrier operation for v1 repos
147 mid: mid_mime uses v2-compatible mids function
148 watchmaildir: use content_digest to generate Message-Id
149 import: force Message-ID generation for v1 here
150 import: switch to URL-safe Base64 for Message-IDs
151 v2writable: test for idempotent removals
152 import: enable locking under v2
153 index: s/GIT_DIR/REPO_DIR/
154 Lock: new base class for writable lockers
155 t/watch_maildir: note the reason for FIFO creation
156 v2writable: ensure ->done is idempotent
157 watchmaildir: support v2 repositories
158 searchidxpart: s/barrier/remote_barrier/
159 v2writable: allow disabling parallelization
160 scripts/import_vger_from_mbox: filter out same headers as MDA
161 v2writable: add DEBUG_DIFF env support
162 v2writable: remove "resent" message for duplicate Message-IDs
163 content_id: do not take Message-Id into account
164 introduce InboxWritable class
165 import: discard all the same headers as MDA
166 InboxWritable: add mbox/maildir parsing + import logic
167 use both Date: and Received: times
168 msgmap: add tmp_clone to create an anonymous copy
170 v2writable: support reindexing Xapian
171 t/altid.t: extra tests for mid_set
172 v2writable: add NNTP article number regeneration support
173 v2writable: clarify header cleanups
174 v2writable: DEBUG_DIFF respects $TMPDIR
175 feed: $INBOX/new.atom endpoint supports v2 inboxes
176 import: consolidate mid prepend logic, here
177 www: $MESSAGE_ID/raw endpoint supports "duplicates"
178 search: reopen DB if each_smsg_by_mid fails
179 t/psgi_v2: minimal test for Atom feed and t.mbox.gz
180 feed: fix new.html for v2
181 view: permalink (per-message) view shows multiple messages
182 searchidx: warn about vivifying multiple ghosts
183 v2writable: warn on unseen deleted files
184 www: get rid of unnecessary 'inbox' name reference
185 searchview: remove unnecessary imports from MID module
186 view: depend on SearchMsg for Message-ID
187 http: fix modification of read-only value
188 githttpbackend: avoid infinite loop on generic PSGI servers
189 www: support cloning individual v2 git partitions
190 http: fix modification of read-only value
191 githttpbackend: avoid infinite loop on generic PSGI servers
192 www: remove unnecessary ghost checks
193 v2writable: append, instead of prepending generated Message-ID
194 lookup by Message-ID favors the "primary" one
195 www: fix attachment downloads for conflicted Message-IDs
196 searchmsg: document why we store To: and Cc: for NNTP
197 public-inbox-convert: tool for converting old to new inboxes
198 v2writable: support purging messages from git entirely
199 search: cleanup uniqueness checking
200 search: get rid of most lookup_* subroutines
201 search: move find_doc_ids to searchidx
202 v2writable: cleanup: get rid of unused fields
203 mbox: avoid extracting Message-ID for linkification
204 www: cleanup expensive fallback for legacy URLs
205 view: get rid of some unnecessary imports
206 search: retry_reopen on first_smsg_by_mid
207 import: run_die supports redirects as spawn does
208 v2writable: initializing an existing inbox is idempotent
209 public-inbox-compact: new tool for driving xapian-compact
210 mda: support v2 inboxes
211 search: warn on reopens and die on total failure
212 v2writable: allow gaps in git partitions
213 v2writable: convert some fatal reindex errors to warnings
214 wwwstream: flesh out clone instructions for v2
215 v2writable: go backwards through alternate Message-IDs
216 view: speed up homepage loading time with date clamp
217 view: drop load_results
218 feed: optimize query for feeds, too
219 msgtime: parse 3-digit years properly
220 convert: avoid redundant "done\n" statement for fast-import
221 search: move permissions handling to InboxWritable
222 t/v2writable: use simplify permissions reading
223 v2: respect core.sharedRepository in git configs
224 searchidx: correct warning for over-vivification
226 v2writable: fix parallel termination
227 truncate Message-IDs and References consistently
228 scripts/import_vger_from_mbox: set address properly
229 search: reduce columns stored in Xapian
230 replace Xapian skeleton with SQLite overview DB
231 v2writable: simplify barrier vs checkpoints
232 t/over: test empty Subject: line matching
233 www: rework query responses to avoid COUNT in SQLite
234 over: speedup get_thread by avoiding JOIN
235 nntp: fix NEWNEWS command
236 t/thread-all.t: modernize test to support modern inboxes
237 rename+rewrite test using Benchmark module
238 nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster
239 view: avoid offset during pagination
240 mbox: remove remaining OFFSET usage in SQLite
241 msgmap: replace id_batch with ids_after
242 nntp: simplify the long_response API
243 searchidx: ensure duplicated Message-IDs can be linked together
244 init: s/GIT_DIR/REPO_DIR/ in usage
245 import: rewrite less history during purge
246 v2: support incremental indexing + purge
247 v2writable: do not modify DBs while iterating for ->remove
248 v2writable: recount partitions after acquiring lock
249 searchmsg: remove unused `tid' and `path' methods
250 search: remove unnecessary OP_AND of query
251 mbox: do not sort search results
252 searchview: minor cleanup
253 support altid mechanism for v2
254 compact: better handling of over.sqlite3* files
255 v2writable: remove redundant remove from Over DB
256 v2writable: allow tracking parallel versions
257 v2writable: refer to git each repository as "epoch"
258 over: use only supported and safe SQLite APIs
259 search: index and allow searching by date-time
260 altid: fix miscopied field name
261 nntp: set Xref across multiple inboxes
262 www: favor reading more from SQLite, and less from Xapian
263 ensure Xapian and SQLite are still optional for v1 tests
264 psgi: ensure /$INBOX/$MESSAGE_ID/T/ endpoint is chronological
265 over: avoid excessive SELECT
266 over: remove forked subprocess
267 v2writable: reduce barriers
268 index: allow specifying --jobs=0 to disable multiprocess
269 convert: support converting with altid defined
270 store less data in the Xapian document
271 msgmap: speed up minmax with separate queries
272 feed: respect feedmax, again
273 v1: remove articles from overview DB
274 compact: do not merge v2 repos by default
275 v2writable: reduce partititions by one
276 search: preserve References in Xapian smsg for x=t view
277 v2: generate better Message-IDs for duplicates
278 v2: improve deduplication checks
279 import: cat_blob drops leading 'From ' lines like Inbox
280 searchidx: regenerate and avoid article number gaps on full index
281 extmsg: remove expensive git path checks
282 use %H consistently to disable abbreviations
283 searchidx: increase term positions for all text terms
284 searchidx: revert default BATCH_BYTES to 1_000_000
285 Merge remote-tracking branch 'origin/master' into v2
286 fix tests to run without Xapian installed
287 extmsg: use Xapian only for partial matches
290 Don't use LIMIT in UPDATE statements
291 Update the installation instructions with Fedora package names
292 Allow specification of the number of search results to return
294 git clone https://public-inbox.org/ public-inbox
295 (working on a homepage... sorta :)