]> Sergey Matveev's repositories - btrtrc.git/log
btrtrc.git
2 years agoFix leaked mmap storage in test v1.39.2
Matt Joiner [Thu, 16 Dec 2021 00:41:31 +0000 (11:41 +1100)]
Fix leaked mmap storage in test

2 years agoFix race in TextPexConnState
Matt Joiner [Thu, 16 Dec 2021 00:07:19 +0000 (11:07 +1100)]
Fix race in TextPexConnState

2 years agoFix go-libutp import when CGO is disabled
Matt Joiner [Wed, 15 Dec 2021 22:44:59 +0000 (09:44 +1100)]
Fix go-libutp import when CGO is disabled

2 years agoTest benchmarks in a separate step and limit executions
Matt Joiner [Thu, 16 Dec 2021 03:19:19 +0000 (14:19 +1100)]
Test benchmarks in a separate step and limit executions

2 years agoFix BenchmarkUpdatePiecePriorities
Matt Joiner [Wed, 15 Dec 2021 08:09:22 +0000 (19:09 +1100)]
Fix BenchmarkUpdatePiecePriorities

2 years agoFix BenchmarkConnectionMainReadLoop
Matt Joiner [Wed, 15 Dec 2021 07:54:47 +0000 (18:54 +1100)]
Fix BenchmarkConnectionMainReadLoop

Broken by piece request order changes.

2 years agoSome optimizations in PieceRequestOrder.Update and item comparisons
Matt Joiner [Wed, 15 Dec 2021 07:07:17 +0000 (18:07 +1100)]
Some optimizations in PieceRequestOrder.Update and item comparisons

2 years agoRetract v1.39.0
Matt Joiner [Mon, 13 Dec 2021 01:16:43 +0000 (12:16 +1100)]
Retract v1.39.0

2 years agoTidy up request heaping comments v1.39.1
Matt Joiner [Mon, 13 Dec 2021 01:12:30 +0000 (12:12 +1100)]
Tidy up request heaping comments

2 years agoAdd Peer.cancelAllRequests for webseedPeer
Matt Joiner [Mon, 13 Dec 2021 01:11:38 +0000 (12:11 +1100)]
Add Peer.cancelAllRequests for webseedPeer

2 years agoUpdate requests after deleting all in some corner cases
Matt Joiner [Mon, 13 Dec 2021 01:09:12 +0000 (12:09 +1100)]
Update requests after deleting all in some corner cases

Choked by non-fast PeerConn, deleted PeerConn. They're not exactly guarded as strictly as they could be, so there's plenty of room for performance improvements here.

2 years agoAssert that a request isn't cancelled before requesting it again
Matt Joiner [Mon, 13 Dec 2021 00:56:57 +0000 (11:56 +1100)]
Assert that a request isn't cancelled before requesting it again

2 years agoUpdate peer requests if not interested when piece priorities change
Matt Joiner [Mon, 13 Dec 2021 00:55:47 +0000 (11:55 +1100)]
Update peer requests if not interested when piece priorities change

This should fix a starvation issue where we don't signal interest to a peer that has us choked.

2 years agoExplicitly use go-libutp synchronous firewall callback
Matt Joiner [Sun, 12 Dec 2021 10:32:29 +0000 (21:32 +1100)]
Explicitly use go-libutp synchronous firewall callback

2 years agoMerge branch 'request-strategy-experiments' v1.39.0
Matt Joiner [Sun, 12 Dec 2021 07:38:33 +0000 (18:38 +1100)]
Merge branch 'request-strategy-experiments'

This should bring in significant performance improvements that fix issues with the peer-requesting that existed from v1.34.0.

2 years agoMore consistent cancellation management
Matt Joiner [Sat, 11 Dec 2021 13:04:06 +0000 (00:04 +1100)]
More consistent cancellation management

2 years agoRemove unused request strategy code
Matt Joiner [Sat, 11 Dec 2021 03:14:24 +0000 (14:14 +1100)]
Remove unused request strategy code

2 years agoDon't group Torrent piece request orders without a storage capacity together
Matt Joiner [Sat, 11 Dec 2021 02:46:29 +0000 (13:46 +1100)]
Don't group Torrent piece request orders without a storage capacity together

2 years agoMinimize allocations due to heap escapes
Matt Joiner [Thu, 9 Dec 2021 10:49:29 +0000 (21:49 +1100)]
Minimize allocations due to heap escapes

2 years agoOnly steal an odd request if the stealer more recently received a chunk
Matt Joiner [Tue, 7 Dec 2021 03:19:44 +0000 (14:19 +1100)]
Only steal an odd request if the stealer more recently received a chunk

This helps break the stealing cycle during endgame, and lets us trickle the request to the peer conn with the best record. It might not be sufficient but works nice in testing so far.

2 years agoOnly use last requested when both requests are already pending
Matt Joiner [Tue, 7 Dec 2021 03:17:43 +0000 (14:17 +1100)]
Only use last requested when both requests are already pending

2 years agoAdd Torrent-level request cancel for consistency
Matt Joiner [Fri, 10 Dec 2021 07:04:45 +0000 (18:04 +1100)]
Add Torrent-level request cancel for consistency

2 years agoUpdate peer cancel assumptions and return
Matt Joiner [Mon, 6 Dec 2021 08:02:29 +0000 (19:02 +1100)]
Update peer cancel assumptions and return

2 years agoImprove panic message
Matt Joiner [Fri, 3 Dec 2021 10:31:40 +0000 (21:31 +1100)]
Improve panic message

2 years agoTry request stealing
Matt Joiner [Fri, 3 Dec 2021 10:30:41 +0000 (21:30 +1100)]
Try request stealing

2 years agoDon't try to request anything without the torrent info
Matt Joiner [Fri, 3 Dec 2021 05:41:52 +0000 (16:41 +1100)]
Don't try to request anything without the torrent info

2 years agoWait between duplicate requests
Matt Joiner [Thu, 2 Dec 2021 02:10:02 +0000 (13:10 +1100)]
Wait between duplicate requests

2 years agoStop iterating pieces when storage is exhausted
Matt Joiner [Wed, 1 Dec 2021 08:31:29 +0000 (19:31 +1100)]
Stop iterating pieces when storage is exhausted

2 years agoUse interfaces to lazily expose the bare minimum inputs to GetRequestablePieces
Matt Joiner [Wed, 1 Dec 2021 08:21:25 +0000 (19:21 +1100)]
Use interfaces to lazily expose the bare minimum inputs to GetRequestablePieces

2 years agoProvide mapping from infohash to Torrent in Input
Matt Joiner [Wed, 1 Dec 2021 04:20:42 +0000 (15:20 +1100)]
Provide mapping from infohash to Torrent in Input

2 years agoImplement piece request ordering with retained state
Matt Joiner [Wed, 1 Dec 2021 03:38:47 +0000 (14:38 +1100)]
Implement piece request ordering with retained state

2 years agoAvoid reflection sorting request pieces
Matt Joiner [Tue, 30 Nov 2021 10:31:32 +0000 (21:31 +1100)]
Avoid reflection sorting request pieces

2 years agoUse indices to lookup pieces for comparison
Matt Joiner [Tue, 30 Nov 2021 04:26:05 +0000 (15:26 +1100)]
Use indices to lookup pieces for comparison

2 years agoCache piece request orderings
Matt Joiner [Tue, 30 Nov 2021 04:18:38 +0000 (15:18 +1100)]
Cache piece request orderings

2 years agoRetract last few minor versions with peer-requesting issues
Matt Joiner [Sun, 12 Dec 2021 07:24:27 +0000 (18:24 +1100)]
Retract last few minor versions with peer-requesting issues

(cherry picked from commit 85a436a5ca75a47a52dcab9648d85f3c3fb53e35)

2 years agoRevert "Link to godocs.io for documentation"
Matt Joiner [Sun, 12 Dec 2021 06:20:02 +0000 (17:20 +1100)]
Revert "Link to godocs.io for documentation"

This reverts commit c2ed60f9f2d100ddd15a38a6c260121a9ad6e893.

2 years agoPartition piece request strategy by storage capacity key
Matt Joiner [Mon, 29 Nov 2021 02:07:18 +0000 (13:07 +1100)]
Partition piece request strategy by storage capacity key

2 years agoDo webseed request updates asynchronously
Matt Joiner [Mon, 6 Dec 2021 08:24:04 +0000 (19:24 +1100)]
Do webseed request updates asynchronously

Fixes a bug where received chunks are immediately requested while still being processed.

2 years agoAdd more identifying information to PeerConn.String
Matt Joiner [Mon, 6 Dec 2021 08:02:55 +0000 (19:02 +1100)]
Add more identifying information to PeerConn.String

Some logs for unexpected protocol violations could be helped by them.

2 years agoAdd generalized decodeJsonByteString and a fuzz target for it
Matt Joiner [Sat, 11 Dec 2021 02:43:25 +0000 (13:43 +1100)]
Add generalized decodeJsonByteString and a fuzz target for it

2 years agoAdd failing bencode fuzz input
Matt Joiner [Fri, 3 Dec 2021 06:15:59 +0000 (17:15 +1100)]
Add failing bencode fuzz input

2 years agoTransmission does not reject cancelled requests
Matt Joiner [Tue, 7 Dec 2021 03:16:02 +0000 (14:16 +1100)]
Transmission does not reject cancelled requests

2 years agoDon't log webseed request context cancellation results
Matt Joiner [Mon, 6 Dec 2021 08:24:48 +0000 (19:24 +1100)]
Don't log webseed request context cancellation results

2 years agobencode: Avoid allocating interface string up front
Matt Joiner [Fri, 10 Dec 2021 04:10:26 +0000 (15:10 +1100)]
bencode: Avoid allocating interface string up front

This should fix crashes during fuzzing. It's not a complete fix, we really want to limit the amount a given Decode can allocate. Maybe Go isn't the right language for this.

2 years agobencode: Fix decoding integer with leading +
Matt Joiner [Fri, 10 Dec 2021 03:29:35 +0000 (14:29 +1100)]
bencode: Fix decoding integer with leading +

2 years agobencode: Enforce dict key ordering
Matt Joiner [Fri, 10 Dec 2021 03:05:00 +0000 (14:05 +1100)]
bencode: Enforce dict key ordering

Fix bencode dict key ordering in HTTP tracker test

2 years agobencode incorrectly parsed integers with leading zeroes
Matt Joiner [Fri, 10 Dec 2021 02:36:53 +0000 (13:36 +1100)]
bencode incorrectly parsed integers with leading zeroes

2 years agoAdd bencode FuzzInterfaceRoundTrip
Matt Joiner [Fri, 10 Dec 2021 02:35:57 +0000 (13:35 +1100)]
Add bencode FuzzInterfaceRoundTrip

2 years agoShuffle duplicate requests
Matt Joiner [Sun, 12 Dec 2021 05:56:00 +0000 (16:56 +1100)]
Shuffle duplicate requests

Add missing import

2 years agoGenerate full sintel magnet link in fs test
Matt Joiner [Wed, 1 Dec 2021 23:50:02 +0000 (10:50 +1100)]
Generate full sintel magnet link in fs test

2 years agoAdd sintel.torrent
Matt Joiner [Wed, 1 Dec 2021 23:49:30 +0000 (10:49 +1100)]
Add sintel.torrent

2 years agoFix races using resources on Close
Matt Joiner [Wed, 1 Dec 2021 07:24:17 +0000 (18:24 +1100)]
Fix races using resources on Close

2 years agoDon't use the Client's IP blocklist after the Client is closed
Matt Joiner [Wed, 1 Dec 2021 04:02:36 +0000 (15:02 +1100)]
Don't use the Client's IP blocklist after the Client is closed

Fixes a segfault due to using a memory mapped blocklist in confluence.

2 years agoRemove unused torrent_pending_pieces.go
Matt Joiner [Wed, 1 Dec 2021 00:27:16 +0000 (11:27 +1100)]
Remove unused torrent_pending_pieces.go

Unfortunately github.com/elliotchance/orderedmap is still used in ./analysis, so no deps are removed.

2 years agoFilter update requests on piece priority change by peer choking and allowed fast
Matt Joiner [Tue, 30 Nov 2021 04:19:59 +0000 (15:19 +1100)]
Filter update requests on piece priority change by peer choking and allowed fast

2 years agoEnsure peerRequests.torrentStrategyInput is set
Matt Joiner [Mon, 29 Nov 2021 02:46:44 +0000 (13:46 +1100)]
Ensure peerRequests.torrentStrategyInput is set

2 years agoPEX: remove workaround for krpc marshaller
Yaroslav Kolomiiets [Wed, 8 Dec 2021 13:02:36 +0000 (13:02 +0000)]
PEX: remove workaround for krpc marshaller

2 years agoPEX: use new NodeAddr search methods in krpc
Yaroslav Kolomiiets [Wed, 8 Dec 2021 10:16:54 +0000 (10:16 +0000)]
PEX: use new NodeAddr search methods in krpc

2 years agoPEX: fluid event log
Yaroslav Kolomiiets [Mon, 6 Dec 2021 18:46:25 +0000 (18:46 +0000)]
PEX: fluid event log

2 years agoFix wasm build
Matt Joiner [Tue, 7 Dec 2021 02:07:58 +0000 (13:07 +1100)]
Fix wasm build

This was broken in the move to zombiezen sqlite, which still doesn't work on WASM/JS.

2 years agoDon't fail-fast on go versions
Matt Joiner [Tue, 7 Dec 2021 01:42:35 +0000 (12:42 +1100)]
Don't fail-fast on go versions

2 years agoHandle 503 returns from webseed peer endpoints
Matt Joiner [Mon, 6 Dec 2021 04:14:59 +0000 (15:14 +1100)]
Handle 503 returns from webseed peer endpoints

2 years agoRecord an observation about why webseed peers are doing most of the work in recent...
Matt Joiner [Thu, 2 Dec 2021 02:48:52 +0000 (13:48 +1100)]
Record an observation about why webseed peers are doing most of the work in recent testing

2 years agoDo webseed request parts sequentially
Matt Joiner [Thu, 2 Dec 2021 02:47:06 +0000 (13:47 +1100)]
Do webseed request parts sequentially

This means we can treat the number of bytes in the result with enough accuracy to decide if we should count it as a wasted chunk. Also I'm not sure why it was a good idea to do parts of a request in parallel anyway, it could just lead to spikes in outstanding requests to the webseed, rather than sticking to the predictable maxRequests limit.

2 years agoHandle cancelled webseed peer requests if they're not active
Matt Joiner [Thu, 2 Dec 2021 02:10:54 +0000 (13:10 +1100)]
Handle cancelled webseed peer requests if they're not active

2 years agoCheck if Torrent is closed before receiving webseed chunks
Matt Joiner [Fri, 3 Dec 2021 10:31:16 +0000 (21:31 +1100)]
Check if Torrent is closed before receiving webseed chunks

2 years agoFix minimum peer extensions error message
Matt Joiner [Fri, 3 Dec 2021 09:36:34 +0000 (20:36 +1100)]
Fix minimum peer extensions error message

2 years agoReduce idle request updates due to choke and unchoke messages
Matt Joiner [Mon, 29 Nov 2021 01:12:30 +0000 (12:12 +1100)]
Reduce idle request updates due to choke and unchoke messages

2 years agoMerge branch 'udp-tracker-no-dial' into te
Matt Joiner [Mon, 29 Nov 2021 00:19:54 +0000 (11:19 +1100)]
Merge branch 'udp-tracker-no-dial' into te

2 years agoRemove unused peer stuff in request strategy
Matt Joiner [Sun, 28 Nov 2021 23:14:16 +0000 (10:14 +1100)]
Remove unused peer stuff in request strategy

2 years agoOptimize the use of worstConnSlice again
Matt Joiner [Sun, 28 Nov 2021 23:07:48 +0000 (10:07 +1100)]
Optimize the use of worstConnSlice again

2 years agoOptimize Torrent.haveAnyPieces
Matt Joiner [Sun, 28 Nov 2021 13:44:24 +0000 (00:44 +1100)]
Optimize Torrent.haveAnyPieces

2 years agoAnnounce to DHT if we want conns, not just peers
Matt Joiner [Sun, 28 Nov 2021 11:46:45 +0000 (22:46 +1100)]
Announce to DHT if we want conns, not just peers

2 years agoWanting peers is a subset of wanting conns
Matt Joiner [Sun, 28 Nov 2021 11:37:56 +0000 (22:37 +1100)]
Wanting peers is a subset of wanting conns

2 years agoDon't want conns for seeding unless we have a piece
Matt Joiner [Sun, 28 Nov 2021 11:40:53 +0000 (22:40 +1100)]
Don't want conns for seeding unless we have a piece

2 years agoPeers with all pieces have wanted pieces only if we want any piece
Matt Joiner [Sun, 28 Nov 2021 11:36:19 +0000 (22:36 +1100)]
Peers with all pieces have wanted pieces only if we want any piece

2 years agoAbstract worseConn comparisons and add tests
Matt Joiner [Sun, 28 Nov 2021 06:06:10 +0000 (17:06 +1100)]
Abstract worseConn comparisons and add tests

2 years agoRename worst_conns.go
Matt Joiner [Sun, 28 Nov 2021 01:53:18 +0000 (12:53 +1100)]
Rename worst_conns.go

2 years agoMerge branch 'bsi-pending-requests' into te
Matt Joiner [Fri, 26 Nov 2021 22:38:58 +0000 (09:38 +1100)]
Merge branch 'bsi-pending-requests' into te

2 years agoUse modernc.org/sqlite version that fixes temp dir race
Matt Joiner [Fri, 26 Nov 2021 06:06:13 +0000 (17:06 +1100)]
Use modernc.org/sqlite version that fixes temp dir race

2 years agoComment out pending requests tests and asserts
Matt Joiner [Thu, 25 Nov 2021 11:59:20 +0000 (22:59 +1100)]
Comment out pending requests tests and asserts

These need to be updated. BSI for pending requests removes an enormous amount of memory use for very large torrents.

2 years agoRevert "Use a flat slice for pending request counts"
Matt Joiner [Thu, 25 Nov 2021 11:32:52 +0000 (22:32 +1100)]
Revert "Use a flat slice for pending request counts"

This reverts commit dfc421824ce6c146ff9d9062ad25748c0f8c9990.

2 years agotracker/udp: Panic on read errors if ConnClient not closed
Matt Joiner [Mon, 22 Nov 2021 21:35:45 +0000 (08:35 +1100)]
tracker/udp: Panic on read errors if ConnClient not closed

There's nothing exposed to handle this error currently, and if it occurs, it's better not to fail in silence. I'm currently debugging a situation where ConnClients stop working, this would be a condition we'd want to know about.

2 years agoImprove logging of UDP tracker packet dispatch error
Matt Joiner [Mon, 22 Nov 2021 21:33:29 +0000 (08:33 +1100)]
Improve logging of UDP tracker packet dispatch error

2 years agoWrap UDP tracker response error
Matt Joiner [Mon, 22 Nov 2021 21:32:25 +0000 (08:32 +1100)]
Wrap UDP tracker response error

2 years agoDon't dial in UDP tracking
Matt Joiner [Mon, 22 Nov 2021 07:05:50 +0000 (18:05 +1100)]
Don't dial in UDP tracking

This could fix an issue where tracker addresses change, but we're already bound to a particular address and so fail to receive any more responses.
It should also make it easier to share UDP sockets between UDP tracker clients, although that's not currently implemented.

2 years agoDoc comments
Matt Joiner [Mon, 22 Nov 2021 07:06:27 +0000 (18:06 +1100)]
Doc comments

2 years agoRemove CGO build flags for pure-Go squirrel branch
Matt Joiner [Fri, 19 Nov 2021 09:29:30 +0000 (20:29 +1100)]
Remove CGO build flags for pure-Go squirrel branch

2 years agoUse zombiezen branch of squirrel
Matt Joiner [Fri, 19 Nov 2021 09:10:07 +0000 (20:10 +1100)]
Use zombiezen branch of squirrel

2 years agogofumpt
Matt Joiner [Fri, 19 Nov 2021 05:19:32 +0000 (16:19 +1100)]
gofumpt

2 years agoUse zombiezen sqlite for piece completion
Matt Joiner [Fri, 19 Nov 2021 05:18:41 +0000 (16:18 +1100)]
Use zombiezen sqlite for piece completion

2 years agoQuote peer extension names to expvar
Matt Joiner [Fri, 19 Nov 2021 02:02:42 +0000 (13:02 +1100)]
Quote peer extension names to expvar

2 years agoPull synchronous firewall callback performance improvement
Matt Joiner [Wed, 17 Nov 2021 03:24:38 +0000 (14:24 +1100)]
Pull synchronous firewall callback performance improvement

Reduces CPU overhead when receiving lot of UDP traffic.

2 years agoOptimize PeerConn.peerHasAllPieces
Matt Joiner [Wed, 17 Nov 2021 03:01:11 +0000 (14:01 +1100)]
Optimize PeerConn.peerHasAllPieces

2 years agoAvoid calling net.Conn methods with the Client mutex locked
Matt Joiner [Wed, 17 Nov 2021 03:00:50 +0000 (14:00 +1100)]
Avoid calling net.Conn methods with the Client mutex locked

The utp firewall callback might want to use it. I can't think of a better way to determine with an incoming connection request should be blocked.

2 years agoUse tagged version of github.com/anacrolix/fuse v1.38.0
Matt Joiner [Tue, 16 Nov 2021 07:27:11 +0000 (18:27 +1100)]
Use tagged version of github.com/anacrolix/fuse

2 years agoSwitch to github.com/anacrolix/fuse
Matt Joiner [Tue, 16 Nov 2021 07:20:02 +0000 (18:20 +1100)]
Switch to github.com/anacrolix/fuse

2 years agoRemove dependency on and exclude of github.com/willf/bitset
Matt Joiner [Tue, 16 Nov 2021 06:36:48 +0000 (17:36 +1100)]
Remove dependency on and exclude of github.com/willf/bitset

2 years agoProvide context to readahead func
Matt Joiner [Sun, 14 Nov 2021 02:52:05 +0000 (13:52 +1100)]
Provide context to readahead func

2 years agoAdd Reader.SetReadaheadFunc
Matt Joiner [Wed, 10 Nov 2021 23:24:28 +0000 (10:24 +1100)]
Add Reader.SetReadaheadFunc

Fixes https://github.com/anacrolix/torrent/issues/687.