API: selective peer replication

I'd really like APIs that offer more control around feeds:

• Drop (delete) a remote or local feed
• Have a userland callback at replication time that lets the user decide if they want to replicate a certain feed that a remote peer is offering

cc @0fork, who I was talking to about APIs to better support userland moderation and authorization models

[mjp0]

What would this require from the implementation perspective?

I would guess you could do a simple block list to drop feeds so that those are skipped in replication and user gets to choose whether they are permanently deleted or simply hidden. I can imagine a scenario where you don't want to actively replicate a feed anymore but it still has contents you would maybe like to access later.

Dropping a feed: right now feed loading is done by looking for the key file in each feed's storage. If you deleted that entry and also removed the feed from this._feeds that should do the trick.

Controlling feeds to replicate: the first thing that happens at replication time is both sides exchange a list of all hypercore keys they have & want to share. Replication could pause here, and there could be an API that gets called out to with the list of feeds, and returns the keys it wants to receive. There could be a similar API for choosing what feeds you'd like to share.

Dropping messages within a feed: we get this for free! hypercore already has a clear API. I'm not sure if it will just automatically re-download it though the next time it connects to a peer with it. (cc @mafintosh on this)

[mjp0]

So to summarize, great news is that there’s nothing here that isn’t in the realm of just sitting down and doing the logic code and convenience functions around the logic.

Regarding dropping messages, isn’t redownloading avoidable by using sparse replication?

That all sounds about right. API design might be the hardest part!

[mjp0]

Happy to help with that. Let me chew on this over the weekend and let’s discuss this further on Monday, and post the conclusion here after.

[cblgh]

@0fork also check out this list of actions that we'd want to be able to do in cabal (a chat client making use of multifeed)

cabal-club/cabal-cli#41

[saranrapjs]

I've done some experiments with this locally, and it seems like with sparse: true, replication won't automatically download anything; e.g. if you clear a range, but want to replicate everything outside of that range, you'd be on the hook for manually downloading everything but that range by some means other than replicate. I do like the simplicity of downstream multifeed consumers being able to simply replicate({ live: true})...

Part of me wishes there was actually a hook in hypercore's replication code that allowed for ignoring offered hypercore blocks, based on some user-land logic. I for one have been looking into a way of specifying something like a maxFeedLength for the underlying multifeed hypercores, which could leverage something like this if it existed! You could also ignore specific messages this way, though maybe it'd be cumbersome

Part of me wishes there was actually a hook in hypercore's replication code that allowed for ignoring offered hypercore blocks, based on some user-land logic.

I bet @mafintosh would be open to that!

[mafintosh]

Yea def sounds useful

…

On Fri, Sep 28, 2018 at 6:51 PM Stephen Whitmore ***@***.***> wrote: Part of me wishes there was actually a hook in hypercore's replication code that allowed for ignoring offered hypercore blocks, based on some user-land logic. I bet @mafintosh <https://github.com/mafintosh> would be open to that! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAW_VROwABzW0wi5kEx_sXpMjynHHBrlks5uflOPgaJpZM4WEP20> .

[telamon]

Controlling feeds to replicate: the first thing that happens at replication time is both sides exchange a list of all hypercore keys they have & want to share. Replication could pause here, and there could be an API that gets called out to with the list of feeds, and returns the keys it wants to receive. There could be a similar API for choosing what feeds you'd like to share.

I have a working implementation for that:
https://github.com/telamon/multifeed/tree/feature/repl-api#replication-policies
It's implemented on top of the changes introduced in #18
I'd like to find substitute for the word policyas I am afraid it might suggest the notion of remote management, which does not quite reflect the function nor the intention.

I'm currently looking into the possibility of dropping a feed, I believe that the only complexity involved is playing nice with the current multifeed storage scheme and make sure that deleted feeds won't break multifeed._loadFeeds().

I think @substack is working on this to implement peer blocks in Cabal.

[Juul]

Yeah I'm having to use multiplex and have multiple instances of multifeed to deal with different levels of access for different peers so something like this would be really great.