Public Database for Active Users

[dorkmo]

would it make since for each user to have a hyperdb type database associated with their account, and the keys were all public so that any other client could edit it, then add their public key to the list when they want to become visible. So if the first user is not following the second user, the second user could reply to the first user, then auto add their key to the database so that the first user could see the second user's reply.

[pfrazee]

We'll see what we can do with hyperdb. Something like this would be pretty awesome but I'm not sure yet if it would work (hyperdb currently requires all writers to be added by the owner(s)). So, we're toying with ideas for a "publicly writable" hyperdb, but we're toying with some other similar ideas to do this kind of discovery.

[fsteff]

Not sure how something like this could work. Multiple writers of a hyperdb are still distinct hypercore feeds and thus not visible to others unless added by the "owner" of the database.

None the less, the problem with the visibility of replies I think is not unsolveable.
I recently thought about a search index build by each user (as a the background service/worker). The search index would be composed of all entries of the search indices of the users this one user follows.
Over time this search index should "converge" to span across the whole network of users.

Such a search index, covering all posts, users and hashtags, could be used in various ways. Not only for searching, but also for visibility of replies, hashtags, discovery, ...
But it still does not solve the problem that a new user won´t be visible to others until someone follows him (needs the discovery-id to be sent over some other channel).

An other problem with anything of this sort is spam and/or harmful actors.

[pfrazee]

@fsteff yeah I wrote about that approach here https://pfrazee.hashbase.io/blog/achieving-scale

[RangerMauve]

I was thinking that a dedicated discovery service would be useful for something like this.

I've written up my thoughts on it here, but the gist of it is you send a POST to an HTTP server saying you want to advertise your Dat URL for a certain "application" type.

The server then checks your dat to make sure you have written to it proving that you own the dat.

Then it writes your URL to it's own dat in a folder per application.

Applications then get the server's Dat URL and search for keys within the folder for their application.

That way a person can link to multiple discovery servers to help with more discovery, and even if the server goes down the actual data is still in a dat archive (and can be forked if needed).

[pfrazee]

@RangerMauve I think that's a good idea. Basically a user directory service. Can I make some suggestions on the design?

[RangerMauve]

Whoops, I read that as can I get, my bad. I'm still in the super early sketchy stages here since I've been waiting for multiwriter, so any suggestions are very much appreciated

[pfrazee]

Yeah it's looking good! Just two suggestions:

• Add PSA support! We use this in the DNS DEP and Beaker has some tooling for PSA on the way. I plan to add a Labs API soon that lets apps use this. (I can explain in more detail if you're curious.)
• There's actually a registry of well-knowns. We used well-known/dat with the expectation we'd submit there eventually. If you think there's a chance you won't be able to get your entry added, I'd suggest using a different location.

[RangerMauve]

PSA support was definately a must have for me since I want to make it easy for others to integrate. The PSA stuff is really inspiring, by the way. :D What sort of lab APIs were you planning? A way for a site to register a type of PSA and a way for the browser to let the user plug in their own?

I didn't know that well-knowns are regulated. What do you think about me placing it in /.dat-pubs/ instead?

[pfrazee]

I'm still thinking about exactly how that lab API should work. Maybe something where you can request a service of a given PSA, and a modal shows up prompting the user to select from known services or to enter the URL for one. For registering, I can definitely imagine something to add self and then beaker would prompt the user, like, "Add foo.com to your list of services?"

Yeah /.dat-pubs/ seems fine, or you might consider generalizing it to something like /.proofs/

[RangerMauve]

Maybe when you request services you could provide a default? That way if it's a less-known service applications will still be guaranteed to work.

The API would be used to trigger the browser to look into /.well-known/psa?

[pfrazee]

@RangerMauve yeah.... I wonder if there could be a business model for Beaker there- setting default services for PSAs? Either way if you put together this service (and commit to running it) I'd be happy to set you as the default.

And yeah it triggers a fetch of the psa doc.

[RangerMauve]

Definately planning on running at least one instance once it's all figured out.

I think that one of the benefits of this approach is that clients don't have to talk to the server unless they're reaching it for the first time or if they're adding a new dat.

It also means that if a service becomes popular, it's more likely that there will be other peers out there seeding it and thus lowering the upload costs on the service provider.

I've been thinking that this service would be how people could create "communities" or "forums" using decentralized tech.

Basically, if you're building a community, you host a discovery service and have everyone join it to discover each other. You could add something like invitation codes or rate limiting over the API while keeping applications compatible with it since the discovery is just a matter of searching a folder.

Could also be a business model to set up hosting for this stuff and administrative tasks. Not that I'd want to go that far with it any time soon. :P

In terms of APIs, I think an ideal would look like const serviceURL = await PSA.requestService(DEFINITION_URL, DEFAULT_URL) so that the user could choose one, or have the default one, DEFAULT_URL in this case, returned if anything goes wrong. DEFINITION_URL is the URL defined in the rel of the PSA link. Though I guess that could be abstracted away with error handlers that do the same thing.

[RangerMauve]

One other thing I've kinda been thinking about, however. I think it would be better for offline-readiness if instead of using the service HTTP URL to get the dat archive, you used the service Dat URL and discovered the list of HTTP servers it relates to through that.

That way the service wouldn't be able to give different dat URLs to different visitors, and you would never have to actually talk to the HTTP API unless you were going to add something to it.

Not sure how that would fit into the PSA paradigm.

[pfrazee]

Yeah that requestService() API suggestion makes sense to me

The offline-readiness argument makes sense but yeah, I'm not sure how to fit that into PSA at the moment. I'm inclined to save that for later

[dorkmo]

are any of the attempts to make dht work in the browser compatible with the mechanics of beaker/fritter? https://github.com/xuset/web-dht

[pfrazee]

@dorkmo We definitely plan to make it possible (using the DHT/discovery-net) to create sockets between users on a site. Whether we can setup a DHT to act like a general-purpose keyvalue store is an open question (ie a DHT to publish your dat site). We're also examining other solutions... it's a fairly hard problem to create a decentralized namespace that anybody can write to without getting overrun with spam.

[dorkmo]

seems like some sort of captcha system is needed? could a client set and host their own captcha rules? there maybe be like a fully public captcha dht where you submit a permission request, then if anyone passes they can be added to the clients approved database. captcha submissions that dont pass are just ignored/deleted? maybe a simple 1 + 2 = ? and have it change based on the timestamp.