feat: adds profile and profiles query api endpoint #23
Conversation
zachferland
force-pushed
the
feat/add-profile-query-endpoint
branch
2 times, most recently
from
6e910b0
to
e649ee7
Compare
| @@ -80,6 +90,8 @@ async function pinningNode () { | |||
| openDB(odbAddress) | |||
| }) | |||
| } | |||
|
|
|||
| cache.invalidate(address) | |||
There was a problem hiding this comment.
Seems like this is in the wrong place. I think it should be called when we replicate an entry in any db.
There was a problem hiding this comment.
yeah could be moved, to be more exact, since we know when changes happen. Was just based on quick initial assumption that something MAY have change.
There was a problem hiding this comment.
actually not going to change until next sprint tests/cleanup. To be more exact, helpful if differentiate between priv vs pub open. And need to fix #27 so that its not called over and over. Easy change after, and still valid at the moment, just a bit less effective.
zachferland
force-pushed
the
feat/add-profile-query-endpoint
branch
from
e649ee7
to
4a3c294
Compare
|
Only a few practical things/configs remain:
|
|
Will it be possible to have multiple pinning-nodes that connect to the same reds instance on AWS when we split up the pinning network? Seems like that would be the best approach, then we can just have one serverless function for querying that db. |
|
easy to connect multiple nodes to redis instance, maybe the caching strategy would change a bit, or be more aggressive if wanted to limit the number of cache misses. We could have a server less function as the api query endpoint, but the pinning nodes would have to be able to serve the same requests, since they would be the fallback for a cache miss |
|
Oh, I see yeah good point. I guess the benefit of using a serverless function would only be there when we start having multiple pinning-nodes then. (the serverless function would check the redis cache first, then it would know which pinning-node to make a request to) |
|
yeah exactly I don't know exactly what you are thinking for splitting up the pinning nodes, but I'm starting to think the best approach would be to use a s3 datastore for ipfs, and the same for the orbitdb cache. Then all ec2 instances can share a single s3 instance, which scales easily, and removes a lot of dependencies around state (and complexity, backups, etc). Now with that node specific state being removed, connections don't have to be made to a particular node, they just have to be made to any node with pubsub/ipfs already running. They can still be distributed evenly by name/hash. Also without that state dependency it easier to scale these nodes up and down on demand. Another plus of this, we could have a server less function for the api queries, where it ask redis, if no cache hit, it can then get the data directly from s3. We could have a function that directly maps the raw ipfs and orbit db data to the response (rather than through the orbitdb/ipfs interface) |
|
@zachferland Let's schedule a call sometime next year to discuss this! |
zachferland
changed the title
|
@oed sounds good, Ill add a call sometime the first week Also this all set, looks pretty similar to when you first reviewed though |
|
Ty, I'll have another look now! |
zachferland
force-pushed
the
feat/add-profile-query-endpoint
branch
2 times, most recently
from
3ed6319
to
9f12edc
Compare
server.js
Outdated
|
|
||
| const profiles = await Promise.all(profilePromiseArray) | ||
| const parsed = profiles.reduce((acc, val) => { | ||
| acc[val['rootStoreAddress']] = val['profile'] |
There was a problem hiding this comment.
hm, shouldn't the key be the eth-address rather than the rootStoreAddress?
There was a problem hiding this comment.
Or how do you map eth-addr -> profile clientside?
There was a problem hiding this comment.
yup that should be eth address, let me make sure the names are clear as well, that was just recently change to use eth-addr instead of rootstoreaddress. The return obj is {ethaddress: addressprofileObj, ehtaddress2: ....., ....}
There was a problem hiding this comment.
Is it really? looks like the profiles object is [{rootStoreAddress, profile}, ...], so the return obj would become {rootStoreAddress: profileObj, ...}
There was a problem hiding this comment.
meant to say request was changed from using root addresses to using eth address, but yes the response is currently incorrect as you described
| await readyPromise | ||
|
|
||
| await Promise.resolve(resolve => { | ||
| rootStore.events.on('replicated', resolve) |
There was a problem hiding this comment.
Why do we wait for this event? replicated is only emitted if you replicate from the network which we won't do here.
There was a problem hiding this comment.
Ok this I'm also wondering, pretty sure I originally removed it, things didn't work and then added it back in. And then saw no time lost in opening and fetching the profile. So not sure exactly what it is doing
There was a problem hiding this comment.
But I will take a closer look to pinpoint or add details for a future issue
| publicStore.load() | ||
| await readyPromisePublic | ||
| await Promise.resolve(resolve => { | ||
| publicStore.events.on('replicated', resolve) |
zachferland
force-pushed
the
feat/add-profile-query-endpoint
branch
from
9f12edc
to
3227ea8
Compare
zachferland
force-pushed
the
feat/add-profile-query-endpoint
branch
from
3227ea8
to
8f3947c
Compare
No description provided.