Synchronization of Social Graphs between Network Nodes - $SA 1.25M bounty!

[Luis-Fernando-Molina]

Context

• The Social Trading App is an App under development within the suite of Apps that Superalgos Provides. It will be used to copy trade other bots providing Trading Signals via Superlgos Network.
• The Social Network App has:

1. a UI
2. a Client Side (runs at the end user's machine
3. a Server Side (runs inside a Superalgos Network Node as a Network Service called Social Trading Network Service)

• One of the missions of the Social Trading Network Service is to maintain a Social Graph.

What is a Social Graph?

It is an in-memory data / object structure which basically defines which bots follows which other bots, which posts corresponds to which bots, who reacted (like, love, hate, etc) to which posts, who replied to which posts. All these relationships are stored in memory at each Network Node running the Social Trading Network Service.

What are Events and Queries at the Social Trading Network Service?

Each time an end user writes a post, follows another user, reacts, etc, in general makes any action that has an impact on the Social Graph, modifying it, we call that an EVENT.

Each time an end user request info of who follows who, which are the posts of a certain user and in general request information that is extracted from the Social Graph but does not modify it, then we call that a QUERY.

You can think of EVENTS like transactions that needs to be applied and modify the state of a database (imagine balances of bank accounts).

What is the Problem to be solved?

The problem comes from the fact that the Superalgos Network:

1. Can have any number of nodes.
2. Nodes are not always running.
3. Any node that comes online needs to synchronize itself with the network before being able to process new EVENTS.

The problem is similar to any blockchain p2p network, where EVENTS are the equivalent to Transactions that modify the state of the the blockchain. In our case we don't neet a chain of blocks as a data structure, but a graph as a data stucture, but beyond that difference, the problem is quite similar.

Online Synchronization

Once a Node is syncronized all EVENTS it receives from users are applied to its own copy of the Social Graph and after that broadcasted to other nodes connected to it. This part is implemented, though not perfectly tested yet.

Events Storage

1. Each node stores every one minute in a file at its local file system, all the EVENTS that happened during the last minute.
2. It also saves a file called Data Range that stores the begin and end timestamps of the first and last EVENTS processed ever. This file will be used later for knowing which Network Node is the most up to date.
3. The folder where it store this EVENTS happens to be a Github Repo, so every one minute changes are committed and pushed to Github. This serves as an always online copy of the EVENTS processed by this network node.

Initial Syncronization

When a Node comes online, it needs to download from somewhere all the events that occurred since it went offline and after that it can process new incoming EVENTs.

1. Each Network Node knows which are all the other network nodes running the Social Trading Network Service becasuse it has access to all Superalgos User Profiles.
2. That means that it can read the Data Range file directly from Github repos of each Network Node and see which one is the most up to date.
3. Then it can temporaly clone that repo and have it locally at its file system.
4. Knowing its own Data Range can scan the files from its last EVENT until the lasst EVENT of the most up to date node, and apply one by one those events into its own in memory Social Graph, and also save the files of the new EVENTS found.

The Bounty

All of the above explained has already been implemented to some degree, but it is now working as expected. We offer this bounty to incentivize a dev to take the current state of the art and take it to a working solution that can be used.

The Current Code

Most of the code is already at this file:

More Info?

1. If you need more info you can request it via comments of this issue.
2. If you have more info to complement the above explanation feel free to add it as comments of this issue.

Free to reimplement with new ideas

If you have better ideas of how to solve this problem and would like to implemented in a different way, feel free to do it. As long as it works in the end we will be happy with that too.

Want to pledge more SA tokens for this bounty?

Make your pledge in a comment and I will update the Title of this Issue.

[prepheadrus]

For the person or team working on its development, I can also promise 250K SA.

[end-debug]

The network now checks the DataRange File every 5 minutes to which Node is the MOST-UP-DATE, then loads all the files from local storage to the social graph.
If these an event sometimes it gets saved.

[BastianMuc]

All of the above explained has already been implemented to some degree, but it is now working as expected.

I assume this is supposed to say "not working as expected" - can any more details on what's the non-working part be provided in here?