Blockchain Engineering - class of 2024 - Team Democracy-3: self-evolving blockchain

[synctext]

Project assignment for Delft University of Technology master course called "blockchain engineering".
Democracy-3 self-evolving blockchain

• On-chain kickstarter model for discovery and funding of plugins. Or scientific formulation; engineer a ledger-based system which has an internal competitive market for expansions and mutations
• Democratic decision process on which mutations are considered good. When approved by threshold voting code is deployed to all users. Bounty is a multi-sig Bitcoin payment for programmers of mutations. This is based on the running code of a shared wallet and collective money.
• Code and system upgrade using plugins, we have prior permissionless code execution. See running code of Android code which bypasses censorship by the Google Play Store.

All three team members got it compiling.

Background reading:

• DevID: Blockchain-based Portfolios for Software Developers

Upcoming sprint assignment

• fetch major code update for superapp modules and IPv8 sub-module
• understand existing code
• Make an "Hello World" .APK which can import inside Superapp
• Fix broken feature: bundling .APK with app and installing it is now broken with upgrade of Android security

[synctext]

• not very fast start of project
• read the link in the assignment
• confusion on the assignment compile-time versus run-time self-evolution.
• Question: most coding will be done is Superapp? Yes ❗
• Upcoming sprint: repair the run-time APK installing
• lacks 1 feature: requires you to be connectable. (ToDo read about: connectable port in Android) or run a known Bittorrent tracker.

[JopSchaap]

What we did this weak:

• We created a test apk that runs inside the app, using the run-time APK installing
• Made/divided tickets on minor improvements on our repo
  • for the tickets see: the repo
• Fixed the run-time APK installing bug. The run-time apk installing now runs on Android 14+
• Set up github repo rules, to ensure a good way of working
• Adds feature that allows renaming of APK

Problems we encountered:

• We encountered the problem where APKs do not torrent
  • We suspect this is an issue with the p2p network
  • We saw sometimes no peers showing up in the debug screen
• Build release apk CI job fails continuously on our github.
  • Is this because of us, or can this just not be fixed.
• Is there an equivalent to pretier in Kotlin? (To format the kotlin files without having to run the gradle thing everytime)

[synctext]

• Possibly a port connectivity issue or not? Test with https://canyouseeme.org/ type of tools
  • Flaky connectivity ? (torrenting works for some team members)
  • Port forwarding in Android studio on Mac just simply works? No need to configure port forwarding between two emulator instances.
  • zero peers found issue
  • debug screen: number of incoming IPv8 UDP packets
• fixed several bugs? Added Android 14 compatibility with safer dynamic code
• To format locally the files ktlint
• Renaming feature: https://github.com/JopSchaap/trustchain-superapp/pull/14/files
• Feel free to use this code from another team
  • Blockchain Engineering - class of 2024 - Team Democracy-1: blockchain networking (b) #7913 (comment)
  • document the usage please!
  • they got transfers operational and Wireshark-debugging perfected
• Freedom-of-Computing contains a gossiping mechanism which periodically shares local applications to other users and downloads unseen applications from other users from FOC.readme
  • first reproduce the networking debug code of other team
  • add their networking debug to your FOC community
  • other peers you discover
  • add to GUI

Sprint ToDo:

• Hopefully you can make more progress in this course!
• Get wireshark going please
• Understanding of connectivity issues. Connectable at a port
• Study carrier-grade NAT puncturing
• Operational debug screen.

Concrete Week 10 goal:

• FOC is fixed and connectivity issue solved
• FOC of discovery of new code is merged with running code of voting
  • current behaviour is to autodownload anything you find. ✔️ after new code is discovered in gossiper, you automatically download it using Bittorrent. However, installing is not automatic.
  • user decides to delete or install the newly discovered code.
  • this decision is shared with with others in an upgrade of the gossip protocol
  • downvoted and upvoted (no need for multi-sig complexity)
  • collect these votes in background what others voted and show to users as FOC enhancement
• Week 10 test: download your .APK, join network, vote on APK proposals, see votes come in.

[grimadas]

Linter for kotlin: https://github.com/pinterest/ktlint

[JopSchaap]

What we did this weak:

• We added a vote button
  • Right now a user is just able to vote once, and cannot cancel their vote
• We added a down-vote button
• We added the ability of the app to store votes by user id
  • We want to make this signed with the user id so peers can share other peers votes
• We restructured the layout of stored app in the file directory, to store apks in a directory which are named by their hash.
  • This is not yet merged since it doesn't work yet for torrented apks.
• We got wirshark going
• We figured out what to do to make the connectivity issue go away
  • We figured out you should have the app installed in two (physically) different devices. (e.g. two emulators on one machine doesn't work)
    • We are however not sure why this is required
• We got the debug screen (from the other group) up and running

Problems we encountered:

Questions:

• Should there be the option to remove votes?
• Does it make sense to get votes for an apk, you do not have torrented yet?
• Is it a problem to gossip all the votes at once, or is there a better approach?

[pbhaskaran]

[grimadas]

For the next week:

• What is member id? When it is created?
• Display votes on the screen. Now the buttons are not dynamic, find easiest fix for that.
• Restructure vote object: it should have item_id, public key (or member id), signature, freshness, vote strength (-1 down, 0 neutral, 1 up)
• Store what you received from the network, you can always garbage collect later.
• The simplest way to implement the gossip algorithm: send to your neighbours if the message is new. Downside - it might be bad for performance
• A bit better version: push-based with ttl (time to live) inside the vote object. Ttl is an integer (if value is > 0) that indicates if the message should be pushed further. Then you decrement it and push to random k peers you connected to.
• Have data for benchmarking to illustrate how bad/good the gossip is. Simplest way for now record time when message was received, and when was send with the peer_id.

[JopSchaap]

What we did this week

• We restructured the vote objects, to include signatures/ public keys
• Signatures are verified with the associated public keys
• Vote objects now also come in a neutral variant
• We implemented a simple hot potato gossip working, that distributes votes to other peers
• The number of votes are now shown next to the vote up/down button
• A very basic pull based gossip, where newly introduced peers ask for votes they are yet unaware of

Things we ran into this week

• We were unsure how to ensure a peer doesn't generate two keys and uses both to vote twice

Things we want to work on next week

• We want to make it possible to revert your vote
• Improve on sharing votes to new peers (e.g. pull part of the protocol)

[AirWinter]

Demo of push gossip: https://jam.dev/c/3b808243-706e-4d41-b082-668026a82a53

[grimadas]

Next week:

• Fix pull based gossip. For example, you can delay it until we are connected. Other option you can used "extra_bits" for introduction_request/response.

• Get the number from a simple benchmark: Time to convergence (when everyone in the network has discovered the apk), and overhead per vote (bandwidth, ...)

• Now the votes are collected with a queue that is polled every second. What is the easiest way to implement it asynchronously? Implement a singleton? Any ideas @InvictusRMC?

• Check if you can install/reinstall super-app with new apk. (It's a sidequest, so you're on a time budget)

[InvictusRMC]

• Now the votes are collected with a queue that is polled every second. What is the easiest way to implement it asynchronously? Implement a singleton? Any ideas @InvictusRMC?

Right now the implementation is indeed very inefficient. What I suggest is making use of a listener. That way we can check whether conditions are met whenever receiving a new block, as opposed to checking periodically.

[AirWinter]

• Removed all the queues which increased the latency for push & pull-based gossiping
• Improved latency of push based from 5-10 seconds down to 0.1 seconds
• Improved latency of pull based from 10 seconds to 0.2 seconds
• Benchmarked the results and made improvements to reduce message size
• Found that Pull Based gossiping couldn't scale as the message would eventually get too big, so switched to using the EVA protocol and tested that it scales (tested up to 50 votes)
• Fixed a bug that occurred when voting on an APK that was gossiped to us
  
  This graph shows the size of the message as the number of votes increases. The original implementation only scaled to 3 votes, then after reducing the message size it scaled to 4 votes but the latest version that uses the EVA protocol can scale way further. The latency was also improved as previously mentioned, and also removed the operational overhead of repeatedly checking the queue.
  
  The graph shows how we reduced the message size, but we also reduced the latency and removed the operational overhead of repeatedly checking the queue.

[JopSchaap]

• We looked into how installing could work
• There are three solutions we looked into
  1. Replacing the standard apk with a small one that only loads the new apk, and thus would allow new versions
     • Problem1: the current superapp constructs views with xml fragments which cannot be loaded
     • Problem2: One could only install one version
  2. use custom class loader which would allow loading updated versions of classes
     • This would be the most optimal way of loading classes
     • This would allow patching the super-app
     • Problem1: We could not get class loading to work in this way (but we think it is possible, if structurally complecated)
     • Problem2: loading xml files still wouldn't work
  3. We could put installed apps on the home screen:
     • This is what we are currently working on
     • Problem1: loading xml files still wouldn't work

[grimadas]

Solid progress

For the next week:

• Keep good documentation to help with onboarding for the next teams
• Exporting to the main screen
  • Priority: to main screen, first manually
  • Then switch to manual import to the main screen
• Gossip improvement: Use short ids when first sending pull-request gossip. The initiator should be the node joining.
• Improve benchmarking figures to make them more interesting: time to convergence and throughput as metrics.
• Side-quest: Learn more about set reconciliation to improve your gossip even more:
  https://github.com/AljoschaMeyer/set-reconciliation
  https://ics.uci.edu/~eppstein/pubs/EppGooUye-SIGCOMM-11.pdf

[pbhaskaran]

[JopSchaap]

What we did this week

• We ran tests on the base implementation of pull based vote gossiping see: comment above
• We improved the pull request gossiping, with only sending not yet known votes.
• fixed bugs
• Write kdoc documentation for classes and methods
• Add additional readme for the freedom of computing sub-module see: README.md
• Fix issues with "installing" apps
• add vote threshold installing
• Sync branch with main branch over at tribler

Things we ran into this week

• We were unsure what an appropriate vote treshold would be(right now it is set to 10)
• Should apps also be automatically uninstalled, if # votes drop below treshold?

[AirWinter]

Improved graph for push-based gossip

APK link
APK Master
Old APK
my-special

[synctext]

• focus on gossip (mostly ignoring the 4G/5G network connectivity issue)
• 500 votes testing
• pull gossip: advertise all votes you have already (ID for a vote; UUID, why not SHA256 (32 bytes) spoofing 💣 ?)
  • infinite growth of advertisement?
  • monthly data bundle gone in 60 seconds?
  • current code: advertise everything you know
• usage of script to generate votes/workload + Android emulator (1x per PC)
• Running code for a democratic process which determines your apps inside superapp
  • each Bittorrent swarm with sufficient votes gets download
  • installed and show on frontpage upon startup 😮
  • Yes, we are aware this lacks security measures !!!
  • Lets focus on this for "Week 10" test
• Nothing scales in this app. Nothing is secure. Will take years/decades to fix!
• Honest conversation: difficult for the app to break outside the tech bubble 🤔
  • why upgrade to this, versus continue to use Tiktok and Youtube ??
  • Needs to go viral and hear friends use it
• Week 10 checklist
  • Running APK
  • possibly the "i" info screen on votes seen
  • live voting test
  • frontpage install test
  • documentation in readme.md
  • release of your .APK ?
  • bonus: make it scale!
  • bonus: create PR of all your improvements upstream

[JopSchaap]

What we did this week:

• We have demo prepared apk prepared at: https://github.com/JopSchaap/trustchain-superapp/releases/tag/demo-release
  • If nothing shows up your preference store might be corrupt because of an old version of the FOCInstaller. To fix you should clear the local storage of the app, see release on how to do that
• We updated the info screen to include Threshold for install value and total number of votes stored
  • Please note that this behavior is not yet merged and thus not shown in the demo

[grimadas]

For next week:

• Investigate the bug showing different number of torrents
• Add more apks (10 and more) either into the default app, or prepare a bunch of link with some dummy apks (some sound, image, search)
• During the demo: show upload of apk and gossip of apk.
• Another demo: popular apk auto-install (threshold of three votes)
• Prepare a compilation README file with your decisions, and limitations of your protocols (what works and what is future work)
• Hand in Apk and GitHub repo wit the code.
• @synctext what is the expected deadline to submit the upstream PR?
• Unit tests and good coverage
• Corner cases - broken apk gossip, big transfer. Have tests to show the coverage for corner cases.

[AirWinter]

Final Ticket Update

This is just a brief overview of our project, check out the README for more insights.

Project Features

1. Fixed issue that prevented APKs from being opened on new versions of Android (14+)
2. Built a voting system for FOC app
   • Pull-based gossiping
     - Triggered when opening the FreedomOfComputing app
     - Sends request to all your direct neighbours
     - Only respond with votes that you don't already have
     - Benchmarked the performance in terms of latency & message size
   • Push-based gossiping
     - Triggered when placing a vote, tries to inform others about that specific vote
     - Inform log(n) of your neighbours about this vote, where n is number of direct neighbours
     - Hot potato gossip with TTL set to 2, so neighbours relay it unless TTL is 0
     - Benchmarked the performance in terms of latency & message size
   • Persisting state by writing to .dat file
   • Signatures to verify votes
3. Installation of APK to homescreen
   • APK automatically installed to homescreen once it passes a certain threshold (set to 100)
   • Manual installing APK to homescreen when they haven't passed the threshold

Documentation

The README contains most of our documentation, as it provides a developer onboarding guide explaining the main methods of the FreedomOfComputing App. It also contains a section which goes more in depth into the features that we implemented in this project, the key decisions we made and highlights some of the current limitations as well as potential future improvements. We also made sure to include detailed KDocs for all new and most existing methods. These comments cover parameters, return statements, and provide an overview of each method's functionality.

Testing and Coverage

The new classes with heavy logic that required testing were FOCVoteTracker and FOCSignedVote
class. For the FOCVoteTracker class, we reached 100% method coverage with 90% of lines covered
(The missing lines are just mainly console log statements used for debugging) and on
the FOCSignedVote class, we reached 100% method coverage and 100% line coverage.

Live Demo

For the live demo download this APK and follow these steps:

Steps:

1. Install the APK
2. Check that you're connected via the debug screen
3. Open FreedomOfComputing
4. This should do a pull request and should fetch all the votes you're behind
5. Place votes on APK which should push to others, also have others place votes which should push to you
6. Have one APK go over the net threshold of 100, this should automatically install it to the superapp home screen
7. Long click on an APK that doesn't have enough votes for the threshold and install it to the home screen manually
8. Have a working APK gossiped automatically
9. Have a broken APK gossiped automatically

Recordings in case live demo doesn't work:
Demo Voting
Demo Downloading APK
Demo Torrenting APK

[synctext]

• clearly a demo designed to impress 🎉 🚀 🎉
• 15k votes received in EVA 😲