Update 1.27.0: Dependency Resolution Improvements

[Sewer56]

Changelog (1.27.0)

Read and Discuss in a Browser.
Previous Changelog.

This is a longer 'Progress Report' kind of post. Consider reading in a browser.

Dependency Resolution Update

This update is focused on more reliable resolution of missing dependencies, notably, reducing the amount of time that the following message is seen:

A fix for end users:

• Launching Reloaded without Internet after a mod download will no longer render missing dependencies undownloadable.
  • We now ensure we don't remove unused dependency metadata if we're missing dependencies.
• Fixed a very rare edge where it was technically possible the launcher incorrectly reports a missing dependency.
  • This would happen if a mod had multiple download locations (e.g. GameBanana+GitHub), but was removed from 1 location.

A fix for mod authors:

• A launcher restart is no longer required to update mod dependency info.
  • This info is what's used to download missing dependencies.
  • You can now safely add/edit a mod in launcher, set dependency, add some files and publish.
  • Previously a launcher restart was required to ensure dependency info was embedded in your mod.
  • We now update the dependency data before publish, ensuring mod authors don't pack mod with missing dependencies.

An Automated Error Handler for Edge Cases:

• We now use the Search Index as a fallback, if all else fails.
  • The search index is what normally makes your mod searches in App super fast.
  • But it also knows the unique ModId of each uploaded mod.
  • So we can use this to find missing mods.
  • This currently works for 11 games that use Reloaded-II on GameBanana (I don't index any other sites currently).
  • Enabling this has some security implications. This is only used as a last resort fallback, when all else fails.

This should hopefully help in getting cases where users experience missing dependencies closer to 0.
Especially in situations where a dependency is hosted only in 1 location, e.g. Only <This Mod Site>

Updated Docs

Documentation and in-launcher tutorial has been updated to feature installing via drag-and-drop as a method of installation.

(Hint: It's animated in the launcher)

Infrastructure Update

The open 1st party package server is an unsung hero ensuring your modding experience is as smooth as possible. This is the place where critical code mods are fetched from automatically when the primary source (like GitHub) hits its limits - a common scenario given GitHub's restrictions of 60 requests per hour per IP.

Given Reloaded's modular nature, you'll typically hit that limit restarting the launcher around 4 times. Now imagine you have a family, or are in a hotel room, or just troubleshooting, this limit may already be exhausted. Now you download a new mod, which requires some framework/critical component and suddenly you're told...

Not ideal.

With the surge of users, traffic on this server has risen drastically. In the past 20 days we have transferred 6TB of data. Month started with approximately 5MB/s and is now 7.5MB/s. This is especially surprising, as the critical code mods stored here are typically under 0.5MB in size.

I only found out a few days ago, as I was looking into why the server was unresponsive. Sometimes queries randomly took ~2-3 seconds to respond, and I was rather flabbergasted.

The Package server runs on a very minimal-effort fork of BaGet, renamed as BaGet-Reloaded to avoid confusion with the original project. Using the Sqlite3 backend.

To the best of my knowledge, the delays in responses were caused by a combination of:

• A lot of writes to the DB, which blocks reads in Sqlite out of the box.
  • Notably, BaGet increments the 'download count' every time a package is returned via search.
• Sustained load on a shared CPU (a.k.a. 'vCPU'). [to a very, very small extent]

I changed the settings on Sqlite3 to use WAL and synchronous=NORMAL. This allows us to perform reads during writes, thus fixing up the performance. In addition, I bumped the server specs a bit, just in case the load continues to climb. Currently the server is completely responsive again.

A Thank You

Services like the Package Server are supported by your donations.
They also help cover occasional game purchases.
Recent Example: Fixing Compatibility with Special K in presence of Steam Copy Protection/DRM for games like Granblue Relink

In the ~4 years I've been accepting donations so far, I've received a total of £358.87 (and 4 games). This has helped cover most of the costs involved. e.g. Package Server, Server Backups, Discord Bot(s), Search Index, Services for some Online Mods, Domain (Website Name) Costs etc.

With these donations I can almost break even. I don't have any concrete numbers (hard to calculate), but based on some quick maths, I've probably spent around £450-500 on Reloaded related things so far.

Thank you.

About the Future

Please remember. For most of its life, Reloaded has been a solo project.
Not just the Manager, but the Loader and many of the core components used across your popular mods.

In any case, the work on the successor continues, albeit slowly.

Every week, I spend around 24-30 hours on Reloaded things, on top of my fulltime job. Hopefully in 2 years, we'll live in a future where users can start experimenting with porting core components from Reloaded-II to Reloaded3. This is not easy to do, but I will relentlessly keep going, until it is done.

The intent of the recent reliability/bug-fix patches is to allow me to step back from maintaining Reloaded-II for a longer time and let me focus on the successor.

If you are a programmer, I should hopefully have a 'Specification' ready not too long from now. The work on this is resuming today, after a long hiatus (linked version is from last year), so various core parts of Reloaded3 will be open to community contributions. Until then, hang tight.