Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
312 Starfield [2024/01/04] [Various community members]
On the commit side, we managed to keep this one smaller (~250 commits rather than the ~500 commits that made up 311). On the complexity side, we probably failed, introducing a couple huge refactorings. A lot of Fallout 4 refactoring and patcher porting, Starfield support, native Linux support and refactoring WB's refresh handling are the biggest contributors to line count and commit number. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### FileInfo(s) (#336) #336 came back with a vengeance this time around. It started with fbb1925, which reworked the AFile API to cover installers and netted us some very nice performance gains and simplifications of refresh APIs - we'll some more work on refreshes later on. ----------------------------------------------------------------------- ### Native Linux Support (#243) Since Windows crapped out on me recently, I decided to ditch it and use Linux full-time. As a result, native Linux support for WB suddenly became a much more pressing issue. 312 improves it to the point where we now mark it as supported, with only some QOL issues remaining (and launchers, but those are a complete WIP anyways, see the later Launchers section in this commit message). There were a ton of random commits that made up this improved support. Basically, whenever I noticed something broken or in need of improvement, I would end up committing the fix, which means the commits are scattered like crazy. Nevertheless, here are some highlights: - e86e939 reworked temporary files handling, but this deserves its own section, which it will get below. - 1993f9b reworked parent handling in wx menus. We were always using Link.Frame.show_popup_menu, but this broke the Bashed Patch menus on Linux entirely. This turned into a mini-refactoring under #190 to track down and do this properly. - b762cc6 made the date and time dialog work on Linux (at the cost of no longer using locale-specific date/time formatting for it). - 20dd955 rewrote Steam game detection entirely to avoid the Windows registry. This was prompted by Starfield not creating a registry key at all, but I was planning to do this anyways, because Linux obviously does not have a registry and because it means users no longer need to launch the game once to generate a registry key. - 61d4d87 is what prompted me to actually mark Linux as supported. This merge added: - Proton detection, meaning the out of the box experience on Linux is now comparable to the one on Windows (in terms of it simply detecting your installed games with no manual intervention needed). - A warning when the Data folder is mounted case-sensitively. - Various fixes for race conditions and more, exposed by Linux's filesystems (perhaps specifically by btrfs?). - Functional Ctrl+Tab handling for cycling through WB's tab and some other misc improvements. - Also worth mentioning here is the File Operations (#241) section, see below for more on that. Linux was then finally marked as supported in c855882. Shortly before 312 was released, @BGazotti also contributed a whole host of small fixes and improvements for WB's Linux support in 00381da. Many thanks! ----------------------------------------------------------------------- ### Temporary Files (#665) This was born out of three needs: - On Linux, the global temporary directory (/tmp) is generally mounted in RAM. This means one can easily run out of space here when e.g. extracting a giant couple BSAs. And even worse, if the system has no swap configured, it can completely lock up when this happens. Wrye Bash should, in fact, *not* lock up the entire system. - We can get a decent speed boost by making sure the temporary directory we're using sits on the same device/filesystem as the Data folder. That way, the OS only has to rename some file paths rather than copying data around. - And lastly, our temporary file APIs were all over the place. There were three distinct types of temp file handling, and various pieces of code seemingly randomly chose which one to use: - bass.getTempDir/newTempDir/rmTempDir - Path.temp and Path.untemp - Just use Path.baseTempDir/tempDir or even tempfile directly and do it completely manually See the commit that introduced this refactoring (e86e939) for a full breakdown of the problems these APIs had. So we designed a new API that can cover all use cases and makes it hard to get wrong. Because, as it turns out, correct temporary file handling is *hard*. And another huge advantage of this design is that it will work with multiple instances of WB running in parrallel, which is an eventual goal. See the aforementioned e86e939 for the full lowdown. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525) The refactoring towards full FO4 Bashed Patch support is still ongoing. The goal was to get it done for 312, but then Starfield released and took the title of this version, plus we didn't want to drag out the release of 312 even further. Still, 312 brings the BP for FO4 very far. Work began in 2327ef4, which cleaned up header flags and implemented the next ten record types. Next up, 548bce5 tackled two difficult record types (NPC_ and OMOD, see the referenced commit for details on their problems) and implemented the next seven record types. With so many record types done, it was time to test them properly. To that end, d941cae ported the first batch of patchers over. In total, we now have 20/33 patchers available for FO4, though not all of them support all the record types they could support yet, simply because those record types aren't implemented yet. 28fb000 continued the records refactoring, though with the added complication that now, each time we implement a record type that an already-ported patcher can target, we also add support for that record type to the patcher in question. In that vein, this merge implements the next thirteen record types and adds them to patchers where appropriate. ----------------------------------------------------------------------- ### Starfield (#667) The star of the show (no pun intended). Note that this is early support, meaning that we don't actually support reading or writing plugins for Starfield yet. The main reason for that is Starfield's... *poor* design, when it comes to the plugin format. You can see the merge commit for more details (ec30f02), but basically, Bethesda clearly did not take moddability into account when designing the plugin format for Starfield. Unless they *drastically* rework the engine before releasing the Creation Kit, the Bashed Patch might not happen, period. ----------------------------------------------------------------------- ### wx (#190) It never ends (okay, maybe soon, but no guarantees). ea96e99 at least took some good steps towards closing off #190 for good by decoupling bosh (the data model) from balt/gui (the GUI backend). An important step towards #600 (headless mode) as well. Some more work landed in 170ad99, where @Utumno introduced gui.Lazy, a wrapper class for components that need to lazily create their native widgets. This led to heavy refactoring of the status bar and its buttons, taking down lots of dodgy code in the process. Also contained in that merge is a sub-merge, bd0a897, which utilized the new lazy classes to really tackle WB's image handling. Especially useful in preparation for the high DPI improvements that will be elaborated on later in this commit message and the launchers (see Launchers section below). ----------------------------------------------------------------------- ### Overlay Plugins (#668) The actual reason to be excited for Starfield's potential is shown in 9d21b40. Overlay plugins are a new type of plugin that does not take up a plugin slot at all, meaning you can have an unlimited number of them(!), with the downsides that such plugins can't contain new records (since they don't have a plugin slot to place the new records into) and must have at least one master. Unfortunately, due to the aforemnentioned massive problems with Starfield's plugin format, overlay plugins and ESLs aren't usable right now. Let's hold out hope that Bethesda can fix this, if only so the ~1 day of engineering I put into supporting them won't go to waste :P But jokes aside, the refactoring in this merge is very much worthwhile on its own. It makes supporting multiple mergeability checks for one game possible (since Starfield supports both ESLs and Overlay plugins - well, in theory it does) and thus opens the door for #596. ----------------------------------------------------------------------- ### Refreshes (#265, #353) We already mentioned fbb1925, which tackled refreshes from the perspective of BAIN and AFile. But midway through 312's development, I wanted to try my hand at implementing a new tab, namely the xSE Plugins tab (#456). That made me run headfirst into a wall, namely the fact that BAIN only knows about plugins and INI tweaks when it comes to trackin and updating other data stores. BSAs were kind of in there too, but not fully. My new tab needed BAIN to keep proper track of it as well, which meant refactoring. That bloomed into a very nice merge in c6ec399, which took down a bunch of old and rusty APIs (e.g. RefreshUIMods, saveListRefresh, _is_ini_tweak, etc.). The refactoring eventually culminated in us centralizing cross-tab communication via bass.Store and taking down a few hundred lines of really annoying, duplicate code. Some followup refactorings grew out of this as well. 70fe061 took down lots of complicated code related to ghost handling, paving the way for much more refresh handling work to come in 313+. Similarly, b8d9e0a refactored ModInfos.refresh, slowly unraveling the complicated stack of refreshes we've got going on. 387c9df tackled refreshes from the perspective of load order handling, attempting to simplify the latter significantly. One big goal here was to preserve as much information about the changes in load order throughout any given LO operation as possible, since that is crucial in order to properly refresh only the changed files - think elegance *and* performance. ----------------------------------------------------------------------- ### File Operations (#241) d897347 reworked our file operations backend significantly, prompted by two things: - Linux support, since we can't delegate to ifileoperation like we do on Windows (duh). - The last couple bits of FOMOD support (fingers crossed). Specifically, it turned out that StarUI, a Starfield mod, would fail to install correctly in WB when run via FOMOD. The reason turned out to be that it wanted to install a single file to two destinations. You can see the linked commit for all the details. BAIN did not have much trouble handling this, since it generally stores everything as destination -> source dicts, but our file operations backends did not support this at all. As part of this commit, we also introduced support for reflinks/file cloning to Wrye Bash. This currently has to be done via a less-than-ideal third-party depedency. An upstream request to the CPython project has stalled. As a result, we only support reflinks/file cloning on Linux and macOS filesystems right now (not a major problem, since ReFS has shown no signs of making it to mainstream Windows deployment yet). But what are reflinks? They behave like regular file copies, except that the backing data is shared between both copies. This is generally enabled by copy-on-write (COW) filesystems, since this feature is pretty much implemented for free on such filesystems. This lets all of WB's copy operations become much faster on Btrfs, XFS, ZFS, APFS, etc. ----------------------------------------------------------------------- ### Auto-Splitting the BP (#533) A long-standing feature request (open for more than three years at this point) has been addressed in 312: automatically splitting the Bashed Patch into multiple files once it ends up with too many masters for a single file. The advantage is obvious, especially compared to the previous behavior (throwing all work away at the end). 6ef2198 introduced the feature, though it turned out to be simultaneously much less work than I expected *and* much more complicated than I expected. Which is weird. ----------------------------------------------------------------------- ### Scattered Packages (#670) I have had this idea for more than three years, at least since I completed work on #380 back in 2020. The last big problem with FOMOD support in Wrye Bash was that a whole bunch of weirdly packaged FOMODs could not be installed in BAIN without a silly workaround (creating an empty plugin and putting it somewhere in the package to trick BAIN into thinking it's a simple package, then using its FOMOD support to remap files and thereby skip the empty plugin, which won't be referenced by the FOMOD's ModuleConfig). The plan was to model the properly, then see what happens. That meant creating an entirely new type of BAIN package. Previously, we had three types: - Simple packages, which contain some files to be placed in the Data folder. - Complex packages, which contain multiple sub-packages. Each sub-package behaves like a simple package. - Invalid packages, whose layout BAIN can't make sense of. Granted, from an end-user perspective, there are more types of 'packages': - Simple/complex packages, which behave like simple packages but have some folders in between the package root and the actual files that need to be placed in the Data folder. In WB's code, these are treated as simple packages. - Markers, which don't exist in the filesystem and are only an aid for users to better organize their packages. In WB's code, these aren't technically packages at all. 759055c introduces a new type of package: - Scattered packages, which have an unrecognized layout, but also come with instructions that tell BAIN how to make sense of the *scattered* mess of files that it otherwise can't parse - hence the name. Combined with some more refactoring to make BAIN recognize scattered packages by the presence of an 'fomod' folder and the aforementioned multi-destination file installation support, this finally tackles the last remaining FOMOD issues (again, fingers crossed). ----------------------------------------------------------------------- ### High DPI (#511) One big goal for 313 is #511, i.e. the ability to customize checkbox colors. But this is much more than just that - it's also about simplifying the resources for checkboxes by using a single SVG instead of a ton of different PNGs, making it possible to have more overlays over the icons (the current implementation of the wizard overlay is just terrible, it's literally a matter of copy-pasting all the checkbox icons, adding a wand over them and then loading all of them manually. This won't scale if we want to, e.g., add an FOMOD overlay). And, of course, this is also about using an SVG to make these scale properly on screens with >100% scaling. To that end, ff276b1 addresses high DPI handling of images. Yes, we already had support for high DPI images via SVGs since #557 in 313, but it turns out that implementation had lots of holes - most notably at 200% scaling, where wxPython decided to scale all our icons *again*, making them absolutely massive and really ugly. With the aforementioned commit, nothing should stop us anymore from tackling #511 in 313. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BGazotti and many more that GitHub's contribution tracker doesn't list.
- Loading branch information
