309 Python 3 preparation, pt. 2 [2021/06/01] [Infernio, Lojack, Utumno]

309 continues where 308 left off, containing the last bits of
refactoring and transition code needed before we can move to Python 3.

All mentioned commits are authored by @Utumno or @Infernio unless
otherwise mentioned.

-----------------------------------------------------------------------

### Python 3 (#460 and #597)

Way too many commits for me to go in depth on all of them here, so a
short list of important commits follows:
 - d22469b replaced all remaining
   usages of `items()` and `values()` with the `iter*` or `view*`
   equivalent, as needed by 2to3 to successfully convert them to the
   Python 3 versions.
 - c120e18 audited all usages of
   `open()` to ensure that they either open in binary mode or pass in
   an encoding. Interestingly, Python 3.10 will finally contain an
   option to emit a warning whenever this is violated:
   https://docs.python.org/3.10/whatsnew/3.10.html#optional-encodingwarning-and-encoding-locale-option
 - e823570 got rid of the last couple
   places that were causing bytestrings to end up in our settings.
 - f2f1140 added code to convert
   bytestrings in existing settings files to unicode strings.

-----------------------------------------------------------------------

### Patchers (#312 and #494)

The eternal war continues in bb52329,
which split the 'Race Records' patcher into several patchers. This was
absolutely essential to porting the patcher to other games (which was
done in the follow-up merge 2c4ce17).
The race patcher was incredibly complex, containing as much
functionality as *seven* smaller patchers packed into one. Most of the
now-split patchers could use more refactoring to shrink them and make
them more maintainable, but this is already a huge step.

There are several side-wars being fought as part of the eternal patcher
war. One of those is the war against records, and another one is the
war against parsers. The last major advancement on this front was a
merge near the end of 307's development that created a proof of concept
design for an ABC (abstract base class) for the parsers.
9840c3b and
bdd19a1 take this much further. In
the process it has unearthed just how unfinished and rough the parsers
really are, including tons of previously undiscovered bugs. This is
still not done, but already a massive improvement.

-----------------------------------------------------------------------

### Enderal SE (#582)

Enderal: Forgotten Stories: Special Edition is one hell of a title.

Adding support for it was mostly straightforward, the only really
interesting thing is that some fairly delicate untangling of load order
properties was needed. First merge was done in
9a9981b, with a follow-up commit
implementing the more difficult parts in
4298fc1.

-----------------------------------------------------------------------

### wx (#190)

Some minor de-wx'ing happened in
5de85b8. Nothing particularly
exciting, most de-wx'ing work is now focused on shrinking balt by
moving its code into basher and/or gui. The same goes for
b4290e6 as well.

7386e12 is another commit thinning
balt and refactoring gui code further (this time targeting file
selection dialogs).

-----------------------------------------------------------------------

### Game refactoring (#584)

Just in time for the Windows Store merge detailed below, we refactored
the way that game-specific about patchers is stored in
a7d7300.

-----------------------------------------------------------------------

### Windows Store support (#585)

Just as we were getting ready to do some extensive testing on Python 3
and perhaps do a release soon, Microsoft bought Bethesda and decided to
drop new versions of four Bethesda games (TES3, TES4, TES5 & FO4) on
us. These proved to be an absolute pain to support due to them being
Windows Store apps, which were clearly not designed to be moddable:
 - Finding the paths at which the games are installed is pretty
   involved. If we had access to UWP APIs it might be much easier, but
   we don't. As a result, we basically have to emulate what those APIs
   do by jumping through the registry ourselves.
 - The game EXEs are locked down hard. That's also the reason why
   script extenders and mod managers like MO2 will not work with these
   versions of the games.
 - Launching the games has to be done through a special command (since
   you can't even access the EXEs).
Nevertheless, @lojack5 and @Infernio managed to add full support for
these games to Wrye Bash. BAIN, Bashed Patches, load order management,
etc. - it all works.

-----------------------------------------------------------------------

### Plugin Checker rework (#83)

A long-standing feature request was to make the plugin checker aware of
record type collisions (where two records have the same FormID, but
different record types - this can cause anything from unexpected
behavior to CTDs). We took that much further and made the plugin
checker scan for all kinds of problems in
6aa9bba, including deleted navmeshes,
deleted references, deleted base records, HITMEs and even added a
heuristic detection for injected record collisions. The Plugin Checker
also contains full explanations for all of these problems, so don't
worry if you don't know what some of those terms mean.

-----------------------------------------------------------------------

### Fallout 3/NV feature parity (#468 and #594)

As of 309, Wrye Bash finally has feature parity with valda's FO3/FNV
ports. The aforementioned merges that split and ported the Race Records
patcher were the second to last step, and the final step was
8977e43, which ported the Tweak Names
patcher.

We even gained another (mostly) FO3/FNV-specific feature that valda's
versions lack in the form of d363475,
which allows BAIN to extract `.fomod` files, similar to how it can
already handle extracting `.omod` files.

-----------------------------------------------------------------------

### LOOT v0.17 compatibility

02435d3 added support for features
introduced in libloot v0.16.1, which will become usable in the
masterlists once libloot v0.17 is released.

-----------------------------------------------------------------------

Massive thanks to everyone who contributed to this release, including:

@Infernio, @Utumno, @lojack5, @sibir-ine, @valda and many more that
GitHub's contribution tracker doesn't list.

.github/workflows/wb_ci.yaml

@@ -27,12 +27,11 @@ jobs:
         python -m pip install -U pip
         python -m pip install -r requirements.txt
     - name: Build Wrye Bash
-      # Note the configs - needed for build.py. Also, we don't build the
-      # installer on CI, fails with an empty traceback for whatever reason
+      # Note the configs - needed for build.py
       run: |
         git config --local user.name "GitHub Actions"
         git config --local user.email "github-action@users.noreply.github.com"
-        python scripts/build.py --verbose --no-installer
+        python scripts/build.py --verbose
     - name: Test Wrye Bash
       run: |
         python -m pytest

Contributing.md

@@ -31,24 +31,20 @@ Please also read the [relevant page in our wiki](https://github.com/wrye-bash/wr
 I'm really glad you're reading this, because we need volunteer developers to
 help advance this project.
 
-Good starting points are the ['good first issues'](https://github.com/wrye-bash/wrye-bash/issues?q=sort%3Aupdated-desc+is%3Aopen+label%3AM-good-first-issue).
-See also [issue 200](https://github.com/wrye-bash/wrye-bash/issues/200) for
-some refactoring tasks we need help with.
+Good starting points are  [issue 200](https://github.com/wrye-bash/wrye-bash/issues/200) and the ['good first issues'](https://github.com/wrye-bash/wrye-bash/issues?q=sort%3Aupdated-desc+is%3Aopen+label%3AM-good-first-issue).
 
 #### Set up
 
-To contribute to the code, fork the repo and set your fork up as detailed in
-[[git] Syncing a Fork with the main repository](https://github.com/wrye-bash/wrye-bash/wiki/%5Bgit%5D-Syncing-a-Fork-with-the-main-repository).
-The recommended way to code for Bash is Pycharm
-([set up instructions](https://github.com/wrye-bash/wrye-bash/wiki/%5Bdev%5D-Set-up-Pycharm-for-wrye-bash)).
+To contribute, set up a fork and PyCharm [as detailed here](https://github.com/wrye-bash/wrye-bash/wiki/%5Bdev%5D-Set-up-Pycharm-for-wrye-bash).
+
 Our wiki contains dev, git and github labelled articles to get you started with
 setting up the repository and git basics. Please read at least:
 
-* [[github] Branching Model & Using The Repository](https://github.com/wrye-bash/wrye-bash/wiki/%5Bgithub%5D-Branching-Model-&-Using-The-Repository)
-* [[github] Branching and merging to dev using rebase](https://github.com/wrye-bash/wrye-bash/wiki/%5Bgithub%5D-Branching-and-merging-to-dev-using-rebase)
 * [[dev] Coding Style](https://github.com/wrye-bash/wrye-bash/wiki/%5Bdev%5D-Coding-Style)
 * [[dev] Deciphering Source Filenames](https://github.com/wrye-bash/wrye-bash/wiki/%5Bdev%5D-Deciphering-Source-Filenames)
-* [[git] Commit guidelines](https://github.com/wrye-bash/wrye-bash/wiki/%5Bgit%5D-Commit-guidelines)
+* [[dev] Commit guidelines](https://github.com/wrye-bash/wrye-bash/wiki/%5Bdev%5D-Commit-guidelines)
+* [[git] Syncing a Fork with the main repository](https://github.com/wrye-bash/wrye-bash/wiki/%5Bgit%5D-Syncing-a-Fork-with-the-main-repository).
+* [[github] Branching Model & Using The Repository](/wrye-bash/wrye-bash/wiki/%5Bgithub%5D-Branching-Model-&-Using-The-Repository)
 
 When ready: do not issue a pull request - contact instead a member of the team
 in the relevant issue or mentioning them on your commit and let them review.

Mopy/Docs/Wrye Bash Technical Readme.html

@@ -1038,8 +1038,10 @@ <h4>Functions in OBMM that are meaningless in Wrye Bash</h4>
 <h2 id="trusted-binaries">Trusted Binaries</h2>
 <p>This format is used for exporting and importing data by the
     <a href="Wrye%20Bash%20Advanced%20Readme.html#settings-trusted-binaries">Trusted Binaries</a>
-    settings page. It is a simple plaintext format, stored with UTF-8
-    encoding in files with the <code>.txt</code> extension.</p>
+    settings page. It is a simple plaintext format, stored with UTF-8 encoding
+    (do <em>not</em> include a BOM when writing, but be ready to handle a BOM
+    when reading since versions of Wrye Bash before 309 wrote BOMs) in files
+    with the <code>.txt</code> extension.</p>
 <p>Each line is either a <em>statement</em> or a <em>comment</em>. If the line
     begins with one of they statement keywords, as described below, it is a
     statement. Otherwise, it is considered a comment and ignored
@@ -1191,7 +1193,7 @@ <h3 id="trusted-binaries-structure">Structure</h3>
 </div>
 
 <div id="footer">
-    <div id="version">Wrye Bash v308</div>
+    <div id="version">Wrye Bash v309</div>
     <a href="Wrye Bash General Readme.html">General</a>&#160;&#x2726;&#160;<a href="Wrye Bash Advanced Readme.html">Advanced</a>&#160;&#x2726;&#160;Technical&#160;&#x2726;&#160;<a href="Wrye Bash Version History.html">Versions</a>
     <div id="oldie">Javascript disabled, menu and slideshows nonfunctional.</div>
 </div>