Pythonification of Theos

[uroboro]

Motivation

It has been over a decade of Theos as the de-facto tool to create tweaks for iOS.

Being based on makefile and perl scripts, it is showing its limitations of maintaining it without much changes. The complexity of the makefile system and the mystery of perl itself has made contributions to the project from outsiders a seldom occurrence. We've managed to extend the makefile system beyond what their creators intended and now we're suffering its consequences.

Parallelizing the builds was a big improvement, but the startup time is still a concern. Similarly, there are other build systems faster or simpler than makefile if we remove the scripting code we put into makefile.

Switching to other build systems and more active/popular scripting languages could improve both issues and improve the survivability of the Theos™ project.

Projects

theos

At the moment, there's a private repository with some ideas.

Changes with this new implementation include:

• yaml project file

  This replaces the current makefile(s) for describing projects in a simpler and unified, structured format. This also includes the control metadata.

• CLI frontend (finally) that manages both build/package/install (equates to current usage of make) as well as calling nic.pl to create new targets.

These changes could be made public as a transition out of makefiles. Having a frontent tool we can hide the internal implementation of Theos.

Planned features:

• Caching most of the checks we run to get information about the user's environment: available tools and sdks, versions of tools. This should help reduce the startup time.

• Caching of the build rules. Instead of having all the makefiles do all the IF checks depending on environment, user makefiles and/or user input, we can store a copy of the final ruleset for the build. A suitable alternative to this custom work could be done by using the Ninja build system: we can hash the previously mentioned yaml file and produce a ninja file that needs to be recreated whenever the yaml file changes.
  edit (05/02/21): To clarify, the suggestion of using Ninja comes from dragon and perhaps just making use of DragonGen when it is released as a package. However, I've never read the source code but only seen snippets of what the author has posted on Discord and I am afraid of the one-liners.

logos

At the moment, there's a private repository with separate parser and generator to simplify maintenance. Current implementation uses libclang to tokenize the input because getting the AST with llvm's libraries would result in errors due to the custom syntax.

Project structure:

• parser

  This component takes care of parsing the source code and generating an AST-like structure with all the necessary metadata about the directives.

• generator

  This component takes care of producing the necessary code backing the sugar syntax that is Logos

  • base
  • substrate
  • internal
  • others?

dm.pl

The standard library provides a function that should be a suitable replacement for our custom compression code:

shutil.make_archive() (documentation)

It is capable of compressing with the following formats: tar, zip, gztar, bztar, xztar

We would have to rewrite the whole script but we'd be able to build it as a python module that can be used from the Theos module without having to shell out.

nic.pl & templates

We can switch to a popular templating system called cookiecutter. cookiecutter implements a similar approach to text replacement as our system; where we do @@VAR@@, they do {{VAR}} (known as moustache templates, popularized in the Jinja templating system for websites).

We can also combine it with PyInquirer to have finite options when collecting the data for creating the project. There's also a fork of cookiecutter called Nukikata including some of PyInquirer's functionality but is not as mainstream. We can present a friendly CLI UI for selecting options for the template variables.

Extra feature to add: If running on a device, we can collect installed app bundle IDs to list them as options for the filter target(s).

Afterthoughts

It's possible that I'm biased towards Python because of me using it daily at my current job, but a couple charts do back up the claim that Python is more popular than Perl (GitHub's octoverse, PYPL PopularitY of Programming Language, TIOBE). I believe it'd be less cryptic to read and write, and even if it's not bundled in every platform we support, it's not difficult to get it.

I've also had my share of inconveniences brought by Perl and make that have discouraged me from working on the project in the last years. I've made a few improvements here and there but nothing that would be considered a significant improvement overall.

[0cyn]

https://pypi.org/project/theos/ Sniped the package name pending the decision, so you guys aren't forced to suffer the same (expensive and slow) process as dragon.

[0cyn]

deleted a previous comment as it came across as very conceited without outside context.

the intended thing i meant to express: i’ve written a tool that does these things already, and it’s my hope theos can properly build off of that and bring what it can’t to the table.

I hate seeing my own mistakes re-created by others, and feel like taking in that project, especially including its flaws, and working from there may be a more effective way to accomplish the goals here.

whether that’s a full rewrite with across-the-board improvements, taking and reusing chunks of code/modules, and otherwise.

what i want to avoid mainly, is two separate groups doing the same thing the same way and reaching the same goal in twice the time than could be possible.

I’m very fully on board with this change regardless, and the lack of my direct contributions to theos should make it clear that my opinions here should carry no weight on their own.

glad to see things finally moving forward. 🎉

[EthanArbuckle]

i put together a python version of dm.pl that can be used https://github.com/EthanArbuckle/dm.py

[EthanArbuckle]

time to package the same directory (averaged over 5 runs)
dm.pl: 0.14s
dm.py: 0.08s
dpkg-deb: 0.05s

[kabiroberai]

iirc Theos uses the lowest compression level in non-release builds anyway so it shouldn’t make too much of a difference to build time. (update: yep, see 840b7e5)

[EthanArbuckle]

cool. These tests were with lzma compression level 9, so slower than what you’d be dealing with during normal builds.

[uroboro]

I went over the code, I like it. I'd add a couple distinctive exceptions

[EthanArbuckle]

noted, I can do that

[tale]

Just wanted to drop in and suggest that the python scripts be compiled into python bytecode.
Linked here is py_compile and compileall, two utilities that can generate python byte code from files.

The benefit this brings is that it reduces the startup overhead of scripts, and for a tool like Theos, I can imagine multiple scripts would be called every time you compile. While this isn't really a priority or something that needs to be worried about for a decent amount of time I just felt like it should be brought up so it isn't forgotten.