Assets API Code Gegerator. The tool generates code by entity_defs/*.def and types.xml for IDE hinting and type checking

[ve-i-uj]

Overview

Assets API Code Gegerator is the tool that generates code by entity_defs/*.def and types.xml. The generated parent classes can be inherited in the code to allows the IDE to hint the entity interface and to type check.

The generated code contains:

• Parent classes of server-side entities from *.def files. The parent classes have methods, properties defined in entity_defs/*.def
• Parent classes for entities interfaces
• Parent classes for entities components
• Generated types by types.xml (including types that return FIXED_DICT converters).

The generated code has links to entity def files, their remote methods and types, which makes it easier to navigate through the code.

The generated classes have the empty body on runtime in KBEngine. So the classes don't have effect for KBEngine runtime.

The generated classes are parsed without errors by Enterprise Architect . This makes it possible to import generated classes into Enterprise Architect and build diagrams to visually describe the client-server logic (for example, through a sequence diagram as the generated classes contain remote methods).

The generated classes are inherited in the "scripts" code by entities. That allows the IDE to hint the interface of remote methods and the entity.

N.B.: I copied this from my project's README (for the date 2023-09-03). The updated version can be viewed here.

I am not a native English speaker so I apologize in advance for my English.

Table of contents

Examples

Install

Configuring VSCode

Dependencies

Generation server-side entity and type APIs

Import modules from the engine (KBEngine and Math)

Add API to entities

Diagram of generated classes

Generation of types from types.xml

Argument names and method documentation

Development Tools

Notes about Entity Component API

Notes reading converters from user_type

Notes about generating types from types.xml

Examples

Examples (using code kbengine_demos_assets):

Entity remote method signature hint, based on Account.def

Entity property hint, based on Account.def

IDE on the generated interface IBaseAccount suggests the name of the property and its type

Determining the property type Account.characters (AvatarInfosList)

In this case, the property type is the type returned by the converter (TAvatarInfosList)

The type of the Account.characters property is AVATAR_INFOS_LIST. B types.xml is written, then AVATAR_INFOS_LIST is FIXED_DICT, with the converter AVATAR_INFOS.AVATAR_INFOS_LIST_PICKLER connected

	<AVATAR_INFOS_LIST>	FIXED_DICT
		<implementedBy>AVATAR_INFOS.AVATAR_INFOS_LIST_PICKLER</implementedBy>
		<Properties>
			<values>
				<Type>	ARRAY <of> AVATAR_INFOS </of>	</Type>
			</values>
		</Properties>
	</AVATAR_INFOS_LIST>

The code generator understands that a converter is connected to FIXED_DICT. But in order for the generator to understand what type the converter returns, you need to add a type annotation to the AVATAR_INFOS_LIST_PICKLER.createObjFromDict method

Connecting an API to an interface

The API for entity interfaces (scripts/cell/interfaces) is generated in the assetsapi.interfaces package. There will be a separate module for each interface, in this module there will be API classes for inheritance. The API parent classes for entity interfaces (scripts/cell/interfaces) already inherit the entity API (KBEngine.Entity), so the entity API hints will be immediately present.

Entity API

Types

KBEngine API

See also Demo based example where all entities, interfaces, components have connected with interfaces/APIs.

Install

REPOS_DIR=<YOUR_REPOS_DIR>
cd $REPOS_DIR
git clone git@github.com:ve-i-uj/enki.git
cd enki
python3 -m pip install --user pipenv
pipenv install
pipenv shell

Configuring VSCode

Below is an example of a workspace settings file for VSCode to work with "assets". The sequence to save the file in VSCode is: "Open Folder" --> "Sava Workspace As" --> Copy the config content to the workspace file

assets/.vscode/kbengine_demos_assets.code-workspace

{
    "folders": [
        {
            "path": ".."
        }
    ],
    "settings": {
        "python.analysis.extraPaths": [
            "${workspaceFolder}/scripts/user_type",
            "${workspaceFolder}/scripts/server_common",
            "${workspaceFolder}/scripts/common",
            "${workspaceFolder}/scripts/data",
        ],
        "files.associations": {
            "*.def": "xml"
        },
        "files.exclude": {
            "**/.git": true,
            "**/.svn": true,
            "**/.hg": true,
            "**/CVS": true,
            "**/.DS_Store": true,
            "**/__pycache__": true
        },
        "python.languageServer": "Pylance",
        "python.analysis.exclude": [
        ],
        "python.analysis.ignore": [
        ],
    },
}

Dependencies

The generated API requires the typing-extensions Python library in the python site-packages of "assets". When the engine will run server-side Python scripts, this library should be there.

There are two solutions here. The first [quick] solution is 1) just copy the library from this project (compatibility not guaranteed). You can copy it manually or add the environment variable ADD_TYPING_EXTENSIONS_LIB=true when generating the API.

Or the second [long] solution is 2) install the library via pip for Python of the same version as KBEngine and under the OS running the KBEngine server (needs Docker installed). The instruction is here.

Generation server-side entity and type APIs

To generate API / interfaces of server entities, you first need to generate the engine API. You must specify the path to the assets folder. The code will be generated in the server_common folder. Note that in this case, the ADD_TYPING_EXTENSIONS_LIB=true environment variable is also added. If the typing_extensions.py library are already added then simply remove this variable.

cd enki
pipenv install
pipenv shell
GAME_ASSETS_DIR=/tmp/kbengine_demos_assets \
    ONLY_KBENGINE_API=true \
    ADD_TYPING_EXTENSIONS_LIB=true \
    python tools/assetsapi/main.py

The server_common/assetsapi package now has the KBEngine API code. You need to import the engine libraries from the generated assetsapi package to specify which API component (base/cell) it uses.

from assetsapi.kbeapi.baseapp import KBEngine

When KBEngine will use "scripts" code then the KBEngine module will be the engine module in the statement above. When the "scripts" code is read by IDE then the KBEngine module will be the api from assetsapi.

If there is an import KBEngine statement in the scripts/user_type modules, it must be replaced with from assetsapi.kbeapi.baseapp import KBEngine. This is necessary because the code generator during code generation reads the modules containing the converters (the scripts/user_type folder) and generates entity methods with type parameters that the converters return (if the converters are annotated with types). Then scripts/user_type reading import KBEngine imports are not recognized.

In the case of kbengine_demos_assets simply remove import KBEngine in the AVATAR_INFOS.py module and AVATAR_DATA.py module (because it is not used). In the KBEDebug.py module, replace import KBEngine with from assetsapi.kbeapi.baseapp import KBEngine.

After that you need to run the command to generate entities api.

GAME_ASSETS_DIR=/tmp/kbengine_demos_assets python tools/assetsapi/main.py

An API for entities must be generated. Now we need to connect it with entities.

KBEngine module import for cellapp and baseapp

Imports for baseapp and cellapp differ in the name of the last module

# For `base` entity component
from assetsapi.kbeapi.baseapp import KBEngine
# For `cell` entity component
from assetsapi.kbeapi.cellapp import KBEngine

but for folders like scripts/user_type, common/user_type or server_common/user_type any of these imports can be used. Because part of the API of the KBEngine module is common for both components, and in the runtime under the hood of the assetsapi package, the KBEngine module will be imported from the engine via import KBEngine (i.e. immediately for the required component).

Import modules from the engine (KBEngine and Math)

Engine modules need to be imported from the package scripts/server_common/assetsapi

from assetsapi.kbeapi.baseapp import KBEngine
from assetsapi.kbeapi.Math import Vector3

Add API to entities

The API will be generated for every entity, which is located in the scripts/server_common/assetsapi/entity folder. For example, for the Avatar entity, the API would be located in scripts/server_common/assetsapi/entity/avatar.py. An API is an interface that needs to be inherited like the first parent class.

The example shows that

1. The workspace settings for VSCode was saved so code navigation works ("Save Workspace As ..." and then copy the example setting file)
2. The "typing_extensions.py" library was added according to instructions above.
3. Entity interface code has been generated.
4. We modified the scripts/base/Avatar.py file.
5. It was deleted import KBEngine and added import KBEngine from the scripts/server_common/assetsapi package instead.
6. Next, the first parent class inherited IBaseAvatar. The IBaseAvatar class was imported from the assetsapi.entity.avatar module (point 5).
7. Then you can typing code and Pylance suggests what attributes the instance has, what remote methods, and also remote method signatures (point 8). VSCode (+ Pylance) displays methods, entity properties from configuration files (scripts/entity_defs), KBEngine module API. Pylance also adds type checking to the methods it uses.

Note that KBEngine.Proxy is inherited last (as is KBEngine.Entity in other modules). This is necessary for multiple inheritance to work properly. Otherwise, you can get an error about the impossibility to carry out MRO. Entity classes can inherit entity-interfaces from scripts/entity_defs/interfaces.

Diagram of generated classes

The diagram is based on the Avatar entity. The specified classes will either be generated based on the Avatar.def file (IBaseAvatar) or will be in the assetsapi package (eg KBEngine.Proxy).

When running code from the KBEngine engine, the generated classes will have an empty body, so they will not conflict with properties and methods from the engine.

Generation of types from types.xml

This tool also generates data types that are used in entity remote methods. Data types are generated based on types.xml types. Generated types are used in the description of method signatures, this is especially true when using FIXED_DICT as a parameter. The generated types are located in the scripts/server_common/assetsapi/typesxml.py module, they can be imported from this module and used in entity methods.

An example of using a generated type

If a converter is connected to FIXED_DICT and the converter has method annotations, then the generated methods will use the type returned by the converter.

Example of adding the type returned by the converter

Add return type annotations (file scripts/user_type/AVATAR_INFOS.py)

Regenerate the API and see the required type in the method

For converters FIXED_DICTs are generated separately (in the assetsapi.user_type package). The generated classes for FIXED_DICT have the TypedDict type. The classes contain information about the keys that can be used in them. So Pylance (type checker) can indicate situations when unspecified keys are used in the dictionary. The name of a FIXED_DICT will be the same as in types.xml, only in CamelCase and with "FD" suffix.

If third-party libraries are used in user_type, then they must be added via the SITE_PACKAGES_DIR variable. This should be the folder of the python libraries.

Example

...

class AvatarInfosFD(TypedDict):
     """AVATAR_INFOS (<file:///./../../../scripts/entity_defs/types.xml#43>)"""
     dbid: Dbid
     name: str
     roleType: int
     level: int
     data: AvatarData

...

Below is an example where the FIXED_DICT AVATAR_INFOS has a converter. The converter returns the custom type TAvatarInfos. The example below shows an attempt to add a key that is not in the description. Pylance in this case indicates an error - this is convenient and helps to catch errors at the development stage.

An example of using a generated type

Argument names and method documentation

The generator can give names to arguments and add to the documentation for the generated method. To do this, when describing a remote entity method, you need to add xml comments as follows

<root>

    <!-- The entity class documentation -->

    <Properties>
    </Properties>

    <BaseMethods>
    </BaseMethods>

    <CellMethods>
    </CellMethods>

    <ClientMethods>
        <!-- The method documentation -->
        <resp_get_avatars>
            <Arg> AVATAR_INFOS </Arg> <!-- parameter_name -->
        </resp_get_avatars>
    </ClientMethods>
</root>

Get generated code with documentation

Generation of the parameter name by comment can be disabled by setting the variable USE_DEF_COMMENTS_LIKE_PARAMS=false. You may need to disable the formation of parameters based on comments, for example, if comments already exist.

Development Tools

Along with assetsapi you can add development tools to server_common. Tools can be added by adding the ADD_ASSETSTOOLS=true environment variable when generating code. In this case, the scripts/server_common/assetstools folder will be created, which will contain the following auxiliary tools.

A description of these tools can be found in the repository of the server script demo I updated.

Notes about Entity Component API

Entity Component API

N.B.: The Demo demonstrates that you can create a RemoteCall to the entity component by first accessing the owner (i.e. the entity) and then accessing the property of the entity (which is the component) and calling the remote method. And this is all done from the body of the component class.

The example in the Demo has a large potential error, which is also confusing when understanding the components API. In theory, the same component class can be used by different entities, and the name of the property that refers to the component can vary from entity to entity. For example, if you add a component with types "Test" of the Account entity, but add it under the name component123, then the code from the demo will stop working. It won't work, because when you call the Test.onAttached method on a component bound to an Account named component123, the owner (Account) will not have the component1 property. Conclusion: it is easier and more obvious to make a remote call directly from the component body itself, without accessing to the entity.

Bad example from Demo:

class Test(KBEngine.EntityComponent):

     def onAttached(self, owner):
         INFO_MSG("Test::onAttached(): owner=%i" % (owner.id))
         self.owner.client.component1.helloCB(111)

Same thing, but more obvious:

class Test(KBEngine.EntityComponent):

     def onAttached(self, owner):
         INFO_MSG("Test::onAttached(): owner=%i" % (owner.id))
         self.client.helloCB(111)

However, if the component is created for a specific entity, then the API of the entity can be hinted in this way:

But it is strongly recommended to connect the API for components without binding them to a specific entity at the code level. The component-to-entity relationship is a one-to-many relationship, not a one-to-one relationship. An example of connecting an API, without being tied to a specific entity:

Notes reading converters from user_type

Reading converters from user_type

assetsapi.user_type is a slightly modified copy of typesxml.py. This package duplicates the typesxml.py module, but with the difference that all FIXED_DICT with converters are replaced here with simple FIXED_DICT (with FD suffix in the name). This is done because the typesxml.py module imports a custom type from the converter module (the custom type is the type that the converter converts FIXED_DICT to). But the custom type module itself uses the FIXED_DICTs generated by typesxml.py. If we import the generated types from typesxml.py into the converter module, we get a cyclic import (an error). To get around this, I generate assetsapi.user_type which is a slightly modified typesxml.py module. The assetsapi.user_type package should only be used for imports into modules in the user_type directory. The purpose of assetsapi.user_type is to allow the FIXED_DICT specification to be specified for scripts/user_type/modules. These generated types are needed only to specify the types used by the converters in the user_type folder. Types from the assetsapi.user_type package should not be used in entity methods, there is a module typesxml.py for this.

Notes about generating types from types.xml

Notes about generating types from types.xml

Collection types that create other collections within themselves on the fly will not be described in detail. For example

<ARRAY_OF_ARRAY> ARRAY <of> ARRAY <of> AVATAR_INFO </of> </of> </ARRAY_OF_ARRAY>

will be generated into a type like ArrayOfArray = List[Array] (the nested type in this case is just an array, not an array containing AVATAR_INFO). If you need a more detailed description of the type, then it is recommended to use aliases. For example

<AVATAR_INFOS> ARRAY <of> AVATAR_INFO </of> </AVATAR_INFOS>
<ARRAY_OF_AVATAR_INFOS> ARRAY <of> AVATAR_INFOS </of> </ARRAY_OF_AVATAR_INFOS>

then ARRAY_OF_AVATAR_INFOS will be generated into a view type

AvatarInfos = List[AvatarInfo]
ArrayOfAvatarInfos = List[AvatarInfos]

In this case, nested types will also be specified, which is much clearer and easier for further maintenance. Plus gives the chance to do type checks.

If the converter does not have a return type, then FIXED_DICT with this converter in the generated code will have the type Any.

[Wandar]

Thank you for sharing<3