Entitas with Roslyn Code Generation via dotnet IIncrementalGenerator #1005
Comments
|
@studentutu thanks! Roslyn Source Generators is definitely sth I would like to research at some point! That might be useful for Entitas. Jenny however follows a different idea where the input can be anything, not just source code or assemblies. It more general purpose for any kind of code generation |
|
One downside is
But I'll have a look |
|
Looks like we would need to switch to Unity when we make changes in order to recompile, which would take too long. |
|
Ok, building in the IDE works too 👍 |
|
I've written a few Source Generators. My note would be to look into Incremental Source Generators if you decide to go this route. The perf of a normal ISourceGenerator isn't that great. Unity 2022.2 supports the required newer Microsoft packages for that. They just haven't fixed their documentation yet. |
|
yep, definitely better with |
|
One thing of note though: I'm having severe problems with The Unity docs on this are kind of sparse - and the one page that does mention this describes an implementation that: So if that is something that one wants to support I'd wait for Unity to actually switch to permanent, modern csproj files and using MsBuild, as outlined in their roadmap talk. Although it is a bit bold to assume that this will just work then. They usually manage to do stuff that is not in line with what .NET devs are used to ;). |
|
Hi, I started testing https://github.com/sschmid/Entitas/tree/roslyn-source-generators/src/Entitas.Generators https://github.com/sschmid/Entitas/tree/roslyn-source-generators/tests/Entitas.Generators.Tests So far I'am happy with |
|
If all goes well I can imagine that it can replace Jenny and setting up Entitas will be much easier. I started with snapshot testing to verify the output of the source generators, it's pretty cool. A test looks like this [Fact]
public Task Test() => TestHelper.Verify(
GetFixture("NamespacedComponentWithOneField"),
new ComponentGenerator());More about snapshot testing: |
|
I made some progress using |
|
Ok, got it to work! Unity docs say you should use this specific version: This version does not contain The current version is 4.6.0, but that one does not work in Unity.
And you can use netstandard2.1 |
|
Currently stuck, because it seems like you cannot resolve generated attributes, e.g. [MyApp.Main.Context, Other.Context]
partial MovableComponent : IComponent { }But since those attributes are generated, the don't seem to be part of the compilation for looking up symbols 😭 This was easily possible with Jenny... Any ideas how to solve this? |
|
To be more specific: var attribute = symbol.GetAttributes().First();With the generated ones the same code returns |
|
@sschmid you can use following
And for the actual game /Runtime .csproj: |
|
afaik step 3. and 4. are already part of the current setup. Can you provide documentation for I don't see how this makes generated code of other incremental generators available to the input compilation of this generator. Can you explain how your approach solves/bypasses this? |
|
@ants Aare Basically, you begin with a normal Generator ( Then in the main Generator, you simply execute each custom one by providing So by using a predefined ordering, you will get a correct compilation of source generators. |
|
By the way, if you will use |
|
Maybe I'm understanding it wrong, but I don't think this solves the problem(?) What you're describing is just a way to call multiple methods one after each other through an interface inside a regular sourcegenerator. First of all this doesn't include recompilation steps inbetween the calls to |
|
right, missed the part of the "generated source code with new attributes". |
Yes, this is part of the design of source generators. This avoids having to have multiple runs and allows the caching mechanisms that make them so efficient. For my ECS approach I'm not using generated attributes because of this. The user defines partial classes for the contexts and components implement |
|
Hi, a quick update and some thoughts: I have a working proof of concept using ContextsNow with namespace support! You can define contexts in your code like this: // MainContext.cs
namespace MyApp
{
partial class MainContext : Entitas.IContext { }
}
// OtherContext.cs
partial class OtherContext : Entitas.IContext { }ComponentsNow with namespace support! You can define components in your code like this: // MovableComponent.cs
namespace MyFeature
{
[MyApp.Main.Context, Other.Context]
partial class MovableComponent : Entitas.IComponent { }
}
// PositionComponent.cs
namespace MyFeature
{
[MyApp.Main.Context, Other.Context]
partial class PositionComponent : Entitas.IComponent
{
public int X;
public int Y;
}
}The generated component extensions work for all specified contexts and can be chained: MainContext mainContext = new MyApp.MainContext();
MyApp.Main.Entity mainEntity = mainContext.CreateEntity()
.AddMovable()
.ReplaceMovable()
.RemoveMovable()
.AddPosition(1, 2)
.ReplacePosition(3, 4)
.RemovePosition();
OtherContext otherContext = new OtherContext();
Other.Entity otherEntity = otherContext.CreateEntity()
.AddMovable()
.ReplaceMovable()
.RemoveMovable()
.AddPosition(1, 2)
.ReplacePosition(3, 4)
.RemovePosition();MatchersI currently generate component indexes for each context, e.g. Matcher.AllOf(stackalloc[]
{
MyFeaturePositionComponentIndex.Value,
MyFeatureMovableComponentIndex.Value
}); |
|
More updates: var (x, y) = entity.GetPosition();
x.Should().Be(1);
y.Should().Be(2);Also, since the only purpose of |
|
Fyi, for those who are interested about the changes, see branch: |
|
More updates: The new approach should work with multiple assemblies per solution. At some point however, you would need to assign an index to each component. With the following idea you can build up your game with multiple separate assemblies and the main project that consumes them can implement a partial method per context and add the public static partial class ContextInitialization
{
[MyApp.Main.ContextInitialization]
public static partial void Initialize();
}This will be picked up by the generator and it will generate everything that used to be in namespace Entitas.Generators.IntegrationTests
{
public static partial class ContextInitialization
{
public static partial void Initialize()
{
MyFeatureMovableComponentIndex.Index = new ComponentIndex(0);
MyFeaturePositionComponentIndex.Index = new ComponentIndex(1);
MyApp.MainContext.ComponentNames = new string[]
{
"MyFeature.Movable",
"MyFeature.Position"
};
MyApp.MainContext.ComponentTypes = new System.Type[]
{
typeof(MyFeature.MovableComponent),
typeof(MyFeature.PositionComponent)
};
}
}
} |
|
More updates: I can recommend this cookbook for incremental generators: Next problem: if something in the generator fails, nothing will be generated. This can easily be reproduced by declaring the same component twice which will break everything. I tried to wrap all Does anyone know how to make handle exceptions in a source generator? |
|
@sschmid you can use tests, and check snapshots? https://andrewlock.net/creating-a-source-generator-part-2-testing-an-incremental-generator-with-snapshot-testing/ Otherwise - you can even use simpler diagnostic as a log inside the compilation |
|
As @studentutu hinted it's probably a good idea to emit diagnostics for these problems if you run into generation failures. On top of that, you can also be proactive and write additional implementations of In my ECS prototype I used those to validate Queries (so impossible queries produce errors). |
|
Btw, I'd like to take a moment an thank everyone for your suggestions and help! It's a lot of fun to work on Entitas like that again! 🫶 Together we can make it 💪 |
|
I never used source generator in big projects. Do they scale well with the codebase size? I just did a simple test which confirms predicates and transforms are ran with every keystroke, no matter if the changes are done in a related or unrelated file, even if it's just whitespace such as spaces or newlines. initContext.SyntaxProvider.CreateSyntaxProvider(
Predicate,
Transform
);The only caching optimization seems to happen when the transform method produces the same result; then it skips the generation part. That sounds kind of scary to me, as a simple class itself has multiple child nodes, each triggering the predicate. So 1 class with a few fields and methods might trigger the predicate 10x. Scale that up to a massive codebase and 🤯 Does anyone have experience with larger codebases and source generators? |
|
Given that dotnet itself is increasingly using source generators, I wouldn't worry too much. Having early exits and a meta model that can easily be created, cached and compared in early stages of the source generator pipeline is absolutely relevant. And so is respecting cancellation requests. The discussion here is quite relevant I think. They explicitly mention it should be able to scale to repos the size of the CLR itself... I'm not sure if the generators are currently called in a multi-threaded way, but given their Single-Pass nature and conceptual order independence I wouldn't be surprised. |
|
@sschmid using source generators on the large project doesn't impact is that much, whole compilation lasts 1-2 minutes, as we are using a separate dll and a separate project outside of Unity. In unity we then simply use that dll in conjunction with Unity view layer logic. |
|
@rubenwe Yeah, I might worry too much :D @studentutu yeah, separat projects will help to improve the generator, but I was mostly worried about how busy the machine is during coding, not just compiling. Every keystroke will invoke the generator pipeline, and that always felt like a waste of resources to me. But I guess I need to learn to live with it :D |
|
Followup performance test result: |
|
Bummer: I tried the current state in older Unity LTS versions, only 2022.3 worked 😭 I also downgraded to |
|
I can share a little bit about the version drama based on my current testing: 1st version clash:
2st version clash:
So my current results:
🤷♂️ but fair enough |
Question:While updating the output of the generated code I was thinking about updating it to use Example: imagine we have a unique var user = context.GetUser();public static UserComponent GetUser(this MyApp.MainContext context)
{
var entity = context.GetUserEntity();
return entity != null ? entity.GetUser() : null;
}Using nullable, it would return public static UserComponent? GetUser(this MyApp.MainContext context)
{
return context.GetUserEntity()?.GetUser();
}Nullables would also allow me to remove throwing exceptions in the generated (e.g. when you try to get a component that has not been set) and move the responsibility to the consumer. Will nullable break your projects? |
When you get closer to completion or after it's in I can also take a look if I can help and find a few extra places to tweak - but honestly 29ms is probably good enough ;D
Please do that! This would allow us to use proper pattern expressions as filters in reactive systems and would simplify code in a lot of places! So something like this: protected override bool Filter(GameEntity entity) =>
entity.hasEquipmentChestType &&
entity.equipmentChestType.Value is ChestType.Small or ChestType.Big &&
entity.isInstantOpen;Could at least become something like this: protected override bool Filter(GameEntity entity) => entity is
{
equipmentChestType: { Value: ChestType.Small or ChestType.Big },
isInstantOpen: true
};And even better, for newer version of C#: protected override bool Filter(GameEntity entity) => entity is
{
equipmentChestType.Value: ChestType.Small or ChestType.Big,
isInstantOpen: true
}; |
|
@rubenwe omg that last code snippet feels fresh, I like it a lot. This entitas version is breaking anyways, so it makes sense to include multiple changes and the cost of implementing it right now is fairly low. |
|
More updates and infos on the route to source generators: In a simple setup with source generators and Unity, the source generator will be active in each asmdef. This is convenient because you can split up your game in multiple sub projects with asmdefs and all works as expected. Example:Split up the app into
namespace MyApp
{
partial class MainContext : Entitas.IContext { }
}The source generator will generate the rest of the classes, like the other part of the partial
using Entitas;
using Entitas.Generators.Attributes;
using MyApp;
namespace MyFeature
{
[Context(typeof(MainContext))]
public sealed class PositionComponent : IComponent
{
public int X;
public int Y;
}
}The source generator will generate extensions like However, there are also some files that have been generated by Jenny so far, that only exists once in the app, like the As described above, the source generator is in every asmdef, so it cannot generate the I actually moved the On another note, the namespace MyApp
{
partial class MainContext : Entitas.IContext
{
public static MainContext Instance
{
get => _instance ??= new MainContext();
set => _instance = value;
}
static MainContext _instance;
}
}and later use it like this var entity = MainContext.Instance.CreateEntity();You're in full control of your contexts now. Of course, you can also just recreate the The context.CreateContextObserver();#if ENTITAS_DISABLE_VISUAL_DEBUGGING || !UNITY_EDITOR
[System.Diagnostics.Conditional("false")]
#endif
public static void CreateContextObserver(this IContext context)
{
Object.DontDestroyOnLoad(new ContextObserver(context).gameObject);
}This call will be stipped out when compiling using the And finally, to follow up on previous thoughts on nullables and other changes, I will try to keep the new generated code as similar to the current version as possible to not break too much (see #1069 (comment)). |
|
I just finished event component and system generation, and finally fixed a subtle bug, where listener entities might leak. Example of the old code: Basically, when unsubscribing from event the listener can be auto cleanup up when empty. But it only removed the component, leaving the entity without any components. Now the listener entity will be destroyed when there are no components left on the entity |
|
Hello! Does anyone know how to set and use global options for generators (or analyzers)? I described my previous solution here: #1069 What I'm currently doing:
optionsProvider.GetOptions(syntaxTree).TryGetValue(key, out var value) QuestionWhy do I need to pass in a syntaxTree? optionsProvider.GlobalOptions.TryGetValue(key, out var value)It sounds like GlobalOptions would be the correct place to check? How can I add custom key-value pairs to GlobalOptions? Seems like Also, I don't always have a syntaxTree that I can pass in. I pass one in where ever I can, and it works, but e.g. when woking with the compilation like with |
|
I'm referring to this that I found in the docs: https://github.com/dotnet/roslyn/blob/main/docs/features/source-generators.cookbook.md#access-analyzer-config-properties
mygenerator_emit_logging = true[Generator]
public class MyGenerator : ISourceGenerator
{
public void Execute(GeneratorExecutionContext context)
{
// control logging via analyzerconfig
bool emitLogging = false;
if (context.AnalyzerConfigOptions.GlobalOptions.TryGetValue("mygenerator_emit_logging", out var emitLoggingSwitch))
{
emitLogging = emitLoggingSwitch.Equals("true", StringComparison.OrdinalIgnoreCase);
}
// add the source with or without logging...
}
public void Initialize(GeneratorInitializationContext context)
{
}
}Unfortunately, |
|
Alternatively, this seems to work: Add msbuild props to |
|
... not in Unity of course 🙈 |
|
One benefit you get with keeping the syntaxTree is that you could potentially configure the generator per component. Not sure if that's ever useful, but you can do it:
|
|
Yay, all generators are complete. I updated my local MatchOne repo to give it a test run. Works great! I will share more soon! |
|
Completed all tasks from above. Will close. |
|
Update:
https://github.com/sschmid/Match-One/tree/entitas-2.0-beta Development is not complete, things may change, but it's the first working version that I wanted to share to get feedback. Once Entitas 2.0 gets closer to release I will share more docs and upgrade guides. For now Match-One is the only "documentation" in form of a working project. Try it, break it and have fun :D |
[ EDIT by @sschmid ]
dotnet's source generators, specially
IIncrementalGeneratormay be a valid alternative to the current approach with JennyLearn about dotnet Incremental Generators:
While migrating to dotnet source generators, I will use the opportunity to update and improve the generated code:
I already made good progress but it's still in research state. Any help from the community is greatly appreciated! Please feel free to engage and help in the conversation if you can! 🙏
Tasks
Generators
OBSOLETEComponent EntityApiInterface Generator (Multiple Contexts)OBSOLETEComponent GeneratorOBSOLETEContexts GeneratorOBSOLETEContextObserver GeneratorOBSOLETEFeatureClass GeneratorAttributes
OBSOLETECustomEntityIndexAttributeOBSOLETEEntityIndexGetMethodAttributeOBSOLETEComponentNameAttributeOBSOLETEDontGenerateAttributeOBSOLETEFlagPrefixAttributeOBSOLETEPostConstructorAttributeOBSOLETEPrimaryEntityIndexAttributeoriginal message from issue author:
Hi,
I have a suggestion and would want to contribute.
As in official code generation guide by Unity - https://docs.unity3d.com/Manual/roslyn-analyzers.html - we can use separate project and create dll which unity will use right in it's code compilation steps.
We already have a separate project for it - Jehny, all we need is to make sure that code gen is done by Microsoft Roslyn Source Generators, and put a label “RoslynAnalyzer” for the DLL inside the release branch (create a Unity package)
This way we don't need to use Jehny Server for constant monitoring of changes, and it will help clean up the workflow.
Here's some code that we need to use for Roslyn Source Generators
Hope it will be helpful.
The text was updated successfully, but these errors were encountered: