Encapsulate the TypeScript parser into a separate lib

[IEvangelist]

Create a new project named TypeScript.SyntaxTreeParser and manually convert the TypeScript parser.ts bits into a C# TypeScript parser (from the compiler itself).

• parser.ts
• types.ts

This library will be capable of representing TypeScript type declarations as an abstract syntax tree.

[volkanceylan]

Hello, today i came across your project while searching GitHub for C# to Javascript converters that use Roslyn. I used Script# and Saltaralle for a long period in the past before switching to TypeScript. Now thinking about writing some source generator converting C# to Javascript similar to others but without the .NET corelib/runtime part, e.g. it will have a custom stdlib and almost one to one conversion from C# to JS (of course by only supporting a subset of .NET). It's just at idea phase for now.

Anway that's not why i'm writing this. I saw this issue which indicates you are looking to convert TypeScript parser to C#. I was also looking to do that months ago but it seemed like a big task, then i found this:

https://github.com/ToCSharp/TypeScriptAST

and used a optimized / updated version of it successfully in my Serenity code generator (below) / source generator (that part is a private repo atm) which parses TypeScript files (not just .d.ts) and extract some metadata like user defined types etc.

https://github.com/serenity-is/Serenity/tree/master/src/Serenity.Net.CodeGenerator/TypeScript

TypeScriptAST was written a few years ago, so it might not be up to date with current TypeScript compiler but it can be a starting point for you if you are really considering this task.

[trungnt2910]

https://github.com/ToCSharp/TypeScriptAST

This is an outdated library and has problems with some newer features when parsing the latest versions of lib.dom.d.ts.

I myself am currently using this library to parse lib.dom.d.ts in a source generator for my project (https://github.com/trungnt2910/DotnetBrowser/tree/master/src/Trungnt2910.Browser.Generators.TypeScript) that generates C# class bindings, but I'm looking for ways to replace it.

As the TypeScript compiler is rapidly evolving, it might become very hard for a port like this to keep track of upstream changes. It might be more desirable to write a TypeScript analyzer in TypeScript using the official compiler code, and then run it in a Roslyn analyzer using a JavaScript runtime for .NET like jint.

Generating an abstract syntax tree for TypeScript declarations is only one small part of the problem. The other part is analyzing the semantics of the declaration. While Roslyn provides a SemanticModel API, TypeScript doesn't seem to provide any (or at least, provide any well-documented public APIs). Therefore, it would be really difficult to resolve all the type aliases, special types like Union types, and so on. Depending on what you're doing you might also have to decide what to generate as classes visible to C# and what to be generated as just interfaces, because in lib.dom.d.ts everything is an interface and an interface can extend multiple interfaces while C# classes cannot.

[trungnt2910]

If this project is still active, I wish to collaborate on this issue, at least for the front-end part that analyzes TypeScript code for .d.ts files.

For the backend I would need to tweak a bit for it to suit my JavaScript interop strategy (using JSImport and object handles instead of using string and the Blazor-specific IJSInProcessRuntime).

[IEvangelist]

Hi @trungnt2910 - yes, very much active. I have a branch where I started work on the TypeScript AST, see https://github.com/IEvangelist/blazorators/tree/ast. This was a huge effort, and it's still very much up in the air. I love collaborating with others, happy for the help. I've been planning on updating this to use the newer JSImport bits, but I know that you can not use the output from one source generator as input to another. JSImport is a source generator attribute, so we cannot use it.

[trungnt2910]

My project manages to use JSImport because its JS interop architecture is fundamentally different, but that's out of scope for this issue.

I am curious how you managed to create those hundreds of classes. Were they generated somehow or were they hand-written?

Also is the intended work limited to porting parser.ts only or will it one day extend to other APIs like binder.ts, which is responsible for symbol resolution?

Finally, please tell me if you know some documentation/guides on consuming the TypeScript compiler API from an outside program. I can find things like this but they mostly document the compiler internals and not how to consume those APIs like how one can use Roslyn.

[IEvangelist]

I am curious how you managed to create those hundreds of classes. Were they generated somehow or were they hand-written?

Which classes specifically are you referring to? If you're talking about the AST branch, then yes, those are hand-written.

Also is the intended work limited to porting parser.ts only or will it one day extend to other APIs like binder.ts, which is responsible for symbol resolution?

The intended work is to make a TypeScript type declaration parser capable of parsing raw .d.ts files and using this metadata to drive my C# source generator. I would define success as being able to source generate arbitrary files, outside the lib.dom.d.ts (which by the way no longer exists), and using that to generate types. As of right now, my project is already capable of source-generating services (interface and class C#), POCOs, and dependency hierarchies.

[volkanceylan]

FYI, I just converted TypeScript Scanner and Parser from latest master of TypeScript repository:

https://github.com/serenity-is/Serenity/tree/master/src/Serenity.Net.CodeGenerator/TypeScript

It produces same set of tokens and nodes for about 6k compiler test cases in TS repository:

https://github.com/serenity-is/Serenity/blob/master/tests/Serenity.Net.CodeGenerator.Tests/TypeScriptTests/TypeScriptTests.TestCases.cs

Above test contains the code I used to generate test cases.

[trungnt2910]

Cool, how do you plan to keep it up with the ever-evolving TypeScript language?

[volkanceylan]

Well, the TypeScriptAST I used before worked well enough even though it was a conversion 7 years ago. The lexer and parser does not actually change that often. And they are relatively small compared to type checker, transformer parts etc
Anyway, I tried to do conversion keeping names and order of methods intact so I will try to identify changes using Git diffs and semi automatic merging. During conversion, I wrote a simple TypeScript to CSharp converter of 500 lines or so just to make it easier to convert/fix issues later. I will probably update it once or twice a year or so.

[trungnt2910]

While it doesn't change, the language keeps adding new and new features, which may (or hopefully may not) be used in core definition files like lib.dom.d.ts (or whatever file it got replaced with).

By the way, do you intend to make a NuGet package out of it?

[volkanceylan]

Yes, I will soon.

[volkanceylan]

Here is the NuGet package:

https://www.nuget.org/packages/Serenity.TypeScript

And the new source link:

https://github.com/serenity-is/Serenity/tree/master/src/Serenity.TypeScript