[Feature Request] Porting to .NET Core

[ankitbko]

Since .NET Core is now in RTM, any plans of porting BotBuilder SDK to Core? I do not see any platform specific references in SDK that could prevent it from running cross platform. I believe the biggest change would be moving away from System.Runtime.Serialization and to json.net(?). Any thoughts?

[willportnoy]

The lack of BinaryFormatter could create some challenges. https://github.com/dotnet/corefx/issues/6564 has some discussions of the issues. I think the biggest challenge may be finding some substitute for the StoreInstanceByType serialization surrogate:

https://github.com/Microsoft/BotBuilder/blob/master/CSharp/Library/Fibers/Serialization.cs#L89

that allows resolution of singleton instances from the container if they're marked with DoNotSerialize:

https://github.com/Microsoft/BotBuilder/blob/master/CSharp/Library/Fibers/FiberModule.cs#L61

Of course, I'm in favor of more tightly constrained dependencies if possible, and moving to .net core seems like "a good idea" other than this serialization issue.

[htaunay]

This would be an excellent feature!
Bot building would fit very well within .Net Core's minimalist paradigm, would definitely chose over node.js
+1

[berhir]

@willportnoy While porting to .net core would be tricky, support for ASP.NET Core on full .net 4.6 could be done very easily.
We already use the Bot Builder/Bot Connector with ASP.NET Core for our bots but we had to use some workarounds to get the Bot Connector working. Would be nice if the Bot Connector could support the new configuration model for BotId and AppSecret and the new controller attributes for [BotAuthentication()].

I know there is already a sample for ASP.NET Core but it doesn't address these problems very well.

[willportnoy]

@berhir Yes, I agree, we could better support ASP.NET Core.

I assume the BotAuthentication attribute would be replaced with some piece of [authentication] middleware in the new model.

Some of the auth configuration for IConnectorClient could flow through the Autofac container registrations.

[murilocurti]

@berhir can you share how you are doing this? Thanks!!!

[v-thgratMSGit]

Closing this issue. Thanks!

[murilocurti]

Why this issue was closed @v-thgratMSGit ? Thanks

[willportnoy]

We've been closing some of the older issues that have been without recent activity. In this case, it's not clear how to port the Bot Builder SDK to .Net Core without the presence of BinaryFormatter, on which we have a somewhat strong dependency (in particular for the serialization surrogates).

[murilocurti]

Ok, tks

[georgiosd]

@willportnoy Sure, but supporting ASP.NET Core even with the full 4.6 SDK is still relevant - shall I open a new ticket about it?

I see some that GraphBot sample is using ASP.NET Core but I don't know if it's fully working?

[DamianReeves]

I agree with @berhir, if we could get an ASP.NET CORE sample targeting .NET 4.6 (full framework) would be great. In particular it would be great to get a drop in replacement for [BotAuthentication()].

[georgiosd]

Hey folks, any plans on this?

[willportnoy]

No specific plans, right now - @georgiosd what form of .net core support are you looking for? asp.net core authentication attribute or the full builder in .net core (harder due to serialization requirements)?

[georgiosd]

The authentication is the first thing I suppose but then what? I'd have to host another internal service and delegate requests there? Kind of defeats the purpose right?

I'd be happy if it supported the ASP.NET Core API with the full 4.6 framework - basically we're starting all new projects in the Core API vs Web API 2 so adding a bot would mean keeping a project in WebAPI 2...

Does that make sense?

[willportnoy]

Yes @georgiosd - we have it filed as an internal backlog item.

[georgiosd]

Thanks @willportnoy - is there an ETA?

[nothingmn]

+1 ETA...

[Deilan]

Even ASP.NET Core support targeting Full .NET Framework 4.6 would be great. Looking forward to it.

[willportnoy]

No particular ETA, but I'll leave this issue open since it's pretty active.

[CXuesong]

@jefflill Just saw your repos, and that's... really a lot of code 🙈 And by hosting on Ubuntu, are you talking about something like Wine?

I tried BinaryFormatter yesterday. At least I am still on my way writing some "polyfill"s to get it work with BotFramework on .NET Core. But yes, there are some nasty things like I cannot, in any way without explicitly putting every property into SerializationInfo, serialize Regex and many, many other classes though they "implement" ISerializable, and I still have some trouble using ISerializationSurrogate to surrogate the serialization of RuntimeType. During de-serialization, BinaryFormatter tells me the object to be deserialized is some type of TypeLoadExceptionHolder, which is supposed to be RuntimeType!

Still, BinaryFormatter provides better support for surrogates than JsonSerializer. JsonConverter.CanConvert(Type type) has different semantics for type parameter from ISurrogateSelector.GetSurrogate(Type type, StreamingContext context, out ISurrogateSelector selector) . During deserialization, it means the declared value type of the property or field, rather than the actual type of the object instance to be deserialized, so I had to use some more nasty hacks(the $resolve thing, rather than $type) into JsonObjectCovnerter.

A little digression here

Recently I am working on some home-made sandbox and surprisingly found out Task cannot be remoted across AppDomain. I thought it should derive from MarshalByRefObject or something... Perhaps they are more or less still going to deprecate those AppDomains, (System.Runtime.Remoting) Remoting, Binary Serialization, etc. Interestingly, they have removed BinaryFormatter from "Unsupported Technologies" section in a later version, but in dotnet/corefx#19119, they observed

But it comes with some serious implications, and it's becoming clear we took things too far. Doing this makes it very easy for code to take dependencies on private implementation details and will end up binding our hands for improvements/optimizations/refactorings/etc. in the future.

Taking in to account the limited set of Exception-derived class that is marked as [Serializable], it's clear they brought back BinaryFormatter only for compatibility purpose, which I think is okay, because we have Newtonsoft.Json now, which is still better than DataContractSerializer.

To make a summary, the current situation is caused by

• It the Fiber they introduced the need for serialization of deleages, and BinaryFormatter is really a handy approach for that. (Though we can do the same with JsonConverter)
  • Then they decide to customize BinaryFormatter using surrogates to automatically restore the injected dependencies during deserialization. (Though we can do the same with some ObjectFactory, I mean, inject the services via constructors, and populates the properties)
• The freedom provided by BinaryFormatter makes it difficult to be replaced.
  • Users won't need to care the serialization details (e.g. [JsonConstructor]), as long as their classes are [Serializable].
  • There are many BCL types in .NET Framework marked as [Serializable] but cannot work with JsonSerializer (Though we can resort to JsonCovnerter)
  • It provides good support for polymorphism (JsonSerializer can emit $type tag, but as I have stated above, it does not pass $type into JsonConverter.CanConvert)
  • It provides good support for surrogates (e.g. IObjectReference; while on the other hand you cannot return an instance of different type in JsonConverter from $type)

In fact, in this situation, we prefer the polymorphism & reference (esp. for circular references) transparency rather than portability across the languages (even not across .NET on different platforms), and we want the serialized data independent from the implementation details (name of the private fields, etc.), which would amount to a mixture of BinaryFormatter and JsonSerializer/DataContractSerializer. That's my 2 cents.

[jefflill]

@CXuesong: Thoughtful post. You definitely know more about serialization than I do but I'm learning as I'm poking around this lower level stuff.

I've been working on enhancing Docker for maybe a year and a half with tools and services that will make it easy to deploy and run a Docker cluster with all (OK, a lot) of the stuff you'll need for a production environment (setup, remote commands, load balancing, logging,...) called neonCLUSTER. This is currently hosted on Ubuntu nodes and I'm running my applications natively using .NET CORE 2.0. I've got a couple months more work before I release neonCLUSTER, probably with an opensource/freemium model.

Eventually, I'll support mixed Docker clusters with both Ubuntu and Windows node but Windows hasn't been a priority for me and I'm hoping that this BotBuilder issue won't force me to do Windows prematurely.

I haven't actually written a real bot yet, but my sense is that I wouldn't really want to code bots to persist very much application state using the built-in BotBuilder state mechanism, especially if the data persisted is going to be an essentially opaque (to external code) byte array. Instead, I'd persist just enough information into the state so that I can explicitly query conversation state from my NoSQL (or whatever) database. This is for future proofing. Over the long-term, the likely hood that you need external processes to access this data and potentially modify this for one reason or another is very high, and I really don't want to make some poor developer to have to crack some crazy binary format some years hence.

My gut feeling (and I could easily be wrong) is that BotBuilder serialization is somewhat over-engineered. I would looked into designing this such that it simply used JSON with concrete types. Internally, BotBuilder would have had their own types that explicitly serialize conversation internal state include the delegates). Then, I would have tried to have the bot developer (somehow) explicitly specify the concrete app state type that itself was implemented only with concrete types. I suspect that for many (most?) applications the app state is going to be just a handful of fields, not a complex object graph. So, simple JSON.NET serialization would just work. Applications with complex graphs would have to serialize these themselves one way or another (and an easy way would be to define the bot state type to include a byte array and then explicitly use BinaryFormatter to serialize the graph into that array.

I decided yesterday morning to have one more go at creating a munged BinaryFormatter called NeonBinaryFormatter. The approach now is to introduce the concept of built-in surrogates for types like System.RuntimeType and Delegate. These surrogates will be implicitly available for backwards compatibility. I believe I have System.RuntimeType and I'm working on Delegate now (hopefully the last problem type required for basic bots). If this works, I'll release NeonBinaryFormatter as an open source Nuget package that allows folks to add additional built-in surrogates.

Your historical comments on .NET Core are interesting. I believe you are probably right about MSFT long-term intentions. I found it interesting that they didn't disable serialization for all of these type in .NET Core 2.0 until just this May (~3 months ago). That was probably very smart move; you don't want to release a shiny new platform that builds in a lot of potential future constraints.

If it had been up to me though, I would have tried to add a BinaryFormatter(bool unsafe = false) constructor (or something) that enabled the old behavior, explicitly telling developers that serialized types may not work cross platform or even across different releases of the same platform. I can see MSFT prioritizing this though. Shipping is a feature, as they say.

I'll let you know if I get this working.

[CXuesong]

Finally! I got the example projects compile with my ported library. You may check out this (could-be-rather-unstable) prerelease. See the notes here. You can also update the packages via NuGet with "-Pre" turned on.

[CXuesong]

@jefflill I am still learning about Binary Serialization, and all the Linux stuff. Oh plus Docker now 🙈 You are doing awesome work!

I intended to write a bot but running on Ubuntu. That is when I found out BotBuilder SDK only runs on .NET Framework. Haven't plunged into the Fiber stuff (if I have time, I will try to comprehend it in the future). But I guess if they separate the states from methods, just like MVC controller, the life would be great. Then it will be possible for JsonSerializer to store only the fields, by value, and then we can make some assumption of "no polymorphism" and "no cyclic references" safely. Under current situation, BinaryFormatter introduced too much freedom, and we just cannot make any assumptions… Yes I believe except from the methods, the Dialog would be a trivial combination of fields. Nobody need to store Exception instance in their state, after all.

Yes, we need to support serializing delegates first, and this amounts to the serialization of Type, and MethodInfo. We also need to serialize Regex because its used everywhere.

As for over-engineering… I'm not sure but perhaps we can do nothing more than migrating… Until someone decides to rework it 🙈

[jefflill]

@CXuesong: I played around a bit with your BotBuilder port this morning and so far so good: IT WORKS! Very cool. I've abandoned my efforts and pointed folks to your repo and Nuget packages. Great work!

It sure would be nice if MSFT were to take a pull request from you and make this official. It's a bit strange that they've been so quiet about this issue, especially with .NET Core 2.0 being their big shiny new thing.

If I were you, I'd submit a pull request just to see their response, and who knows, perhaps they'd accept it and you'd be famous :)

[CXuesong]

@jefflill Oh I'd love to be famous XD After all, who wouldn't?

Though it may take a while to prepare a PR :-P

[AmirSasson]

@CXuesong, anxiously waiting for your PR to be approved. You would be famous and we will be happy 😊!
Started to have doubts in our desicion to go with C#..
Any progress with that?

[rizamarhaban]

Btw, if anyone wants to try creating chat bot using ASP.NET Core 2.0 (.NET Core 2), I make some simple working sample and blog it. Source code also available and currently I'm adding the project template as BotApplicationNetCore2 for VS2017 15.5 Preview 5.

Check it out: https://www.rizamarhaban.com/2017/11/27/creating-chat-bot-using-asp-net-core-2-web-api-microsoft-bot-framework/

[akontsevich]

@rizamarhaban thanks for the sample! Is it possible to the same for VS Code in Linux, for example? What are the steps could be? Thanks!

[CXuesong]

@AmirSasson I can only say, if you want to consume Bot.Builder on .NET Core with C#, for now, you may try out my unofficial releases of BotBuilder.

I guess nodejs certainly has no compatibility issue for now…

[nwhitmont]

Good news everyone!

New Bot Connector support for ASP.NET Core 2.0 and 1.1

https://blog.botframework.com/2017/11/28/new-bot-connector-support-net-core-2-0/

[akontsevich]

Hurray!!! :)

[Geertvdc]

@nwhitmont yes the Connector is there now but the actual Bot.Builder package is still old and not supported by ASP.NET Core 2.0 will this be upgraded as well?

[JasonSowers]

@Geertvdc No Bot.Builder Will not support .Net Core. Our next version (v4) of the SDK will fully support it though. At this time we do not have hard dates for a release of it, but it should be in the not too distant future.

[Geertvdc]

is this V4 being build as open source? can we see it somewhere / contribute?

[JasonSowers]

@Geertvdc Right now we do not have information to release publicly please follow #3891. We will update that issue as we can.

[nrobert]

v4 with .Net Core is out, we may close this issue, don't we?

• dotnet: https://github.com/Microsoft/botbuilder-dotnet/wiki
• js: https://github.com/Microsoft/botbuilder-js/wiki

[CXuesong]

Agreed! Though it may take some time for the next generation SDK to stabilize.

Wait for it.

[oscarwest]

SWEET

[hitaspdotnet]

Update?

[nrobert]

See my answer below, it's already out for weeks, what do you mean by "update?"?

[Geertvdc]

@hitaspdotnet the new version of the API that supports .net core is located here: https://github.com/Microsoft/botbuilder-dotnet

they moved it to a separate repo

[hitaspdotnet]

@nrobert issue still open

[vishwacsena]

Bot Builder V4 SDK available in preview here will fully support dotnet core. There are no plans to add any additional support for dotnet core to the V3 SDK.