Project System: Please Consider POCO as Schema for Project File/Structure #37
Comments
|
From @bbarry on May 12, 2016 13:32
The xsds are in your msbuild bin folder: C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild It doesn't help much (the xs:documentation tags almost look like they are generated by some ghostdoc-like tool; for example is documented as "An MSBuild Project" in Microsoft.Build.Core.xsd), but they are there. There are a few useful comments in the file though. |
|
From @Mike-EEE on May 12, 2016 13:38
LOL... right, of course... EVERYONE knows that! ;) thanks though, @bbarry. One less mystery to worry about. :) Does it make sense what I am saying with POCO definition vs. .xsd schema approach? If POCOs were used then there wouldn't be a need to store these xsd's in these highly intuitive, accessible, and discoverable locations. 😛 I am open to feedback and would definitely appreciate another perspective! |
|
From @bbarry on May 12, 2016 15:17 I suppose it means the POCO is the xsd... I really don't care what format is used or what intermediates are involved between an on disk representation of a visual studio / msbuild project and some set of tasks occurring (as long as they work). I do care that the on disk representation is:
|
|
From @Mike-EEE on May 12, 2016 15:31 Excellent list @bbarry. That sums it up nicely and captures my sentiment as well. |
|
Another observation/key takeaway made this thread that I think is worth repeating/capturing here is that there is a conceptual difference in "project definition" (.NET Core approach) and "project build process" (MSBuild approach). I think this is where a lot of angst resides currently between the two systems and what the two sides are battling. As someone who has had their hands in MSBuild for over a decade, this is a big "aha" for me, personally, so I thought I would share/capture it here. Improving/clearing up the separation between these two concerns would go a long way. As well as having a POCO-based model that allows to describe them both in a tool-friendly way. 😄 |
|
I wanted to check in with this thread. Lots of really great discussion here, but not much over here. :) Since there seems to be a project system underway here, I wanted to reach out and hopefully get some feedback on the matter. It would also be valuable to maybe get some perspective/background on how this project is integrating with MSBuild, and what its take is on a POCO-based project defintion. This strategy/design of course will allow/support different data formats so that developers can utilize the best tools they are familiar with to work with those formats and, by virtue of the design, the model. 😄 After looking around this repo's documentation, it does appear that this is going to serve as more of a project definition, rather than process, do I have that right? So, there will be a conceptual separation between definition (here) and process (MSBuild)? If so, that would really be handy/useful/synchronous as that is what a lot of developers seem to be requesting. 😎 |
|
How about keeping both project.json and .csproj, where project.json is hand edited and .csproj is generated by When @Mike-EEE stated the difference between "project definition" and "project build process", this came to my mind. |
|
Thanks for the ideas/feedback @gulshan! Really the idea here is to make everything both hand-editable and designer friendly. I reference Xaml as the perfect paradigm and example of this. Even when you are not in a visual designer, the tooling "lights up" to present to you a properties window where further designer tooling may be available. I provide a good example of this here. The process/solution you describe is a good idea and one maybe worth exploring. However, at the end of the day this still relies on existing implementations which are not POCO-based and are data file format dependent (JSON for the project.json model and XML for the .csproj model). These are two incredibly prohibitive, counterproductive qualities for developers/development and ones that should be avoided in all future solutions, which is what this issue aims to accomplish. 👍 BTW, the definition vs. process "aha" magic meme should go to the lovely and gracious @shederman. 😄 It's quite accurate, isn't it! |
|
@Mike-EEE I am opening a new issue for that- #181 |
|
@gulshan ... yep, got the notification! I will be watching. Remember in all things: POCO and agnostic data formats. 😄 |
|
@Mike-EEE I appreciate your drive and enthusiasm on the project format, however, we have no intention of changing the project format to XAML or any other new format. You've got some great ideas, however, large sweeping and cross-cutting feature suggestions that affect many teams across Visual Studio are better filed on http://visualstudio.uservoice.com (which I believe this one already has). Issues, features and bugs filed in this repo are intended to be actionable work items that a dev on our team or the community can pick up and drive to conclusion. This is not one of those issues. Based on the fact that we're sticking with the existing MSBuild format, if you have actionable suggestions help improve the existing format, or bring some of the project.json features across to csproj, feel free to file them. Otherwise, I'm going to close this issue. Regards, Dave |
|
@davkean Thank you for taking the time to explain your position and for providing some guidance. As you noted I have already supplied a vote on UserVoice, currently having a vote for every day of the year. 😄 So I guess I am a little confused here on the best course of action, or ensuring that I am speaking with the right team. Currently there is this project, and then there is also MSBuild. There also seems to be a "tooling" team/project somewhere? Then there is also the fact that the vote that has been submitted that shows support but not much feedback/discussion. Is there a team and/or resource that you can recommend so that I am devoting my time and energy towards these ideas efficiently and effectively? Thanks again! |
|
The project format is used and consumed by lots of teams inside the Visual Studio group, and around Microsoft. While MSBuild owns the format itself - any change they make can impact 10's of teams, so it's not something taken lightly. Given that project.json/xproj was an attempt to change the project format, and was found to be too damaging and impactful on ecosystem, I think you'll have a very hard time convincing Microsoft to move to a new different project format after we just made the decision to move back to MSBuild. Your efforts would be better focused on ideas on making the existing format's experience better. You mention tooling; think about the kinds of tooling we could add to make working with csproj easier without throwing the baby out with the bath water. These are the sorts of issues you should be filing. |
|
@davkean I think something is getting lost in the translation here as this is indeed aiming to make the current format and experience better. project.json/xproj went off to try to re-invent the wheel so it is not surprising we have come full circle back to the original. :) Conversely, what I want to do is make the current process/experience better. This is done by first using a POCO-based method rather than a data-document model as it is now. This simplifies the current experience as developers are working directly with a POCO model, and also the infrastructure no longer needs to map document data to the POCO elements. Additionally, once this is done, developers will be free to use any serializer they wish to define and describe their project documents using the tooling they prefer. Data formats are no longer an issue (make a lot of developers happy) and the system as designed has a better experience. |
|
@Mike-EEE I can speak for both MSBuild and my team, we have zero plans to enable developers to serialize the project format into a data format of their choosing. Project files are XML, and will remain XML. Let's move past this format discussion and move onto discussions that focus on our roadmap ahead. |
|
I do appreciate you taking the time to discuss this @davkean. I know you are busy. :) I just want to make sure that you understand that this at its core is not a format discussion, but rather improving the way projects (and application objects in general) are described, not only from a conceptual perspective and also from a tooling perspective (where formats would then indeed come into play). I also realize and understand you (currently) have zero plans around this, clearly. 😉 We have been baked in this very backwards paradigm of configuration/object description for over a decade now. It not only prohibits/impacts developer experience, but makes solutions overly complicated (not to mention, provides poor guidance for the other projects that copy/paste its flawed design). My hope with this issue along with creating the UserVoice vote is that we start to provide awareness around this flawed concept and thinking, and start moving to a more Xaml-way of object description model that provides way less development friction (and improves adoption). (I feel I must reiterated that am not locked into Xaml "as a format", but am locked into its approach and development paradigm.) Again, I appreciate your time here. It is always an honor to speak with a blue badge. :) |
|
Hi @davkean I would rather not be a thorn in your side, and please know my intent is to help carry the conversation here, especially with all the project.json madness currently under way. In fact, I saw that you are doing the right thing and inviting developers to join the conversation here. It doesn't look good to invite developers to contribute ideas and foster that discussion and then shut them down (regardless of the political situation -- which I understand and respect, mostly. 😄 ). The other consideration is that I am providing this link out there to the interwebs and inviting people to (also :) ) join the discussion here, and it will look bad if they click it to see it as closed. So, I'm wondering, is it possible to mark this issue as a "discussion" or "future ideas" "suggestion" or something more appropriate? I would also recommend doing this for other "issues" that might crop up that are not entirely bugs or work items. I'm trying to help you out while also ensuring that others feel welcomed and part of the discussion to make the best of the products we know and <3. :) Thank you for any thought and consideration! |
|
Hey everyone, I think this proposal is a fantastic idea and the attempt and the same approach was just undertaken by the Gradle team (Android/Java build chain), with them implementing a Kotlin version of the build script - supporting refactoring, auto-complete and many other benefits that specifying configurations in a POCO can bring. It's notable that EF (from EF 2 onwards) took the same approach with POCO definitions, moving config from XML into OO, and the effect was a clear deprecation of the old, outdated format. MSBuild is long due for a rethinking, and if there's anything learned from the JSON .xproj format is not that we can't attempt at making it better, but that when we do make the step as a community we need to be bold and take the right step, all the way. I think that the POCO configurations ARE the the right step, that XML is heavily outdated, and that JSON was a poor choice for many reasons (no documentation tags being a deal-breaker right from the gate). Perhaps something like YAML would have been much more appropriate, or if we want to be daring enough - actually take on something in C# that the .NET community already knows. @davkean way to push this forward, you have my votes and support from this all the way. |
Thank you for your support @marchy! You are absolutely right in that EF moved to a POCO-model and dropped their XML format altogether. That is a fantastic example, and one I wish to illustrate. Because they did this, now I can define my EF objects in my application in any way I deem fit and necessary (based on my or my team's preference). Here is my illustration of this principle in action. In my case I prefer Xaml as I get built-in tooling and designer support right out of the gate. (I explain this in detail here). Accordingly, when I create my EF objects now, as they are POCO's, I simply create a Xaml file and describe them as such. Here is an example installation file of the EF entities I am currently using in the project I am working on: When working on these files I do not have to worry about the "document model" or "schema" or having to track down an .xsd somewhere, or even head over to google/bing to track down what a certain element is and its child elements. I can simple "Go to definition" on the symbol and get taken directly to the source code. It all "lights up" because it is all based on the POCO class definition. This is how I see/prefer/desire all application object definitions to be described, and a perfect example of what I am striving after. Additionally, since it is making use of Xaml in this case, I can take advantage of the format-specific features, such as markup extensions (as you see throughout the file above). Thanks for reminding me. 😄 |
|
@Mike-EEE I understand that you're a fan of XAML, but what does the declarativity of XML actually bring to the table when it has to be backed by compiled a .NET object model to do anything? Wouldn't it be much better to just write against the .NET object model and scrap XML altogether? That's how Cake works. You write C# "scripts" that operate directly on the build engine's .NET API without any abstraction layer. Cake uses Cake to build itself and its own build script is in my opinion quite elegant. The stuff you actually want to be declarative, like dependencies, could exist in the Also, that |
|
Hey @asbjornu thank you for joining the conversation!
You know, I saw you post this in another thread and I never did provide my comment, which is THAT LOOKS AWESOME. :) In fact, I still have cakebuild.net open in a tab to check it out. :)
There are two primary benefits from using a data format:
Ah to you it is, but to me it's pure bliss! Here again we get into format preference, which is one of the topics this issue addresses. With Xaml, I rest at ease knowing every single symbol in that file is connected to a .NET POCO somehow and somewhere. I can easily navigate through it and go directly the to object it represents. There is a 1:1 mapping between every symbol in that document and its .NET equivalent. It's why tooling works so well with it. :) As for GUIDs, etc. It's a data (initialization/installation) file so it's going to have lots of data. If you prefer something more terse, maybe you would prefer to have it described in JSON or YAML instead? 😄 That is exactly one of the points of this discussion, so that once the backing project model is well-defined and accessible, then developers can develop/define/describe their projects/definition in the format they see fit. |
davkean
added
the
Discussion/Question
|
@Mike-EEE I'm glad you like Cake. It is indeed awesome! 😄 And if we can agree to decouple programming the build engine from describing the project and its dependencies, I think we can agree to disagree on format preferences. Because then, in theory, the declarative configuration of your project can be done in XML, YAML, JSON or whatever and the programming of the build engine can be done in any language supported by the Roslyn and/or .NET language toolchain. For most projects, this decoupling would mean they would only have a very minimal declarative config file in the format of their choice and people could optionally add a procedural build file written against MSBuild's .NET API in a language of their choice. |
|
YES to all of the above, @asbjornu! We (now 😄) have shared vision! Remember that the project file is simply a configuration file, it describes the data, not what to do with it (the build engine). Much like in the referenced Xaml file above, it contains all the entities (data), but there is obviously no logic/processing there. That is handled by an internal component. The major problem we are running into here is that the data file has historically been defined by an arcane/obscure document object model, which is then parse and subsequently mapped to a POCO. This pollutes the design and makes it very difficult to work with -- developer and tool alike! Part of the problem we indeed have now w/ MSBuild is that it currently does both, this has been identified/explored already in this thread (and others) and we all agree this is a poor separation of concerns. All the while, (exactly as you say), if you have the data file, then it is conceivable that build systems (including MSBuild but not exclusive to it) can utilize this file to perform their own processing with it. Hey, even Cake, maybe? 😄 All done in the format (and subsequent tooling) that is preferred by developers and the teams/organizations they work on/with, reducing cost, and a little bit of angst/anger/stress while we're at it. 😎 |
|
Yes! I have nothing left to add. I think we're done here. |
|
SHIP IT!!! Oh, wait. 😄 |
|
Maybe I'm dense and this discussion is simply way over my head, but it seems to me that at the end of the day there's no difference between a data model that is defined as a POCO and subsequently expressed as a data format, and a data model defined as an XSL and subsequently expressed as a set of objects. Either way, same data, two ways to express it. Furthermore, this concept that the on-disk format can be "whatever the developer wants" is also nonsense. What if want X12? What if I want ASTM? Who is going to write the parser for that? Me? If the first step to using your build system is to implement major components yourself, it's not going places. Lastly, Cake and other fluent APIs can die in a fire. The first question to be answered here is, is it better to implement a toolkit for a build system (Cake), or is it better to implement a file that defines a project and have a build system that's smart enough to figure out what needs to be done based on that project definition (MSBuild+csproj/project.json)? |
|
Hey @godefroi thanks for jumping in. 😄
Correct. Except that one requires at least three artifacts (XSD paradigm -- which actually makes four artifacts once you account for the element-to-POCO mapper) and the other only requires two (POCO-based)
If there is a .NET serializer for those formats then you won't have to. :) What you would have to account for is creating a provider that tells MSBuild how to use that serializer to load and save files. Also keep in mind that the idea here is that all the major formats are accounted for up front so unless you REALLY REALLY REALLY like your format and want to dive into creating a provider for it (which really shouldn't be too difficult), you won't be spending a lot of time implementing "major" components. The idea is to leverage existing components/infrastructure and make the system as flexible as possible to provide options that improve/increase favorability and ultimately adoption. |
In my view, the one does not exclude the other. Instead of the merged bastard format that is MSBuild XML we have today, the two needs should be split up in two different formats; one declarative (XML or whatever) for describing the project and its dependencies and one procedural (like Cake) for programming the build engine. Having both of these needs covered by one declarative format makes it the worst thinkable format for both. |
|
A little nugget article I happened across that demonstrates a POCO-based approach. It's in ASP.NET Core/JSON, but it's the mindset I think is what is important. It is placing a class definition as the configuration schema over a document model as the schema, which this leads to less code to manage/maintain. It's just prettier. :) Here's Rick Strahl: https://weblog.west-wind.com/posts/2016/May/23/Strongly-Typed-Configuration-Settings-in-ASPNET-Core |
Creating the standard sln flies for the root of the sqltoolsservice project and adding xproj files for the individual projects. Has been confirmed to open and build with VS2015. xUnit tests will work, but only from Test Explorer (ie, ReSharper test runner doesn't work with dnx) Note: This may be subject to change as new standards for .NET Core project files are developed. See dotnet/project-system#37 for more details.
|
Whoooaaa... am I to understand correctly that the CPS project system already IS using Xaml in some capacity? If that is the case, maybe sanity will rule the day after all. 😛 |
|
Yes, we use XAML to describe items in the the MSBuild project and how that gets passed to various services in the project system. |
|
The irony. So it's not exactly the project system that is the mitigating factor here, but rather MSBuild. Hopefully over time (another decade or so? 😛 ) your good practices/approach will spill over there and we can start moving in the right direction here. 😎 |
|
Not sure if I got it right, but the proposal is to move from XML to XAML? |
|
@304NotModified the idea/concept/goal is to move from an XML-mapping-DTO-to-POCO paradigm (complex) to a POCO paradigm (simple). Any object and artifact is simply a serialized POCO with a well-defined and known schema: the class definition. Please do not get focused on the Xaml, as if a POCO-based approach is used, then any format can theoretically be used: XML, Xaml, JSON, TOML, YAML, even INI if you would like. I use "Xaml" as it is the best representation/implementation of what is desired here. Hope that makes sense. (And @jnm2 haha thank you for the -- ghostly -- feedback. I indeed get carried away sometimes, all in good fun/nature though. I'll try to keep it in check... try. 😉 ) |
|
@Mike-EEE Lol, nice! I felt like a terrible person after saying it. Thanks for being cool. |
|
^^ For that, you get one GitHub reaction. 😄 (Sorry, I really like the fact you can do more than one!) |
|
Thanks for the summary @Mike-EEE! |
|
Hey @Mike-EEE I've started looking at doing something like this via a custom msbuild task over at https://github.com/aL3891/CustomPackageReferences, i'd love your input :) |
|
Very cool @aL3891 ! Thanks for the mention. 😄 FWIW I was/am working on a POC as well for dotnet/msbuild#613. Just a real simple console app will read in a JSON/XML and yes XAML :) to launch a set of commands. Ironically, I got caught up in XML as it didn't render it as pretty as JSON (that's what I am working on now). But hopefully you can see where I am going with that with the files provided. As for your efforts, I guess I need to be caught up to speed on PackageReferences. I am wondering how this will allow us to describe build/project files in our own preferred formats? |
That is basically what this project does :) You just implement |
|
OK I am seeing it now @aL3891. So this is for project references, then? So you can describe your project (package) references in any format you wish? I like the approach here, my only concern is that there is much more to a project file than references. :) Are you looking to add more of that here or do I have this misunderstood? |
|
Well its for package references (I realize I said project references above, that was a typo) It is only for package references at the moment, although you could set other build properties if you'd like, even if there are some limitations on what you can do in a target. From what I've seen though, being able to specify packages is the feature people seem to be missing most. |
|
Ah! OK @aL3891 haha TBH I am not 100% up to date caught up with VS15 and all the new finangled nomenclature. :( So package/project was confusing. Thank you for clarifying. I am still on VS2015 (not to be confused with VS15 ;) ;) ;)) so ATM not able to open that solution without a bombardment of errors in the IDE. So I do have a suggestion and have opened your first issue to discuss it. See you there. :) |
|
I'm going to close out this discussion, we've released VS 2017 RTM - and enabled the new project system and format initially with .NET Core/.NET Standard projects, and then we'll bring it later to all C#/VB project types, see: https://github.com/dotnet/roslyn-project-system/blob/master/docs/repo/roadmap.md for information. Any further feedback is better opened as individual issues on this repo, the SDK repo or the MSBuild repo. |
|
Thanks @davkean. All interest and discussion has indeed been moved over the MsBuild repo as you suggest. I have appreciated your patience and dialogue in this matter. 👍 |
add extension link
From @Mike-EEE on May 12, 2016 8:5
I (unfortunately) made a lengthy, somewhat off-topic post/rant about this in another thread yesterday so I am going to throw it down here in a dedicated issue and see how it goes.
There's talk about the new build system. There has been previous efforts over at MSBuild to examine the build system and improve upon its format. These were ultimately denied, expressing a "better safe than sorry" (huh? 😛) perspective. So, with all the energy/enthusiasm/magic over here, this appears to be the new lightning rod for developer creativity. 😄
My ask is simple: Continue to use the XML format (or even JSON), but move away from the element mapping-to-POCO strategy and make the csproj/xproj/proj file a fully (XML or JSON) serialized POCO instead, so that at the very least we know the precise POCO (and definition/schema) that we are dealing with and constructing. This helps not only developer experience (and thereby sanity 😄 ) but also any tools/designers that wish to build off of it.
Currently, the project file looks like something like the following:
This is great for defining arbitrary data in a file, but it is completely useless for someone who opens up the file and wondering: "what the **** is a
<Project />element?!"The above design requires at least three artifacts:
Oh but wait, there's more! Due to the above design, I bet you there is at least a fourth artifact!
4. The mapping class/utility that reads in the data file as an
XmlDocumentand laboriously reads in element by element, attribute by attribute to create and instantiate the POCO object(s) with their initialized values. OH THE INEFFICIENT HORROR!!Conversely, if we take a page from Xaml file design, we have the data file's schema baked right into the file by way of the class definition. As a result, we only require two artifacts:
This is a MUCH easier to design to produce, maintain, and manage. Tooling can tie right into the class file (using Roslyn??? 😄 ) and get all the information it needs. Developers do NOT need to hunt and chase down schemas in some arbitrary, hidden location (registry?! LOL, might as well be!) to figure out the file they are working with!
This is a very (relatively) simple design improvement that will make a world's (universe's!) difference in project look and feel. I cringe every time I have to go into the .csproj file because it is SO ARCANE AND MYSTERIOUS TO WORK WITH!
Finally, by using a class definition as the schema, another headache that this eradicates is the inconsistency between camelCase and PascalCase when declaring/using symbols/attribute names within the file (ALL THE INCONSIST!).
Anyways, thank you for any consideration and dialogue around this idea.
Copied from original issue: dotnet/roslyn#11263
The text was updated successfully, but these errors were encountered: