(v2.0) REPL benchmark stats #1457
Comments
|
Binary sizes (uncompressed): Looks like the unminified size is 30% bigger than Fable 1.3.x, but that goes away after minification. |
|
|
This awesome @ncave! Thanks a lot for this data, it will make it much easier to check if future developments go in the right direction 👍 The results are very interesting and promising indeed. It's really encouraging to see the Fable part is as fast as .NET (although it may be because we're skipping the optimization phase) and I'm sure we can make the FCS part faster too. It was expected the unminified bundle was somewhat bigger (because of name mangling) but also that the result would work better with JS minifiers :) I wonder if the improvement when creating the Interactive checker is also due to the WebAssembly bits in Long.js, as the int64 type is used a lot when reading the assemblies. |
|
@alfonsogarciacaro Good news everyone! After switching the FCS lexer from char to uint16, the bench times improved |
|
On medium size files, down |
|
This is great @ncave, awesome job!!! 👏 👏 👏 |
|
Awesome work @ncave :) Thank you for what you are doing 👏 👏 |
|
I'm very happy that we finally have a good example of the performance boost of using typed arrays thanks to you @ncave :) I also wonder if we should consider compiling |
|
@alfonsogarciacaro Sure, let's try it, since we can measure perf now. |
|
@ncave I've started some work to compile chars as numbers in this branch. I managed to make the tests pass, but I'm concerned there may be situations where |
|
@alfonsogarciacaro Do you have an example? |
|
@alfonsogarciacaro I mean, is it different than casting char (or int or float) to an object and passing it around, where you lose type? |
|
A bit contrived, but something like this: open System.Collections
let chunkTyped size (xs: 'T seq) =
Seq.chunkBySize size xs
let chunkUntyped size (xs: IEnumerable) =
xs |> Seq.cast<obj> |> Seq.chunkBySize size
testCase "Casting string to IEnumerable" <| fun () ->
let xs1 = "fffff" |> chunkTyped 3
let xs2 = "fffff" |> chunkUntyped 3
Seq.toList xs1 |> equal [[|'f';'f';'f'|]; [|'f';'f'|]]
Seq.toList xs2 |> equal [[|'f';'f';'f'|]; [|'f';'f'|]]The second assert will fail in Fable dev2.0-chars (it passes in .NET F#). Although now that I think about it, it may not be something very important because we're already relying in knowing the generics at compile time for many collection methods, so casting to untyped Another issue I just noticed though is when printing chars (as in |
|
@alfonsogarciacaro I don't know if compiling chars as numbers will be an improvement, trying to perf test the |
|
Hmm, I did remove some defensive code to simplify things, but it's weird that error didn't happen before. For now, I just added a dummy Attribute type so it's possible to inherit from it and then rebased dev2.0-chars branch. |
|
@alfonsogarciacaro I don't see any difference in the bench performance of |
|
Perfect, thanks a lot for checking @ncave. Ok, I will just submit a WIP PR for now, and we can revisit it later. BTW, I want to start building the REPL again in Appveyor (I keep having problems when building the REPL locally on macOS). Are you just using the REPL Fake target for that or doing anything else? |
|
@alfonsogarciacaro I was just building the bench for now ( |
|
@ncave I've made some changes in the bench app, I hope you're OK with them. I separated the REPL from the bench (so it's not necessary to build the REPL every time we want to make changes in the bench) and also created two different projects for each platform (JS and .NET). These are the results in my machine, after unziping the BTW, I removed the We probably can announce the beta now and prepare and online version of REPL 2 🎉 |
|
@alfonsogarciacaro That's fine, although my preference would be to keep the bench as a single project. Its whole purpose is it to run the same exact code on different platforms and compare results, which will be a bit harder to maintain if said code is in two different projects. |
|
You're right, but it was a bit difficult to keep a single project after separating the REPL (the JS version doesn't have a direct reference to Fable.JS). However, the main file is still the same for both projects ( |
|
#1487 fixes the REPL skipping some bindings issue. As suspected, the real numbers are not as spectacular as before since the compiler has to do more, here are the latest more realistic numbers: Still, only about 4x-5x slower (vs the highly optimized .NET Core) is not too bad as a start. |
|
@ncave Latest Fable 2 online REPL puts the FCS/Fable compiler in a worker so it doesn't freeze the UI. I also added an option to enable FCS optimization, logging for some performance stats (visible in the browser console, similar to the ones you created for the bench) as well as the sudoku and ray trace samples. Hopefully this makes it easier to play with the REPL to find more optimization opportunities :)
|
|
@alfonsogarciacaro Yes, Fable 2.0 seems about twice as fast as Fable 1.3.x. |
|
@ncave I'm mixing here the Fable 1 stats from the first comment with Fable 2 data from this one. Does this look right? |
|
@alfonsogarciacaro More or less, yes. Not sure what you're trying to do, up-to-date measurements or reposting from top of thread? REPL performance didn't change much after switching back the chars impl, which is good. We can re-measure after the properties branch settles. |
|
Thanks @ncave. I just wanted to have the proper comparison between Fable 1 and 2 to show in blow posts and talks :) |
|
@alfonsogarciacaro To be fair, some of the Fable2 improvement on the REPL workload comes from switching the FCS-Fable lexer from char to uint16, which would have benefitted Fable1 too, so I'm not sure how useful those numbers are for comparison. Still, the REPL2 is faster, that's a fact. |
|
Yeah, but nobody knows that 😉 Just kidding, it'd be nice to try to run the benchmark with the |
|
Could you please specify what version of FCS/Fable needed to build this project? Is there any tutorial available (as it will probably require some trick to build it). Right now it fails with: |
|
@zpodlovics If you just want to run the REPL benchmark locally: If you by any chance run any perf analysis, please share, thanks in advance. |
|
@ncave Thank you! Almost there. First it missed some kind of resource file FSComp. I generated it running However when I try to build the bench project after lot's of compilation it fails with: Any idea? What files should I have in |
|
@zpodlovics I apologize, here is the missing codegen step: I have updated the steps above as well. |
|
@ncave Thank you for your help! I was able to build the repl. I was also able to do some profiling. The profiling done with perf + nodejs
fable2-repl-2018-08_31_1-speedscope.zip Node ran multiple threads, the first thread is not really insteresting, you can change between the threads by clicking to the up (^) /down (v) icons on the top middle part of the visualization screen (^
|
|
Up-to-date stats based on the main FCS-Fable PR branch, showcasing the performance impact from switching the To recap, using |
ncave
changed the title
Some preliminary baseline stats:
Looks like there is some improvement over Fable 1.3.x :)
The Fable times are close to dotnet, hopefully FCS javascript can be optimized further.Update: See bottom of thread for up-to-date stats.
The text was updated successfully, but these errors were encountered: