JSBSim for SPEC CPUv8 #834
Comments
|
Hi @heshpdx, sounds interesting. Having a quick glance it looks like the closing date for the first step submission is 13 Mar 2023.
I'm a bit confused about the reference link to my pull request for - "Add support for forces in stability axes and moments in wind and stability axes" |
|
Ah, just realised you probably just typed "hash 2" and Github auto-completed that to reference a pull request, with the 2nd pull request in the Github repo happening to be my pull request at the time. |
|
Yes, the first step is submission of entry, which takes five minutes. That is what the 13 Mar 2023 deadline is for. Sorry about referencing PR 2! I edited and removed the link. |
|
I agree with Sean – this is very interesting and I think it is appropriate to use a JSBSim case as a potential benchmark.
|
|
@jonsberndt do you want to go ahead and fill in the submission entry? You have the right initials 😉 |
|
I just received the first portability report, from gcc/mingw running on Windows server 2019. As an example, this is the kind of problems we solve and then upstream the fix if needed. Do we expect issues with mingw or with MSVC? |
No we don't expect any issues with mingw and MSVC. Each commit results in builds for all the supported platforms, e.g. see https://github.com/JSBSim-Team/jsbsim/pull/836/checks
Plus a number of tests are run each time as well. In terms of the particular error report above, how is JSBSim being built? Is it using our CMake setup? In particular I'm wondering whether there is some missing define to do with JSBSIM_API and dll import? |
|
Thanks, your info provides proof that I am missing something. The SPEC CPU harness builds code differently than CMake. I had to break humpty-dumpty apart and put him back together again, and perhaps we are missing a flag. We can't support dll import so that functionality is turned off; that is probably it. |
|
LoL – sorry I didn’t see this until just now. I might not be able to get to it until this weekend. If there’s no rush and no one else has addressed it, I’d be glad to.
|
|
FYI, I am reviewing the benchmark for at https://www.spec.org/cpuv8/entry_form.html and intend to submit the entry this evening or tomorrow.
|
Yep, I was going to say, no immediate rush, just have to make sure it's done by the submission deadline of 13 March 2023. |
|
I submitted JSBSim for consideration this evening. |
|
Thanks! The committee voted yesterday and accepted it into the search program. I will continue to be your project contact. 😄 I am curious about your opinions on building with We began testing with There are some nuances in SPEC rules, to separate FP benchmarks from INT benchmarks. The rules state that benchmarks that run more than 10% FP instructions are designated as FP, and less than 1% are considered INT. The ones in between 1% and 10% are in no-mans-land and are requested to fix their profile if they can. It turns out that JSBSim comes in at 6% FP instructions. Now, the rules can be changed and I know there will be discussion as to what it means to be an FP application because I am working with multiple benchmarks that come in around 5% FP. I predict that the committee will categorize JSBSim as integer. If this is the case, then the issue above is a moot point because in general, the INT suite does not support verification with |
|
Also, can someone help craft cmdlines? I naively ran all the examples and selected the ones which were running the longest. I came up with these: But when running these back-to-back we just clocked 50 seconds on our reference machine. We need a runtime on the order of 180 to 300 seconds of interesting work, which motivates this post. I would love suggestions, especially scripts that exercise different parts of the code, and use different aircraft or engine models. It doesn't have to be in the repository; if you have large models of your own I welcome those with open arms. Thanks! |
I will try and post some scripts within the next week or so if that's OK. |
Can you elaborate on this? JSBSim is not an integer application (edit: at least I don't think of it as one). Is there some more context that can be provided? |
|
new cmdlines: sure thing, we are not in a super-rush. We have about 7 weeks to get this right. FP vs INT: Step 3f at the rules: https://spec.org/cpuv8/#step3f
The current profile shows that 6% of total instructions are FP. Your opinion matters! If you think it is an FP app then we can take that into account. When I look at the perf profile, I see JSBSim has a much higher than average number of icache and itlb misses (commensurate for a simulator), which is not usually found in "traditional FP" apps. Or maybe the "traditional FP" apps in SPEC CPU are more focused on HPC and not simulation. Let's see what happens; this is not a deal-breaker by any means. Actually it forces the committee to have a discussion. |
It's a very interesting discussion! When I think of JSBSim code I think of the equations of motion, which are a very math-intensive set of operations (matrix and vector operations). In cases where there is data logging going on there would be some file IO. Depending on which aircraft is being modeled there are varying numbers and complexities of what I think of as math models: atmosphere, gravity, engines, aerodynamics, etc. I'm surprised that the actual floating point operations are a relatively small portion of what the CPU sees. I'm considering some scenarios that might lead to a higher percentage of FP operations. I'll try and supply some new scripts over the weekend. Also, I can't say that I ever coded in a way that considered optimization for the CPU. I'm curious if there are [non-draconian] things we can do to the code that might improve performance. |
@heshpdx can you share a link to the profile results? |
|
@seanmcleod I ran cachegrind this evening and that was interesting. Not entirely sure how to interpret all the output or to use it for anything yet, but I ran this:
This can be turned into something more readable like this:
Seems that find_child() (a property system call) represents over 3% of the calls in a JSBSim run - at least for the example I ran. |
|
I also see |
|
Attaching some details. Prof is just the info I pasted above. |
|
I ran cppcheck on the source tree to check for any crazy errors. (This was more out of curiosity than anything else., since I have proposed cppcheck as another candidate for CPUv8.) In case it is helpful I put the output below. Some of the issues tend to be innocuous or irrelevant, so your mileage may vary. |
Hi! Any luck in making longer scripts? Thanks! |
|
Hi, Mahesh, Sorry it has taken me a while to get back to you. Define "interesting work". ;-) There is one script that can pretty easily be modified to run for any length of time. That is the scripts/ball_orbit.xml script. I went in and changed that as follows:
That is, the end time was increased to 86400 seconds. It ran for about ten minutes. I'm thinking if you set that to 30000 then the sim should run for about 4 minutes. But, is that "interesting"? I think I could also modify one of the C310 scripts to fly a route for that long. Jon |
|
If the work is repetitive (the same code is being executed over and over with same/similar data), then it is not interesting. If a script exercises different parts of the code with different data, that would be ideal. If I were to use knowledge of my own CPU simulator to make an analogy to your simulator (even though it is a different domain), it would be something like the following. Not interesting:
Interesting:
When I pitched the benchmark to the committee, I had the aircraft directory open, and they saw the model for a pterodactyl and got excited that we would have that in one of the cmdlines. I don't see any scripts with pterodactyl though. Can we use that model somehow? Do others have more complex scripts which simulate other kind of phenomena? I saw people on the forum asking for Martian atmosphere for the NASA helicopter. Also, some of the scripts where the aircraft crashes - those are good too. |
I'd define it in terms of good code coverage 😉 It looks like multiple scripts are being run, i.e. we don't have to get everything done in a single script. In terms of coverage I'd suggest the following as a starting point:
|
|
Thank you for a much better description. Yes, different types of engines and crafts, and timed events. |
|
If it works to have multiple scripts involved to interestingly exercise more of the code instead of a single, mythical "super-interesting" script ;-) then I think Sean's suggestion is a great way to go. I will note, though, that at the lowest levels much of the same code is being continuously exercised, just with different data. The C310 script that takes off and flies a route is pretty interesting for a bit at least. I think it still works. Also, would perhaps running a couple of instances of JSBSim simultaneously be helpful? I'll see if I can come up with a batch of runs to make. |
|
I have a symbolic link from JSBSim in the local directory to the executable in the build/src directory. Here's a potential set of runs: ./JSBSim scripts/c3104.xml Note the "&" at the end of the final three lines. I copied this into a unix terminal and it ran through to completion. Duration was several minutes. |
|
Thanks. Yes, I have been trying the scripts in the repo, see #834 (comment). Are there other scripts lying around on people's workstations that are not in the repo? Usually a benchmark consists of 3 or 4 cmdlines (not 15), which is my motivation to search for long running scripts. Putting jobs in the background: That is a specific style of running that we definitely will employ at a later phase of this benchmark creation. For now I am just looking for single-threaded set of runs. I tried changing one of the scripts to replace the aircraft with the pterodactyl but that failed quickly since I'm a newbie 🦖 |
|
I received word today that "SPEC wishes to further evaluate your entry." Regarding the Pterosaur, maybe this will help: https://wiki.flightgear.org/Pterosaur |
|
Is there anything still being waited on - I know I had said I'd get you some updated scripts. Still need those? |
|
Hi! Yes, please send some new/updated scripts when you have them. Adding time to make ball_orbit.xml run longer is a good idea since it continues computation. Can you try making the ball hit shards of a busted Russian satellite? (Basically create a new flight path). Bonus points if you can also put the Pterosaur into orbit :) I'm trying to get this benchmark (and many others) through Step 2 which includes documentation and licensing checks, so that is taking up my time these days. There is a soft deadline of April 16 for Step 2, and as such I haven't had a chance to make new jsbsim scenarios myself. Should we scrape the issues database for models and scripts that users play with? A quick search shows this #177 (comment) Maybe a promising new model not in the repo? |
|
I've made some headway in increasing the runtime. I found that I can change the "dt" in the scripts, which I imagine is the epoch length for rerunning calculations. A combination of finer granularity of time along with increasing the "end" time for simulation can give longer CPU runtime. Is this ok? I don't want to stray far from the original intent of the workloads. This is a diff from my SPEC harness. |
|
Quick update. Things are looking good, and I was able to increase the runtimes via reducing dt in the scripts. We have been testing across various platforms and compilers. The NVidia compiler exposed one issue that I would like to share with the community. At the beginning of main, there is code for enabling FP exceptions: I browsed the history and saw that these controls were put in for debugging: Right after that, some real div-by-zero's were found and fixed in 2016 (6e325f0 and c5e887b), but then the exception enabling was never disabled. This code leads to some performance slowdown due to checking for exceptions after each operation, and handling when an exception occurs. We did actually see any div-by-zero on the committed instruction path; the issue was on the speculative path where exceptioning math was being performed but never committed -- and this behavior varies by microarchitecture and compiler! Anyway, we fixed this by disabling fp exceptions completely, to normalize behavior on all benchmarking systems. Removing the code improved the benchmark performance between 0.3% (Intel x86) to 1.0% (Ampere arm) and 3% (Huawei arm), the three machines at my disposal (if you'd like to confirm, the biggest gainers I saw were scripts/x153.xml and scripts/c3104.xml which get 3.5% faster on x86). It may be prudent to put a |
|
@heshpdx what step is JSBSim at in the 6 step process? |
Step 3! |
|
We passed Step 2 and are in the midst of Step 3. Most of the benchmarks are in Step 3, waiting on the Windows profile to come back. I think I'll be able to get past Step 3 by the end of this month. |
|
Okay, so I jumped the gun when I told someone today that JSBSim code was now part of the SPEC benchmark 😉 |
|
I work on the SPEC CPU committee as a colleague of @heshpdx. I work on porting and profiling workloads on Windows. One task is to ensure that the work done and the workload outputs are consistent between Windows and other OSes. I am investigating a case where the output produced for the script ball_orbit.xml is different when I compile JSBSim with the Microsoft MSVC compiler, versus when I compile it with the gcc compiler in a MinGW environment. The discrepancy is so precise & consistent that I hoped an expert in JSBSim might recognize what what variable or function might be going awry under MSVC. To be clear, I don't think this is an issue in JSBSim. In a few specific and consistent cases, the MSVC-compiled version reports a value of "Argument of perigee" that is (360 - X) compared to the gcc-compiled version. I noticed that this occurs when (and only when) the matching "True anomaly" value is ~ 360. An output example when compiled with MinGW/gcc:
The same output when compiled with MSVC:
I can "correct" this difference in the output by inserting this into src/models/FGPropagate.cpp#533: |
|
To augment @bowers's post, we recently taught SPEC CPU's verification harness to tolerate coterminal angles. Therefore, if the MSVC output for Argument of Perigree were reported as -158.5' instead of +158.5', that would be coterminal with +201.5' (which is the "golden reference output") and we would not be in this predicament because the output would verify. So perhaps that gives a clue as to what is happening. |
|
Okay, just double-checking, so it looks like the only assignments to And in the example output above, the jsbsim/src/models/FGPropagate.cpp Lines 518 to 529 in ae4652a And so the difference between gcc and MSVC is due to a different result between them with regards to line 523 and this conditional: if (eVector(eZ) < 0) |
|
Okay, so doing a quick lookup on And taking a look at the output for MSVC I see the following pattern:
Which looks incorrect, i.e. between time 2400 and 3000 it implies the ball has moved in reverse, a value of 201.5 would make more sense. Taking a look in the debugger for time 3000,
VState.vInertialVelocity - {-9352.1465544806997, -23694.549595047396, 0.0000000000000000}
angularMomentum - {0.0000000000000000, 0.0000000000000000, 553426172137.42932}
eVector - {-0.0014013482342756234, 0.00055309720560003939, 0.0000000000000000}@jonsberndt, @bcoconni can you spot anything obvious going wrong? |
|
Hmm, trying to picture the orbit I took a look at the reset file used by the script. jsbsim/aircraft/ball/reset00_v2.xml Lines 8 to 19 in ae4652a So it appears that the ball starts off at (lat, lon) of (0, 0) and then given the yaw angle specified and the x-body velocity it starts an equatorial orbit? Which seems to match in terms of the initial state report looking at the ECEF coordinates and ECEF velocity. ------------------------------------------------------------------
State Report at sim time: 0.000000 seconds
Position
ECI: 21725646.32546, 0, 0 (x,y,z, in ft)
ECEF: 21725646.325460 , 0.000000 , 0.000000 (x,y,z, in ft)
Local: 0.000000, 0.000000, 800000.000000 (geodetic lat, lon, alt ASL in deg and ft)
Orientation
ECI: -89.99999999999999, -0, 90 (phi, theta, psi in deg)
Local: 1.272221872585406e-14, -1.412450015376051e-30, 90.00000000000001 (phi, theta, psi in deg)
Velocity
ECI: 0, 25473.40428154583, 0 (x,y,z in ft/s)
ECEF: 0, 23889.14516700002, 0 (x,y,z in ft/s)
Local: 0.000000 , 23889.145167 , 0.000000 (n,e,d in ft/sec)
Body: 23889.145167 , 0.000000 , 0.000000 (u,v,w in ft/sec)
Body Rates (relative to given frame, expressed in body frame)
ECI: 0, 0, 9.277188200408133e-19 (p,q,r in deg/s)
ECEF: 0, 0.004178074132240404, 0 (p,q,r in deg/s)
---- JSBSim Execution beginning ... --------------------------------------------
Time Notify (Event 0) executed at time: 0.005000
Height AGL (ft): = 800000.000000
Altitude (Geod, ft): = 800000.000000
Latitude (Geod, deg): = 0.000000
Vehicle Radius (ft): = 21725646.325460
Inertial Vel Mag (ft/s): = 25473.404282
Body X Velocity (ft/s): = 23889.145167
Simulation Frame: = 1.000000
Density: = 0.000000
Angular momentum (ft^2/s) = 553426172126.323120
Inclination (deg) = 0.000000
Right ascension (deg) = 0.000000
Argument of perigee (deg) = 0.000000
Period (s) = 5374.952766
Eccentricity = 0.001507
Apoapsis radius (ft) = 21791206.646590
Periapsis radius (ft) = 21725646.325460
True anomaly (deg) = 0.000000
Start: Monday October 23 2023 22:17:03 (HH:MM:SS)So as a complete orbital mechanics layman, shouldn't the |
|
In case it matters, here is the change I made in the |
|
I'm still probing. Two observations: I can get gcc to report same values as MSVC by replacing |
So in my mind jsbsim/src/models/FGPropagate.cpp Lines 507 to 515 in ae4652a |
|
https://oer.pressbooks.pub/lynnanegeorge/chapter/chapter-3-the-classical-orbital-elements-coes/
So we should be detecting all these cases and setting the argument of perigee to 0. I'll submit a pull request to fix this. |
Another way to maybe get around the argument of perigee problem is to consider using the new script I committed that has an orbit with a non-zero inclination:
Note that this script requires an update to the code and a recompile. |
Hi @bcoconni and friends,
I’m a CPU architect at Ampere Computing where I do performance analysis and workload characterization. I also serve on the SPEC CPU committee, searching for benchmarks for the next version of SPEC CPU. We try to find computationally intensive workloads in diverse fields, to help measure performance across a wide variety of behaviors and application domains. Based on the longevity of JSBSim, its large active community in aeronautics, education, and game play, I am proposing the JSBSim flight dynamics models be included in the next set of marquee benchmarks in SPEC CPU. The committee is running a “search program” or contest, where people can submit their benchmark ideas. I am humbly asking if you can please submit JSBSim for consideration. You can see all the information here, along with the submission form: https://spec.org/cpuv8/ There are monetary rewards for the submitter if the entry is chosen to move forward.
Full disclosure: Since I am on the committee, I cannot submit entries for winning prizes. But you can, and I encourage you to do so, because:
Full disclosure 2: I have already done 90% of the work. I took the JSBSim mainline source, placed it into the SPEC harness, and wrangled it to build and run on multiple platforms with command lines that conform to memory footprints required by the benchmark rules. I started the process of verification of results across multiple platforms, which led me to file PR #830. The part that is remaining is the documentation, fixing cases where the simulation script output is slightly different between optimization levels, and adjusting the scripts to satisfy the run-length requirements for CPUv8 rules. There is also opportunity to use different scripts/aircraft/engines other than the examples that I dug up while browsing the repo.
There may be other things that we find that could be pushed back to the JSBSim mainline, as we work through portability issues. So that is also part of the journey. If this sounds interesting, please let me know and/or take five minutes to submit an entry at this website: https://www.spec.org/cpuv8/entry_form.html. This is a tremendous opportunity to have a lasting impact on the CPU industry.
Cheers,
Mahesh
The text was updated successfully, but these errors were encountered: