-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixing up shallow water model (cleaned up version of #3486) #3502
base: main
Are you sure you want to change the base?
Conversation
Nice! Post some videos |
@simone-silvestri Will do! At the moment I am using
This failed because of the error below. Any idea what I need to do to fix this?
|
Some good news is that more tests seem to be passing compared to the prevoius PR. One of the messages I saw was there was a cancellation signal, see below. Lots of the tests just cancelled, and I'm not sure why.
|
@simone-silvestri Unfortunately, my run died just before 20 days. I guess the simple advection scheme that I was using without viscosity wasn't stable enough. Could you help me to set up the one you suggested on the previous PR? I tried was you suggested but got an error, which is copied above. |
I don't know what you saw. https://buildkite.com/clima/oceananigans/builds/14750 is that almost all tests pass. |
Thanks @navidcy , that does look very promising! The error that I saw is copied below. I am not quite sure where this comes from though.
|
Hey @francispoulin, sorry for the mistake. We probably should change the way we specify the methods for vector invariant in a vertically using Oceananigans.Advection: OnlySelfUpwinding
VectorInvariant(vorticity_scheme = WENO(), kinetic_energy_gradient_scheme = WENO(), upwinding = OnlySelfUpwinding(; cross_scheme = WENO()) or VectorInvariant(vorticity_scheme = WENO(), divergence_scheme = WENO()) This last one is easier and works because of how the defaults are defined. It makes sense in 3D (where you want |
Thanks! I have tried that and it seems to run. I will restart my simulation. It's pretty slow as it did 20 days in about 1 real day but I'll see how far it gets before I need to continue it. |
Run on a GPU? |
No! Thanks for asking and I realized I was testing on a CPU. Not to get it to run on a GPU. |
I have had some success but am still having difficulties with the height field becoming negative and then a numerical instability developing. Of all the things I have tried so far, thanks @simone-silvestri for all the suggestions, the following scheme was able to go the furthest, just after 50 days.
This is the resulting movie. Certainly things are energetic in the shallow ocean, and presumably why difficulties are developing. near_global_shallow_water_1440_600_surface.mp4 |
Looks like there is a problem near physical (not immersed) boundaries for the height. |
Looks like also in the original case there was the same problem. But for some reason, the simulation was not crashing. near_global_shallow_water_1440_600_surface.mp4 |
Thanks @simone-silvestri for the observations. I will think about that tomorrow. Today I also tried to do a flat bottom run, just to see what would happen with no bathymetry. I added the last line below, which I thought would set the bottom of the ocean to 5km below the free-surface. Unfortunately, when I ran the simulation nothing happened after even a year.
This clearly didn't work. Any observations as to what I did wrong? |
I have looked at the lines of code I copied above and understand the following:
One concern that I have is that afterwards I see what I did wrong in my attempt to define a flat bottom. When I tried to set |
I tried the simulation commenting out the line I also put this line back in and forced the minimum dept to be 50 m instead of 10 m, and that became unstable after 53 days instead of 52 days. Not much of a difference. |
I thought I would try running the global model in the hydrostatic context with only a few points in the vertical, to see what that looks like. The only one I could find is in multiregion but that seems to require multiple GPUs. I remember there was code back in the day that ran on one GPU. Is this code still around? If not I suppose I can modify the multi region one but I thought I would ask before I got started. |
btw @francispoulin, when we merge this all commits will get squashed into one (just mentioning this in case this is why you closed #3486 in favor of this PR) |
I redid the global shallow water run with Sadly, even with closure the run still died after 54 days. Slightly longer than without closure but not a significant difference. Here is the result of the simulation, in case anyone is curious. near_global_shallow_water_1440_600_surface.mp4 |
This is a result from our updated code that shows the free-surface height restricted to 9.9 and 10.1. You can clearly see that the instabilities happend around three particular regions. near_global_shallow_water_1440_600_surface.mp4 |
Here are some plots that maybe help to explain the difficulties we are having here. First, a plot of the free-surface height and we see two regions in the Indian ocean where the height is close to zero. Second, this is a close up of the region in the centre of the Indian ocean. Third, we see the topography for the same close up region. Fourth, this is a zonal slice of the bathymetry, and see see that the height gets very shallow, it's @simone-silvestri Do we want to change the tolerance parameters of the simultion or smooth the bathymetry? |
I would try smoothing the bathymetry to see if the problem persists or disallow shallow regions below 100m |
Thanks for the suggestion @simone-silvestri . I did the easier thing and cut off the depth at 100m. It runs until 3 days before we get blow up. The movie is pretty fun to watch and we see a lot of gravity waves being radiated in Antarctica, but the blow up seems to be south of Australia. I don't know if the gravity wave radiation happens in the hydrostatic model at all. Do you recall? This is in the invivscid limit. I will try again with some viscosity to see what affect that has. near_global_shallow_water_1440_600_surface.mp4 |
I made the change you suggested and do not allow any bathymetry within the top 100 meters, using the following lines of code
When I plot a slice through the same region as before, we see that the bathymetry now reaches the surface, as you can see below. I wonder whether having one point at the surface still causes problems? If the highest most point was lowered to the same level as the two points to the right, do you think this would be better behaved? |
These explorations are great but if in the end we figure out that it was the bathymetry fiddling that was the issue perhaps we should think of ClimaOcean as the place to hold this validation script? It includes tools to manipulate bathymetry… |
I am sorry that I hadn't looked at ClimaOcean.jl before. I just cloned it and it looks great! Thanks for putting this together. I am happy to have this script be hosted anywhere, after we get it working. I am going to run your quarter of a degree example and see how that runs for a year, and see whether the parameters differ at all with what we are using in ShallowWater. One question is, do we need this global model to run before we merge this PR? I am hoping this will make the shallow water model stable, but I am curious to know what people what before that happens. A long list is better than no list at all. |
Yestereday, @simone-silvestri and I had a chat and discussed the current state of the shallow water model. We found that if we ignore this PR and run the validation script on main, then it runs for 20 days before it blows up. This is different from what we saw a year or two ago, but that is the current state of affairs. We observed that there were very strong zonal velocities, We took a side step and decided to study an acquaplanet case on main. We removed the continents and made the bathymetry flat. Below are the plots of Some good news is that we do see the ACC and it's moving in the right direction! @navidcy We also see that the velocites are much more reasonable, around One odd bit is the height field seems to be I am including an animation of Thoughts? near_global_shallow_water_1440_600_surface_halo.mp4 |
How is the simulation forced? I don't understand why it blows up. Is there a violation of CFL? |
It is being forced by realistic wind stress and bathymetry. The same forcing exactly (I believe) that's used in the hydrostatic model. The height field goes to zero and that causes the numerical instability. We are not using the time step wizard but even though the winds are over 25 m/s, it doesn't go unstable until after 20 days. A surprise perhaps. The high velocities happen in the most polar regions, where the depths tend to be shallow. |
To recap: When we have an acquaplanet with a flat bottom, the shallow water nearly global quarter of a degree model is stable for at least two years (not shown here). This is in contrast to when we included coastlines where it becomes unstable after about 20 days. To study an intermidate situation, where we have continents but with a flat bottom, I set the depth of the ocean to be 6 km for no especially good reason. This simulation becoems unstable after 60 days. This suggests to me that the problem is with the immersed boundary, where we include the coastlines. @simone-silvestri near_global_shallow_water_1440_600_surface.mp4 |
Ok, closing in! The last test is without an immersed boundary but with a bathymetry (you need to cap the bathymetry to something like -50m) If this works then we can focus all the attention to the immersed boundary |
Just to be clear, the test I posted today was with an immersed boundary (continents) and a flat bottom. The previous test, what we called aquaplanet, was with no immersed boundary and boring flat topography. Could you clarify as to what you are suggesting? I was using the immersed boundary and am a bit confused. Sorry. |
I mean using no immersed boundary but a topography that does not reach the ocean surface. The immersed boundary and topography are two different things in the |
But @francispoulin is using flat topography --- this doesn't reach the surface, its flat. |
yeah, but on a flat bottom, there are no continents. I would like to see continents that interact with the height in the shallow water model but leave out the immersed boundary that crashes the simulation. This test is required to see if we need to focus only on the immersed boundary, or there are other problems in this simulation that have to do with the dynamics of a non-flat bottom |
Okay, what if I try the following that uses topography but no immersed boundary: I use can use the same topography as in the original setup but where we have continents (NaNs) I set the depth to be -100m. Is that what you have in mind? |
yeah, I was suggesting that. The original topography limited to a "below-water" value. I guess -100 is ok. |
I tried setting the bathymetry using the following lines of code. Anything look obvously wrong?
I can confirm that the minimum depth is -100 m and the maximum depth is -6 km and the plot of bathymetry looks good to me, see below. Unfortunately, the simulation becomes unstable within an hour because of the depth becoming negative. Maybe because of the very strong gradients happening where coastlines are known to occur? |
Closes #3051
This revised the shallow water tendencies, adds a new valiation example with 1D topography and starts to get the global model working.
I'm happy to say that the near global simulation is now running!