Coordinated Restore at Checkpoint #631
Comments
|
Why would a framework need to close files on checkpoint? I have extensive experience dealing with checkpoint and restore with an OSGi framework (Equinox) with Liberty InstantOn: https://openliberty.io/blog/2023/09/26/spring-boot-3-instant-on.html No changes were needed to the Framework to deal with using CRIU (which Azul CRaC uses under the covers) to checkpoint the Java process. I've had to make plenty of changes to Open Liberty to prepare and restore the runtime above the framework. But nothing in the framework needed to change to make this work. |
|
Nobody should put a dependency on jdk.crac/javax.crac at this point. That would make it impossible to use other checkpoint/restore solutions. A vendor neutral dependency would be on org.crac. That is what the Spring project depends on so that it can run on any JVM (not just Azul JVM). That is also something I've also implemented on top of Liberty InstantOn for the Spring Boot 3 enablement shown in https://openliberty.io/blog/2023/09/26/spring-boot-3-instant-on.html |
|
just some additional info: https://github.com/CRaC/docs/blob/master/STEP-BY-STEP.md |
|
I today had a short discussion with @HanSolo and found out that there is no JSR for CRaC but a JEP is in preparation.
I don't know. If I try to create a checkpoint with the Azul CRaC feature I get several exceptions like this: This is also documented here: Maybe someone from the OpenJDK or Azul team could answer this? @HanSolo could you answer this or add someone from the CRaC team to this ticket so we can get more technical details? This would definitely help in adding support for creating checkpoints via CRaC for an OSGi application. |
|
To answer the question about "Why would a framework need to close files on checkpoint?" -> When creating a checkpoint, the state of the JVM incl. the application will be stored to a set of files. If the application has a file open at the time of the checkpoint, it will assume that this file is still open after the restore procedure. But if no one re-opens this file after restore, the application will crash. This is the reason for the Resource interface which has the two methods beforeCheckpoint() and afterRestore(). In case you use a framework which might have itself files or any kind of socket connection open, the framework also needs to close and re-establish these connections after restore. This usually cannot be done by the developer of the application but the framework needs to support CRaC (implement the Resource interface at the right places) to be able to close and re-establish these connections independent from the application that is using the framework. |
|
Files the framework has open for loading bundle content there is an assumption that the underlying file content is constant from the checkpoint to the restore. Any scenario that attempts to change the content of such bundle files from checkpoint to restore will be invalid. Checkpoint should not fail for scenarios where the file cannot change from checkpoint to restore. CRIU itself allows for this kind of scenario with no issues. It will fail to restore if such files change content from the checkpoint to the restore. I don't want to add something to the OSGi core specification that forces a framework to do prepare/restore operations on resources that must not and cannot change between checkpoint and restore. That will result in many complicated questions to answer around what the framework must do when the bundle file content is missing or changed between the checkpoint and restore. |
|
Why should these files change between checkpoint and restore? If we checkpoint the JVM incl. Frameworks and Apps, after a restore the Framework and Apps will assume these files are still open, because for them nothing changed between checkpoint and restore. But the connection to the files is gone because at checkpoint the JVM was shutdown. When re-opening the files after restore everything is fine. You don’t need to force anyone to do something, it’s more a support for checkpoint/restore on framework level. Same thing was implemented in Micronaut and currently in Spring 6.1. This is to help users to make use of checkpoint/restore, not to force anyone to do something. |
|
So, what I understand so far is:
@HanSolo statet:
This would imply that there is a technical limitation to also have the open file handling to be done on checkpoint and restore. @tjwatson statet:
That sounds to me like there is no technical limitation in handling the open files on checkpoint and restore. The question now is, which direction to go next. Is the statement actually correct that it technically is possible to handle open files in checkpoint creation and restore? Would that then be an option to be added to the Azul/OpenJDK CRaC support? Or is Equinox automatically able to deal with the scenario that the connection to an open file is gone an recreates it? That would actually mean that we would need some option in the Azul CRaC VM to no fail on open files. Maybe configurable somehow? Adding @jjh-aicas to the discussion, as he has some experience in that topic as far as I understood from talking to him. BTW:
IIUC the CRaC support using the org.crac API there is more about automatically creating a checkpoint at a certain point of time when it makes sense from the framework point of view. Not about open resources. Right? Or was there any other support added? |
|
Related to Spring/Micronaut integration: If you use resources managed by Spring for example, it will automatically close and re-open them for you without the need to implement the Resource interface on your own. |
|
If OpenJ9 InstantOn is working it should also be possible to make it work with the OpenJDK implementation. IBM uses the exact same idea and also use CRIU. Will check with them. |
There is no technical limitation in CRIU. @ymanton can confirm as a contributor to CRIU from OpenJ9. |
|
What I think is more interesting to talk about from an OSGi specification POV is what should happen if a service is registered with the service interface I can imagine an implementation (in a bundle outside of the framework) tracking Then we can talk about how other things should behave in the checkpoint/restore scenario for other resources like the ones managed by the data source service or the JPA service specification. |
I think most class loaders will keep a JarFile open for performance reasons vs. open/close it each time a class is read. That being said, Equinox does keep a default (set to 100) number of JarFiles open. If more then 100 bundles are installed then it will start closing JarFiles that are least used. This was necessary to reduce the number of file handles necessary to install something like 1000s of bundles at the same time. Some OS'es (e.g. Linux) can put limits on the number of file handles it will allow a process to keep open. So it is not technically impossible for the framework to close the bundle resources. But for checkpoint/restore scenarios that should not be required if we assume the content of the file must not change between checkpoint/restore. But there will be loads of issues trying to implement a strategy that requires the framework to close all resources on
Both options would lead to randomly failing. One will be a random failure to load a class, leaving the process in a bad state before checkpoint. The other will randomly fail checkpoint. |
This is correct; processes having open files can be checkpointed and restored. Aside from special cases, the files themselves are not part of the checkpoint, only some metadata about them is remembered (path, size, etc) and CRIU will check on restore that open files are still in the same locations and sizes haven't changed and then re-open, or else abort the restore. |
|
@ymanton are open sockets handled the same way by criu? Or could that be an issue? |
I believe open sockets can be handled by CRIU if you are using certain options (e.g. We avoid having any open sockets before checkpoint. |
|
@fipro78, sockets can be re-established, but it depends on the type of socket and usually requires additional steps. Listening sockets are the simplest and can usually be restored transparently. Connected sockets are a different story. The easiest case in this category is sockets that are not bound to specific devices and where the protocol supports (and users expect) broken connections, where CRIU can just break the connection and both sides can handle it as usual. Dealing with such sockets by instead trying to reconnect them, or sockets bound to specific devices, or with other kinds of sockets that users don't expect to break (e.g. local, netlink) is more involved. As @tjwatson suggests it's easier to sidestep these issues by closing all sockets across checkpoint/restore. Some of what CRIU can do makes sense for process migration, which deals with a single logical instance of the program, but not for startup improvement, where we want to restore multiple instances of the checkpointed program. |
|
Hi all, First of all, thanks for the try-out! In CRaC, those CheckpointOpenResourceException are built to provide the safety of the checkpoint image, so there should be no implicit dependencies on the state of OS or the filesystem. CRaC does not distinguish sockets and files, and reports any possible problem at the checkpoint time, so you won't find your image does not restore later because of an implicit dependency. The exception means there is an open file, and CRaC implementation does not have any assumptions about it. So, for example, if the file is gone for some reason, the VM will just be unable to restore . The common rule is indeed to close such files for checkpoint, and open then again on restoration. Then you'd be able to handle all the situations: the file is gone, the file has been changed, etc. For example, if this is a configuration file, you may re-read and re-apply the config. But it's clear that in some cases it's useful just to assume file will be always there, as provided by containers. So, you can communicate this via command line settings, see FD Policies doc. The functionallity will be available since crac-17+6, and will be included in zulu17.46 and zulu21.30. You can specify automatic closing and reopening the file, and if, for example, the file is gone, you'll get the java exception that you'll be able to handle. You can also can configure The org.crac provides the Resrouce interface that allows arbitrary handling of resources, including sockets and files on checkpoint. Usually the handling is pretty simple. AFAIU you already maintain the cache of 100 jars and there is a mechanism to keep them under the limit. In similar cases, it was enough to lower limit temporarily to 0, that will effectively close every jar. Probably, you may also need to block threads which are trying to load new classes from opening jars while checkpoint is in progress. All of this should work only during checkpoint and restoration. |
|
Hi Anton, thank you a lot for this explanation. Estecialy the FD Policies will help a lot. |
|
@AntonKozlov I noticed on https://cdn.azul.com/zulu/bin/ that the versions you mentioned are not yet available. When will those versions be available for testing? And btw, is it possible to create a custom JRE via jlink that supports CRaC, and create a checkpoint with that? This would make it possible to create the smallest possible runtime for an application with the fast startup via checkpoint. |
|
The builds are expected, hopefully, this week. Jlink can be made to work, but unfortunately a workaround is required https://github.com/CRaC/example-lambda/blob/master/crac-steps.sh#L18. In the rest, jlinked image works fine. This is not an excuse for not fixing it :) Sorry, the problem was not tracked. Now we'll look for the fix. |
|
I think basically from the Framework POV we need some kind to "join at a checkpoint" so an agent can ask the Framework to join (and other items like DS as well) to have some kind of barrier where it stops, then a Checkpoint can be created and then everything goes on (probably to the next checkpoint). Something similar is already described here: https://docs.osgi.org/specification/osgi.cmpn/7.0.0/service.coordinator.html |
stbischof
changed the title
How would the framework know where to stop? I'm not following this idea. If someone wants to drive a discussion on requirements we should have them consider following the requirements process: https://github.com/osgi/osgi/blob/main/CONTRIBUTING.md#requirements-discussion |
|
@AntonKozlov I have created a small setup with scripts to execute the multi-step process for creating a Docker container with a checkpoint. After running the I followed the documentation Running CRaC in a Virtualized Environment (Docker) to create the images. If I use Issue 1: Start the container with the checkpoint data via You will see the following error: So it looks like CAP_CHECKPOINT_RESTORE is also required on restore. The documentation doesn't say that this is necessary. Issue 2: Start the container with the checkpoint data via You will see the following error: Even adding I have no idea what the issue might be. If I use Issue 3: Using Checking the When I try to start the container creation with Podman I get the following error: I have no idea what is going wrong. Is it an issue on Windows? Is it not possible to create a checkpoint for my app? Is something else missing? The attached zip archive contains four folders. The crac_ folders contain the Dockerfiles and the scripts to build the containers. I tried to run the scripts from Windows directly and from an Ubuntu 20.04 WSL. Maybe someone can test this on Linux or Mac? Or maybe someone sees what I have done wrong? P.S. the criu_ folders contain the same example using OpenJ9 CRIU Support. There I can't even create the checkpoints. @AntonKozlov |
|
Hi @fipro78, with small modifications, the test runs on linux-x64. But we'll check it on WSL, thanks for the ready-to-use example. --- a/CRaC_Test/crac_executable/create_checkpoint.sh
+++ b/CRaC_Test.new/crac_executable/create_checkpoint.sh
@@ -1,7 +1,8 @@
# sleep for 5 seconds to ensure everything is ready
sleep 5
# get the PID of the java execution
-PIDJ=$(ps -ef | grep -v grep | grep java | awk '{print $2}')
+# PIDJ=$(ps -ef | grep -v grep | grep java | awk '{print $2}')
# create the checkpoint
+PIDJ=app.jar
echo "execute: jcmd $PIDJ JDK.checkpoint"
-jcmd $PIDJ JDK.checkpoint
\ No newline at end of file
+jcmd $PIDJ JDK.checkpoint
diff --git a/CRaC_Test/crac_executable/start.sh b/CRaC_Test.new/crac_executable/start.sh
index dde81af..e905908 100644
--- a/CRaC_Test/crac_executable/start.sh
+++ b/CRaC_Test.new/crac_executable/start.sh
@@ -3,4 +3,4 @@
# ( echo 128 > /proc/sys/kernel/ns_last_pid ) 2>/dev/null || while [ $(cat /proc/sys/kernel/ns_last_pid) -lt 128 ]; do :; done;
# java $JAVA_OPTS $JAVA_OPTS_EXTRA -jar app.jar "$@"
# replace the line above with the following line to open the app with a console instead of a socket
-java $JAVA_OPTS $JAVA_OPTS_EXTRA -Dgosh.args="" -jar app.jar "$@"
\ No newline at end of file
+exec java $JAVA_OPTS $JAVA_OPTS_EXTRA -Dgosh.args="" -jar app.jar "$@"In order to restore without additional capabilities, java have to be PID 1 process. We'll improve the docs with a note. The example prints the TTY error, which can be neglected It seems CRIU attempts to retore TTY setting, but it will be more correct to do this in CRaC JVM. This indeed does not directly relate to OSGi project. So we can continue the discussion via support@azul.com. Or should we create a repo with an example under github.com/crac/...? |
|
@AntonKozlov thanks for looking into this. I will try your modifications also in my prepared setups. I am totally open about the further discussion. Once it works I wanted to blog about it anyway. What would you like to be in the repository? |
I think a simple demo would be more than enough. The archive you posted looks like a good starting point. We'd be able add some CI, documentations, etc incrementally. |
|
@AntonKozlov |
Dirk Fauth @fipro78 tries to bring in crac Support.
Main issues seems to be
open File Handlesin the create Checkpoint stage.Why are they open, and could they be closed? and on restore be opened?
What must be specified (hooks?) here that the Frameworks must support?
What are the implications to the Feature Launcher #386
https://openjdk.org/projects/crac/
https://docs.azul.com/core/crac/crac-introduction
The text was updated successfully, but these errors were encountered: