Let commands in %startscript section ouput to terminal when starting an instance

[Heng-Zhou]

Is your feature request related to a problem? Please describe.
Ticket #2501

Describe the solution you'd like
I believe most of users want to see the output message of the running command in terminal real-time. So please let the commands in %startscript section output to the terminal, at least give users an option like --no-redirect to see the message in terminal when starting an instance.

Again, please don't close my ticket in such a hurry. Please let other people listen to me. I just want to ask Singularity to work the way a normal user would expect. That you think unreasonable does not mean it is really unreasonable.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[dtrudg]

Please could you describe in detail exactly what behaviour you would expect? Specifically:

• What you believe should happen if an instance is started with --no-redirect, and the terminal where it was started is then closed.
• If the instance should still run in the background. This would mean that any output would appear in the terminal in a way which interferes with any other work that continues there.

Due to the way instances are implemented, it is not necessarily trivial to implement a --no-redirect.

Because you can simply tail -f the log files that the instance writes to, it's unlikely that this feature will be added unless it is requested by a lot of users, or the code changes are contributed in a PR by a member of the community.

[Heng-Zhou]

Thank you for your reply. As for your concerns:

1. The main idea of my intention is that the behavior meets common users' expectation. That is, the output should be what the commands display when they are run normally on Linux instead of in a container. So, specifically, I hope Singularity can forward the output to both the terminal and log file like tee, no matter the terminal is still there or closed. Since the gist is for the behavior to meet a normal user's expectation, if the terminal is closed by the user, the user should not expect to see any output in a terminal that does not exist.

I agree that --no-redirect is a confusing name of the option, which suggests not redirecting output to log file. What I want is instead forward the output to both terminal and redirected file, so maybe --tee is a better name, because it is reasonable to assume that most Linux users know what tee command does.

2. For this concern, we can simply respect the behavior of the commands in the %startscript section when they are run normally in Linux (not in container), because that meets users' expectation. If the commands output something on terminal that interferes with the current work that continues there, the user should expect that effect and does not complain, because that is the native behavior of the command. The duty of Singularity is not regulate or improve the existing (bad) behavior of commands, but truthfully mimic the behavior of the command. Just let it be if interference happens, as we often see when a booting just finishes (some remaining messages of services interfering with login prompt).

But a good news is that the interference is not that common. What users want to see (or expect to see) is usually a banner of the software (like name and version), confirmation message that the service is started successfully or failed, as well as some important configuration information like port being listened. Such a message are stdout and is output to terminal only at the beginning of the service. Likewise, if the service exits, there could be some stdout message showing that the service exits successfully, as well as possibly what resources are released. Both these cases will not interfere with terminal. What you are concerning about is the output of service in the middle of other work running on terminal. But if there are any such messages, it is highly likely these messages are errors which should be displayed in stderr, not in terminal (aka stdout) to cause any interference. Of course, there could still be stdout messages in the middle from a service, like a web server which reports requests it receives. But again, that's the behavior of the command when it's running normally in Linux (not in container). The user should expect that interference and hence no complaints. For cases like this, I usually leave the terminal open and open a new terminal to do other jobs. Or, if the user really wanna disable output, s/he can remove the --tee option I suggested above. You can set the default to be no --tee.

I think forwarding messages destined to stdout to both stdout and a file is not hard. It is an average exercise of Unix programming, cf. e.g., "Advanced Programming in the Unix Environment", Third Edition. By W. Richard Stevens and Stephen A. Rago, Addison Wesley. 2013.

[dtrudg]

Thanks for the detail.

1. The main idea of my intention is that the behavior meets common users' expectation. That is, the output should be what the commands display when they are run normally on Linux instead of in a container.

Our expectation of how Singularity is used is that singularity run and the %runscript would be used to get this behaviour. Starting an instance is, instead, explicitly for something in the background, long running and detached, that you don't interact with directly.

Note that you can do a singularity run mycontainer.sif & as a kind of half-way approach.

I can see your point for some cases, but I don't think it's necessarily the most common expectation for an instance. For those who might read this and are also familiar with Docker, instances are like starting a docker container without -it... there is no direct output. You have to inspect the logs.

We can leave this issue open to see if it attracts support from others, or if anyone is interested in contributing the code change to implement the feature.

I agree that --no-redirect is a confusing name of the option, which suggests not redirecting output to log file. What I want is instead forward the output to both terminal and redirected file, so maybe --tee is a better name, because it is reasonable to assume that most Linux users know what tee command does.

Agreed ... --tee might be a good one.

I think forwarding messages destined to stdout to both stdout and a file is not hard. It is an average exercise of Unix programming, cf. e.g., "Advanced Programming in the Unix Environment", Third Edition. By W. Richard Stevens and Stephen A. Rago, Addison Wesley. 2013.

Unfortunately it's not as easy dealing with stdout / stderr of containers as simple processes. The container has to start in the background, completely detached from the terminal, with some kind of attachment to the log files, that will be maintained at all times. Then, thesingularity process in the foreground has to transition into a process that attaches to the container stdout / stderr to show it on the terminal also. When we setup and start a container, there are several processes involved, across multiple exec boundaries, namespaces etc. There is a lot of complexity. The create container->start container in background->attach from CLI pattern is what's implemented by all the container runtimes that support background containers similar to Singularity instances, and offer the ability to display the output to a terminal.

We actually have some code that does pieces of this flow for the OCI-mode (but not the native runtime / instances) e.g. here: https://github.com/sylabs/singularity/blob/main/internal/pkg/runtime/launcher/oci/oci_attach_linux.go

For instances, we currently just replace the Stdout / Stderr file descriptors with file descriptors for log files. There is no persistent process redirecting output through a buffer, nor a socket that we can attach to - so it's actually a considerable amount of work needed to implement this. https://github.com/sylabs/singularity/blob/main/internal/pkg/instance/instance_linux.go#L271

[Heng-Zhou]

I'd close it.