Later interval doesn't work well with RestRserve and doesn't run with Rscript

[dereckmezquita]

Edit: I updated the title of my issue. Now I'm looking for a way to run later and RestRserve together and communicate between the two. Please skip to this comment:

#179 (comment)

I will leave my previous comments for context.

---

Hello, I'm still learning about later and event loops. I would appreciate any help.

I wrote this script to test later and create a "private event loop". I'm still learning about these. Is there any documentation somewhere with examples of how to use them? All I found was the help pages but with no examples - toy use case demos would be very helpful - I want to use it to get data from an API and run a REST API without blocking the global even loop (the REST API).

I'm more familiar with just using later::later by itself but not with private event loops.

Anywho, I wrote this script that runs well in interactive mode but only opens and closes with no printing if run with Rscript. In the end this script I want to write with the loop is meant to be server side code so I have to be able to run it using Rscript.

I plan on using this in a RestRserve server, so I need this to run without blocking the global event loop - hence, I started looking into private event loops.

I tried to keep the global loop open by sleeping for 10 seconds; the initial print of "Hello world!" worked, but not my loop.

#!/usr/bin/env Rscript
box::use(later)

print("Hello World!")

# Define the function to be executed
print_hello <- function() {
    print(paste("Hello World:", Sys.time()))
    later$later(print_hello, delay = 0.1)  # Schedule the next execution in 100 ms
}

# Create a private event loop
private_loop <- later$create_loop()

# Run the function in the private event loop
later$with_loop(private_loop, {
    later$later(print_hello, delay = 0.1)  # Schedule the initial execution in 100 ms
    later$run_now()  # Run the event loop
})

# pause
Sys.sleep(10)

[dereckmezquita]

For some more context here is what I'm trying to do. This is an example of a toy REST API using RestRserve. Note that RestRserve has to be run with Rscript and so should plumber another REST API framework.

#!/usr/bin/env Rscript
box::use(RestRserve)
box::use(later)
box::use(jsonlite)

# Function to fetch and prepare data
fetch_data <- function() {
    print("New data...")
    # Fetch data from external API
    data <- rnorm(100, 0, 1)
    
    # Save data to a file or a database
    saveRDS(data, "data.rds")
    
    # Schedule the next data fetch
    later::later(fetch_data, delay = 2)
}

# --------------------------------------------
# private event loop so as not to block RestRserve
private_loop <- later$create_loop()

later$with_loop(private_loop, {
    later$later(fetch_data, delay = 2)
    later$run_now()
})
# --------------------------------------------

# Function to serve data
serve_data <- function(.req, .res) {
    # Load data from a file or a database
    data <- readRDS("data.rds")
    
    # Serve data
    .res$set_body(data)
    .res$set_content_type("application/json")
    .res$set_status_code(200L)
}

# Create an application
app <- RestRserve$Application$new()

# Add a route to serve data
app$add_route("/", "GET", serve_data)

# Start the server
server <- RestRserve$BackendRserve$new()
server$start(app)  # This will block the script

I discovered that if I ran the server in interactive mode (which I'm not supposed to do) and I can use the server and go to the end point. However the later loops are never ran. If I quit the process by doing CMD + C then the server shuts down and I can see my later process loop starting running.

Is RestRserve monopolising the global event loop somehow? How can I get this working? I do note that this seems to be a different yet related issue to my original post.

    server <- RestRserve$BackendRserve$new()
    server$start(app)  # This will block the script
{"timestamp":"2023-10-29 16:47:56.632314","level":"INFO","name":"Application","pid":64074,"msg":"","context":{"http_port":8080,"endpoints":{"GET":"/"}}}
-- running Rserve in this R session (pid=64074), 2 server(s) --
(This session will block until Rserve is shut down)
^CCaught break signal, shutting down Rserve.
[1] TRUE
Warning message:
In server$start(app) :
  Starting RestRserve app from interactive session might cause unstable work of the service.
Please consider to start the application from a shell in non-interactive mode:

> Rscript your_app.R

r$> [1] "New data..."
r$> [1] "New data..."
r$> [1] "New data..."
[1] "New data..."
[1] "New data..."
[1] "New data..."

[dereckmezquita]

After experimenting, I was able to determine that it was indeed the RestRserve server blocking the main event loop (I don't fully understand what's happening so I would appreciate someone to review my experiment/deductions @dselivanov).

I was able to put this together where I run the RestRserve on a separate process with callr::r_bg. This is not ideal and could cause problems with the multithreaded nature of RestRserve. Moreover, in my mind that is exactly what later is meant to solve (not blocking the event loop) @wch.

Does anyone know another way of doing this?

Note this script can also be written the reverse way where my "Hello world" loop is in the background process.

#!/usr/bin/env Rscript
# This works, the way this works is that the server is started in a separate R process
# and the main R process is used to run the event loop

box::use(later)
box::use(callr)

# Create a separate event loop
private_loop <- later$create_loop()

# Define the function to be called for the 2-second loop
print_hello_world <- function() {
    print("Hello world")
    later$later(print_hello_world, 2, loop = private_loop)
}

# Schedule the function to be called in the separate loop
later$later(print_hello_world, 0, loop = private_loop)

# --------------------------------------------
# Run the RestRserve backend in a separate R process using callr
server_process <- callr::r_bg(\() {
    box::use(RestRserve)

    # Create a new RestRserve application
    app <- RestRserve$Application$new()

    # Add the "echo" endpoint to the application
    app$add_get("/echo", function(.req, .res) {
        .res$set_body("hello")
        .res$set_content_type("text/plain")
    })

    # Create a RestRserve backend
    backend <- RestRserve$BackendRserve$new()

    # Start the RestRserve backend
    backend$start(app, http_port = 8080)
}, stdout = "server.log", stderr = "server.log")
# --------------------------------------------

# Run the separate event loop to print "Hello world" every 2 seconds
while (TRUE) {
    later$run_now(loop = private_loop)
    Sys.sleep(1)
}

[dereckmezquita]

Finally, I want to add that I thought about creating two separate scripts, one for my REST API and another for my interval logic. However, I need these to run together as I want to be able to receive commands from the API and pass them to my logic being run in the interval.

In short, I'm building a trading bot for academic purposes in R/shiny and Cpp/Rcpp, will publish open source :)

Edit: I am finally circling back to the original title of my post. If I use later with Rscript it doesn't run. It just opens and closes. I find I have to use a while loop to keep it open and use later::run_now. If I just use a while loop, then later never executes. This defeats the purpose of later, I could just call my logic/function in the while loop.

---

Edit: I believe the issue with RestRserve and running with Rscript not working are two separate issues, should I create a separate issue for the Rscript problem @wch?

I've done some experiments trying to keep the process open when using Rscript by using an infinite while. In my mind this should work - the while loop should run in the global event loop and then my later loops should run separately. Here's what I put together. I tried to mimic JavaScript's behaviour.

I noticed any time I used an infinite while loop even in interactive mode later does not execute. Does later only work in interactive mode?

In summary:

If I run in interactive mode later works nicely. If run in Rscript then I need a way to keep the process open artifically, however, if I use an infinite while loop then later never executes.

---

library("later")

setInterval <- function(func, interval, loop) {
    func <- rlang::as_function(func)

    loop <- later::create_loop()
    repeat_func <- function() {
        func()
        later::later(repeat_func, interval, loop = loop)
    }

    later::later(repeat_func, interval, loop = loop)
    return(loop)
}

clearInterval <- function(id) {
    later::destroy_loop(id)
}

setTimeout <- function(func, delay, loop, ...) {
    func <- rlang::as_function(func)

    loop <- later::create_loop()
    later::later(\() {
        func(...)
    }, delay, loop = loop)
}

keep_alive <- function() {
    while(TRUE) {
        Sys.sleep(Inf)
    }
}

# ----------------------------------------------
hello <- function() {
    print("Hello world.")
}

loop <- setInterval(hello, 2, later::global_loop())

# if run interactively without `while` loop then this works
# we get "Hello world." and it stops after 5 seconds
setTimeout(\(hello) {
    clearInterval(loop)
}, 5, later::global_loop(), hello = "Killing loop")

# but if run with `while` loop then we don't see "Hello world."
keep_alive()