execute in OpenResty eventually returns empty sets for valid queries

[jaw-sh]

I am new to using Lua and am experimenting with using OpenResty+Scylla to achieve a session check. It works well, except in production use after several hours I get a bizarre issue.

With the code below, the results of this .execute() eventually start returning empty sets. There is no error. It just starts lying and saying that no rows return, which results in a condition identical to a real session failure.

To be clear, this code works. It is in production environments, after several hours, that all queries start silently failing and returning #rows==0 for no reason.

I'm hoping there is an obvious answer to this. When OpenResty inits, it creates a connection to a single local node (Scylla binds to the Docker host IP and the OpenResty instance is dockerized). At no point does the connection close. I am wondering if this is an issue with never restarting the connection?

-- accept a uuid string which may be nil and check for its presence in Scylla.
function _M.check_session(uuid)
  local rows, err, cql_code = _M.execute("SELECT * FROM sssg.sessions WHERE id = ?", { uuid })

  if err then
    ngx.log(ngx.ERR, "[SSSG] Failed to query sessions table: ", err)
    error(err)
  end

  if not rows then
    ngx.log(ngx.ERR, "[SSSG] [", cql_code, "] Failed to query uuid: ", err)
    return false
  end
  
  if #rows == 0 then
    ngx.log(ngx.ERR, "[SSSG] Found no session.")
    return false
  end

  if not rows[1].valid then
      ngx.log(ngx.ERR, "[SSSG] Found invalid session: ", inspect(rows))
    return false
  end

...

[jaw-sh]

I believe this issue has something to do with ulimit. Increasing the number of open files available to the docker container running OpenResty resolved the issue on one device. The other device I was not able to resolve this issue one but I believe it probably had to do with its disks not being SSD.