deregistering process issues? #108
Comments
|
Hey @davidpan, I can't reproduce this 😕 Processes are correctly deregistered (deleted from |
|
Thanks for the reply. run bin/rails solid_queue:start run bin/rails solid_queue:start --trace environments: MySQL database version is: 8.0.35-0ubuntu0.22.04.1 (Ubuntu). Project-related environments: Project-related gems: solid_queue.yml file: default: &default
dispatchers:
- polling_interval: 1
batch_size: 500
concurrency_maintenance_interval: 300
workers:
- queues: "chrome_single"
threads: 1
processes: 1
polling_interval: 1
development:
<<: *default
test:
<<: *default
production:
<<: *defaultjobs : self.queue_adapter = :solid_queue
queue_as :chrome_single
limits_concurrency to: 1, key: ->(id) { id }, duration: 5.minutesThe job is accessing web pages to collect content using chrome's headless mode. |
|
Thanks! That's very helpful as without the error I had no idea where to pull the thread from 😅 I've tried to reproduce this again, with a job that sleeps, but no luck. Could you try that again, but with a longer shutdown timeout, that you can configure as: config.solid_queue.shutdown_timeout = 60.secondsand right after issuing And copy the output? |
|
After configuring for 60 seconds, retrying multiple times does not reproduce the error. run log: SHOW ENGINE INNODB STATUS: SELECT * FROM information_schema.innodb_trx: |
|
Hey @davidpan, no, it shouldn't make a difference with respect to the processes registration and deregistration. That would only affect jobs in-flight, and in your case, you're still using up all the time: My suspicion is that whatever transaction was open and locking your FWIW, I didn't touch |
|
Undo the previous configuration change: # config.solid_queue.shutdown_timeout = 60.seconds Retrying 2 times reproduces the problem. @rosa Running logs on the command line: SHOW ENGINE INNODB STATUS: SELECT * FROM information_schema.innodb_trx: |
|
A test job that can be used to reproduce the problem. Press Ctrl-C when prompted. class CtrlCTestJob < ApplicationJob
self.queue_adapter = :solid_queue
queue_as :chrome_single
limits_concurrency to: 1, key: ->(i) { i }, duration: 5.minutes
def perform(i)
# Do something later
pp "start #{i}...."
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument('--headless')
web_driver = Selenium::WebDriver.for(:chrome, options:)
web_driver.manage.timeouts.implicit_wait = 20
web_driver.get "https://www.bing.com"
web_driver.manage.window.resize_to(1024, 1024)
pp web_driver.title
pp "press Ctrl-C "
sleep(10)
web_driver.close
web_driver.quit
end
endlog: |
|
Thank you! That's very helpful, I'll dig into these. |
|
@rosa I too am getting the "Error deregistering process" error. This has also resulted in unique constraint errors on claimed_executions when trying to start solid_queue. I still actually have the lingering processes in my database so I thought I'd do some poking around. When trying to destroy a process I get the following: This in my local machine, a MacBook Pro. Using Postgresql 14 |
|
Oh, thank you so much @jmarchello! I think I know the problem, I'll try to work on a fix tomorrow, I didn't have time last week in the end. Thanks all for your patience 🙏 |
|
@rosa I may have realized that my particular issue was entirely my fault :). We use UUIDs for our ids so I had edited the initial solid_queue migration to use UUIDs, but I missed the |
|
Ahhhh! Thank you! 🙏 The infinite loop indeed didn't seem related to the error I think I have here (a race condition with the supervised processes and the supervisor trying to deregister) but it was interesting nonetheless! I was going to ask whether you perhaps had somehow a supervisor supervising itself 😅 |
|
@davidpan I finally could reproduce your issue, in my case I had to set config.solid_queue.shutdown_timeout = 2.secondsI'll work on a fix! |
|
Comment the following previously added. #config.solid_queue.shutdown_timeout = 60.secondsModify gem source to github: gem 'solid_queue',:github => 'basecamp/solid_queue'execute bundle install Running the test several times, pressing Ctrl-C before web_driver.get accesses the web address has a higher probability of success, and if you press Ctrl-C after accessing the web page, it has a higher probability of failure. A sample error log is shown below: |
|
Thanks @davidpan! Could you try doing this? Looks like you're still running the code without the fix in these logs above. |
|
@rosa Sorry, didn't realize it hadn't been merged yet. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After running 'bundle exec rake solid_queue:start' for a while and stopping it with Ctrl-C, I noticed that the record of stopped processes in the 'solid_queue_processes' table was not deleted.
Upon restarting, I received an error message stating 'deregistering process 1 - {}' and the corresponding record still existed in the table.
The text was updated successfully, but these errors were encountered: