Segmentation fault in test server #87
Comments
|
I am having the same problem. |
These are the first reports of this; I'm not aware of any outstanding/known bugs. Please do investigate, and if you can create a minimal, reproducible example, I might be able to help track down the problem. |
|
I can't reproduce the problem on Nitrous. I'll see if I can find a VM image where it happens. |
|
Unfortunately I cant reproduce it consistently either... it fails "sometimes" - 1 out of 3-4 test runs. |
|
tl;dr Start and stop a single Grocer server for all tests. I encountered this same issue when running specs on my local Ubuntu VM. I noticed that a Grocer server would be initialized and closed for the specs that were failing: server = Grocer.server(port:2195)
server.accept
...
server.closeWhen I ran each spec individually, I had no issues. But when I ran the whole suite, I was able to consistently get seg faults. So I moved the grocer server code into a before and after all block in my spec config: config.before(:all) do
@grocer_server = Grocer.server(port:2195)
@grocer_server.accept
end
config.after(:all) do
@grocer_server.close
endAnd now my specs all pass without seg faults. |
|
Thanks for the update! Apparently this solved it for you, since all your tests that use Grocer are inside one spec file. If they are spread through many spec files, |
|
I had to go with the singleton too. now the segfaults don't appear any more. |
|
Thanks for all of the investigation, all. While not ideal, I'm glad you found a work around. If anyone would like, we'd certainly love to have someone investigate the root cause so we can fix the server code and not have to rely on these workarounds. Anyone have bandwidth to take a deeper dive? |
|
I am sorry, but I will be unable to do that at the moment. I don't have much experience with APNS and iOS, and unfortunately I don't have the time to dive into this now. In case things change on my end, I will create a PR to let you know. |
|
I am encountering the same error after adding rspec tests for push notifications. I have tried the above suggestion: pusher_notification_spec.rb ...and added the spec/support/grocer.rb file All my grocer tests are in one file. When I run the file individually, there is no issues. When I run the whole suite I get a segfault every time. Does anyone have any further suggestions? At the moment my only option is to comment out the tests, as we need to run the whole suite regularly. |
|
Segmentation failure is also happening on my machine and also on CI platforms. All the backtraces have one thing in common - Linux machine (I am sure you guys have also observed it). Not sure if that is the cause of this. Does anyone have any further suggestions? |
|
@varunlalan I was going to link to some Issues I've found in other Ruby gems that point to a problem with OpenSSL and or the certificiates, but no I see you've already opened an issue on OpenSSL. 😄 I'm at a loss for where to start as I've never seen this error myself, and am unsure how to reproduce it. |
|
I have one test case when I am using mock server and one test case when I am using Apple development server. The segfault happens only when these two tests are executed together. |
|
I was having the same problem, and the only solution I came across was mocking the module Grocer
class Pusher
class_attribute :notifications
def push(notification)
self.class.notifications ||= []
self.class.notifications << notification
end
end
endI'm accessing stored notifications using |
|
Hey, I'm experiencing a crash in a Sidekiq task that pushes to notifications using APN. When this segfault happens our whole Sidekiq instance hangs, and I have to restart a worker. It looks like the relevant stack trace is this: /home/deploy/app/shared/bundle/ruby/2.2.0/gems/grocer-0.6.1/lib/grocer/ssl_connection.rb:43: [BUG] Segmentation fault at 0x007fb2886bbcf0
-- Control frame information -----------------------------------------------
c:0046 p:---- s:0210 e:000209 CFUNC :connect
c:0045 p:0216 s:0207 e:000206 METHOD /home/deploy/app/shared/bundle/ruby/2.2.0/gems/grocer-0.6.1/lib/grocer/ssl_connection.rb:43
c:0044 p:0019 s:0202 e:000201 METHOD /home/deploy/app/shared/bundle/ruby/2.2.0/gems/grocer-0.6.1/lib/grocer/connection.rb:29
c:0043 p:0014 s:0199 e:000198 METHOD /home/deploy/app/shared/bundle/ruby/2.2.0/gems/grocer-0.6.1/lib/grocer/connection.rb:55
c:0042 p:0007 s:0194 e:000193 METHOD /home/deploy/app/shared/bundle/ruby/2.2.0/gems/grocer-0.6.1/lib/grocer/connection.rb:23
c:0041 p:0045 s:0190 e:000189 METHOD /home/deploy/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/delegate.rb:87
c:0040 p:0013 s:0182 e:000181 METHOD /home/deploy/app/shared/bundle/ruby/2.2.0/gems/grocer-0.6.1/lib/grocer/pusher.rb:8
-- Other runtime information -----------------------------------------------
* Loaded script: sidekiq 3.4.1 app [0 of 25 busy] |
|
I'm experiencing the same issue as @lukaslozovski running grocer from within sidekiq. Stacktrace is identical. |
|
I have a suspicion that that error is caused by this part https://github.com/grocer/grocer/blob/master/lib/grocer/ssl_connection.rb#L39-l43 It might be that in the context of sidekiq, that piece of code gets run multiple times, and once a @ssl.connect gets called once, another call causes a segfault. My solution was to move to only sending one notification at a time, which defeats most of the power of sidekiq. |
|
So I was having the same issue where specs where running into a seg fault in CI. I just removed my acceptance tests to get passed the issue that was really slowing down our development cycle 😞 . For what I see the bug is caused by this issue in ruby https://bugs.ruby-lang.org/issues/12292. Seems we have to wait until ruby 2.4 for the issue to be resolved. In the meantime I will try to move my project to ruby 2.4.0-preview1 see if that solves the issue. |
|
@Baril27 Nice find. FWIW, the
UPDATE: We will, in fact, see it in the next Ruby point release, v 2.3.2: https://github.com/ruby/ruby/blob/ruby_2_3/ChangeLog |
|
@stevenharman Thank you for pointing that out. Will wait until the 2.3.2 release. |
I've started seeing segmentation faults during test runs of the Grocer server on our CI platform. These do not appear locally, and so far not on production either. The CI and production platform are Ubuntu 12.04 and the Ruby version is identical for all, 2.1.1.
Is this a problem in Grocer or do I have to look elsewhere?
The text was updated successfully, but these errors were encountered: