Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allocated memory issue #204

Open
netwire88 opened this issue Dec 3, 2021 · 3 comments
Open

allocated memory issue #204

netwire88 opened this issue Dec 3, 2021 · 3 comments

Comments

@netwire88
Copy link

netwire88 commented Dec 3, 2021

I have the following in my Gemfile

ruby '2.7.5'
gem 'rails', '~> 6.1', '>= 6.1.4'
group :development do
  gem 'derailed_benchmarks', '~> 2.1', '>= 2.1.1'
end

However, when I run bundle exec derailed bundle:objects production, I see the following

allocated memory by gem
-----------------------------------
  43060426  dead_end-1.1.7

and when I run bundle exec gem dep -R
I see that dead_end is only used by derailed_benchmarks

Gem dead_end-3.1.0
  Used by
    derailed_benchmarks-2.1.1 (dead_end (>= 0))

I'm a bit confused why:

  1. the gem's showing up in production
  2. it seems to be creating huge memory footprint upon launching of app

Any thoughts?

I'm not sure if it's related to this issue action_mailbox/engine still loads even after not requiring it in application.rb where @andrehjr says it's derailed_benchmark's use of require "action_controller/railtie"

@schneems
Copy link
Member

schneems commented Mar 8, 2022

the gem's showing up in production

Because it's added to derailed in development. So when you're running in production for real (not just simulating it with derailed) you also won't have dead_end.

it seems to be creating huge memory footprint upon launching of app

I've described the issue here ruby/syntax_suggest#124 (comment). The TLDR is that any gem that patches into require (such as dead_end or bootsnap etc) will show up here as derailed ALSO patches into require and that's how it gets it's info.

If it's causing confusion we could perhaps introduce an env var into dead_end that would optionally disable the monekypatching behavior. We could also move it from a general derailed require to a per-task require.

@schneems
Copy link
Member

schneems commented Mar 8, 2022

Also that output is from memory_profiler. One question to ask is: why isn't this a problem for bootsnap? Did get it special cased somehow?

@beporter
Copy link

Just ran into this myself, and was admittedly confused when the single largest consumer of "allocated memory by gem" by a long way was (indirectly) from the package I just installed to examine the memory itself (derailed_benchmark).

$ bundle exec derailed bundle:objects
Measuring objects created by gems in groups [:default, "production"]
Total allocated: 94891342 bytes (956756 objects)
Total retained:  19576854 bytes (177325 objects)

allocated memory by gem
-----------------------------------
  60670886  dead_end-3.1.1
   6590809  mime-types-3.3.1
   4019827  json-2.5.1
   3301861  set

# **SNIP**

Thankfully this GitHub issue came up relatively quick from searching, but it's not a particularly great first-time experience to immediately get led on a (short) wild goose chase inflicted by this very gem. If there is a way to suppress derailed's own dependencies from the output (when they are not otherwise used by the host application of course), I think I'm in favor of that.

schneems added a commit to ruby/syntax_suggest that referenced this issue May 23, 2022
Currently dead_end works by monkey patching require. This causes confusion and problems as other tools are not expecting this. For example zombocom/derailed_benchmarks#204 and #124. This PR utilizes the new SyntaxError#detailed_message as introduced in ruby/ruby#5516 that will be released in Ruby 3.2.

That means that developers using dead_end with Ruby 3.2+ will experience more consistent behavior.


## Limitations

As pointed out in #31 the current version of dead_end only works if the developer requires dead_end and then invokes `require`.

This behavior is still not fixed for Ruby 3.2+ 

```
$ ruby -v
ruby 3.2.0preview1 (2022-04-03 master f801386f0c) [x86_64-darwin20]
$ cat monkeypatch.rb
  SyntaxError.prepend Module.new {
    def detailed_message(highlight: nil, **)
      message = super
      message += "Monkeypatch worked\n"
      message
    end
  }

  # require_relative "bad.rb"
  # Note that i am commenting
  # out the require, but leaving
  # in the monkeypatch
⛄️ 3.2.0 🚀 /tmp
$ cat bad.rb
  def lol_i-am-a-synt^xerror
⛄️ 3.2.0 🚀 /tmp
$ ruby -r./monkeypatch.rb bad.rb
bad.rb:1: syntax error, unexpected '-', expecting ';' or '\n'
  def lol_i-am-a-synt^xerror
```

Additionally we are still not able to handle the case where a program is streamed to ruby and does not exist on disk:

```
$ echo "def foo" | ruby
```

As the SyntaxError does not provide us with the contents of the script.

```
$ echo "def foo" | ruby
-:1: syntax error, unexpected end-of-input
def foo
```
schneems added a commit to ruby/syntax_suggest that referenced this issue May 23, 2022
Currently dead_end works by monkey patching require. This causes confusion and problems as other tools are not expecting this. For example zombocom/derailed_benchmarks#204 and #124. This PR utilizes the new SyntaxError#detailed_message as introduced in ruby/ruby#5516 that will be released in Ruby 3.2.

That means that developers using dead_end with Ruby 3.2+ will experience more consistent behavior.

## Limitations

As pointed out in #31 the current version of dead_end only works if the developer requires dead_end and then invokes `require`.

This behavior is still not fixed for Ruby 3.2+

```
$ ruby -v
ruby 3.2.0preview1 (2022-04-03 master f801386f0c) [x86_64-darwin20]
$ cat monkeypatch.rb
  SyntaxError.prepend Module.new {
    def detailed_message(highlight: nil, **)
      message = super
      message += "Monkeypatch worked\n"
      message
    end
  }

  # require_relative "bad.rb"
  # Note that i am commenting
  # out the require, but leaving
  # in the monkeypatch
⛄️ 3.2.0 🚀 /tmp
$ cat bad.rb
  def lol_i-am-a-synt^xerror
⛄️ 3.2.0 🚀 /tmp
$ ruby -r./monkeypatch.rb bad.rb
bad.rb:1: syntax error, unexpected '-', expecting ';' or '\n'
  def lol_i-am-a-synt^xerror
```

Additionally we are still not able to handle the case where a program is streamed to ruby and does not exist on disk:

```
$ echo "def foo" | ruby
```

As the SyntaxError does not provide us with the contents of the script.

```
$ echo "def foo" | ruby
-:1: syntax error, unexpected end-of-input
def foo
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants