Update Ruby 2.7.0 → 3.1.5

[depfu]

Here is everything you need to know about this upgrade. Please take a good look at what changed and the test results before merging this pull request.

What changed?

Release Notes

3.1.5

3.1.4

3.1.3

Posted by nagachika on 24 Nov 2022

Ruby 3.1.3 has been released.

This release includes a security fix. Please check the topics below for details.

• CVE-2021-33621: HTTP response splitting in CGI

This release also includes a fix for build failure with Xcode 14 and macOS 13 (Ventura). See the related ticket for more details.

See the commit logs for further details.

3.1.2

Posted by naruse and mame on 12 Apr 2022

Ruby 3.1.2 has been released.

This release includes security fixes. Please check the topics below for details.

• CVE-2022-28738: Double free in Regexp compilation
• CVE-2022-28739: Buffer overrun in String-to-Float conversion

See the commit logs for further details.

3.1.1

Posted by naruse on 18 Feb 2022

Ruby 3.1.1 has been released.

This is the first TEENY version release of the stable 3.1 series.

See the commit logs for details.

3.1.0

Posted by naruse on 25 Dec 2021

We are pleased to announce the release of Ruby 3.1.0. Ruby 3.1 keeps compatibility with Ruby 3.0 and also adds many features.

YJIT: New experimental in-process JIT compiler

Ruby 3.1 merges YJIT, a new in-process JIT compiler developed by Shopify.

Since Ruby 2.6 introduced MJIT in 2018, its performance greatly improved, and finally we achieved Ruby3x3 last year. But even though Optcarrot has shown impressive speedups, the JIT hasn’t benefited real world business applications.

Recently Shopify contributed many Ruby improvements to speed up their Rails application. YJIT is an important contribution, and aims to improve the performance of Rails applications.

Though MJIT is a method-based JIT compiler and uses an external C compiler, YJIT uses Basic Block Versioning and includes JIT compiler inside it. With Lazy Basic Block Versioning (LBBV) it first compiles the beginning of a method, and incrementally compiles the rest when the type of arguments and variables are dynamically determined. See YJIT: a basic block versioning JIT compiler for CRuby for a detailed introduction.

With this technology, YJIT achieves both fast warmup time and performance improvements on most real-world software, up to 22% on railsbench, 39% on liquid-render.

YJIT is still an experimental feature, and as such, it is disabled by default. If you want to use this, specify the --yjit command-line option to enable YJIT. It is also limited to Unix-like x86-64 platforms for now.

• https://bugs.ruby-lang.org/issues/18229
• https://shopify.engineering/yjit-just-in-time-compiler-cruby
• https://www.youtube.com/watch?v=PBVLf3yfMs8

debug gem: A new debugger

A completely rewritten debugger debug.gem is bundled. debug.gem has the following features:

• Improve the debugging performance (it does not slow down the application even with the debugger)
• Support remote debugging
• Support rich debugging frontend (VSCode and Chrome browser are supported now)
• Support multi-process/multi-thread debugging
• Colorful REPL
• And other useful features like recod & replay feature, tracing feature and so on.

Ruby had bundled lib/debug.rb, but it was not well maintained and it had issues about performance and features. debug.gem replaced lib/debug.rb completely.

error_highlight: Fine-grained error location in backtrace

A built-in gem, error_highlight, has been introduced. It includes fine-grained error location in backtrace:

$ ruby test.rb
test.rb:1:in `<main>': undefined method `time' for 1:Integer (NoMethodError)
1.time {}

^^^^^

Did you mean?  times

Currently, only NameError is supported.

This gem is enabled by default. You can disable it by using a command-line option --disable-error_highlight. See the repository in detail.

IRB Autocomplete and Document Display

The IRB now has an autocomplete feature, where you can just type in the code, and the completion candidates dialog will appear. You can use Tab and Shift+Tab to move up and down.

If documents are installed when you select a completion candidate, the documentation dialog will appear next to the completion candidates dialog, showing part of the content. You can read the full document by pressing Alt+d.

Other Notable New Features

Language

• Values in Hash literals and keyword arguments can be omitted. [Feature #14579]
  • {x:, y:} is a syntax sugar of {x: x, y: y}.
  • foo(x:, y:) is a syntax sugar of foo(x: x, y: y).
• Pin operator in pattern matching now takes an expression. [Feature #17411]

Prime.each_cons(2).lazy.find_all{_1 in [n, ^(n + 2)]}.take(3).to_a
#=> [[3, 5], [5, 7], [11, 13]]

• Parentheses can be omitted in one-line pattern matching. [Feature #16182]

[0, 1] => _, x
{y: 2} => y:
x #=> 1
y #=> 2

RBS

RBS is a language to describe the structure of Ruby programs. See the repository for details.

Updates since Ruby 3.0.0:

• Generic type parameters can be bounded. (PR)
• Generic type aliases are supported. (PR)
• rbs collection has been introduced to manage gems’ RBSs. (doc)
• Many signatures for built-in and standard libraries have been added/updated.
• It includes many bug fixes and performance improvements too.

See the CHANGELOG.md for more information.

TypeProf

TypeProf is a static type analyzer for Ruby. It generates a prototype of RBS from non-type-annotated Ruby code. See the document for detail.

The main updates since Ruby 3.0.0 is an experimental IDE support called “TypeProf for IDE”.

The vscode extension shows a guessed (or explicitly written in a RBS file) method signature above each method definition, draws a red underline under the code that may cause a name error or type error, and completes method names (i.e., shows method candidates). See the document in detail.

Also, the release includes many bug fixes and performance improvements.

Performance improvements

• MJIT
  • For workloads like Rails, the default --jit-max-cache is changed from 100 to 10000. The JIT compiler no longer skips compilation of methods longer than 1000 instructions.
  • To support Zeitwerk of Rails, JIT-ed code is no longer cancelled when a TracePoint for class events is enabled.

Other notable changes since 3.0

• One-line pattern matching, e.g., ary => [x, y, z], is no longer experimental.
• Multiple assignment evaluation order has been changed slightly. [Bug #4443]
  • foo[0], bar[0] = baz, qux was evaluated in order baz, qux, foo, and then bar in Ruby 3.0. In Ruby 3.1, it is evaluated in order foo, bar, baz, and then qux.

• Variable Width Allocation: Strings (experimental) [Bug #18239]

• Psych 4.0 changes Psych.load as safe_load by the default. You may need to use Psych 3.3.2 for migrating to this behavior. [Bug #17866]

Standard libraries updates

• The following default gem are updated.
  • RubyGems 3.3.3
  • base64 0.1.1
  • benchmark 0.2.0
  • bigdecimal 3.1.1
  • bundler 2.3.3
  • cgi 0.3.1
  • csv 3.2.2
  • date 3.2.2
  • did_you_mean 1.6.1
  • digest 3.1.0
  • drb 2.1.0
  • erb 2.2.3
  • error_highlight 0.3.0
  • etc 1.3.0
  • fcntl 1.0.1
  • fiddle 1.1.0
  • fileutils 1.6.0
  • find 0.1.1
  • io-console 0.5.10
  • io-wait 0.2.1
  • ipaddr 1.2.3
  • irb 1.4.1
  • json 2.6.1
  • logger 1.5.0
  • net-http 0.2.0
  • net-protocol 0.1.2
  • nkf 0.1.1
  • open-uri 0.2.0
  • openssl 3.0.0
  • optparse 0.2.0
  • ostruct 0.5.2
  • pathname 0.2.0
  • pp 0.3.0
  • prettyprint 0.1.1
  • psych 4.0.3
  • racc 1.6.0
  • rdoc 6.4.0
  • readline 0.0.3
  • readline-ext 0.1.4
  • reline 0.3.0
  • resolv 0.2.1
  • rinda 0.1.1
  • ruby2_keywords 0.0.5
  • securerandom 0.1.1
  • set 1.0.2
  • stringio 3.0.1
  • strscan 3.0.1
  • tempfile 0.1.2
  • time 0.2.0
  • timeout 0.2.0
  • tmpdir 0.1.2
  • un 0.2.0
  • uri 0.11.0
  • yaml 0.2.0
  • zlib 2.1.1
• The following bundled gems are updated.
  • minitest 5.15.0
  • power_assert 2.0.1
  • rake 13.0.6
  • test-unit 3.5.3
  • rexml 3.2.5
  • rbs 2.0.0
  • typeprof 0.21.1
• The following default gems are now bundled gems. You need to add the following libraries to Gemfile under the bundler environment.
  • net-ftp 0.1.3
  • net-imap 0.2.2
  • net-pop 0.1.1
  • net-smtp 0.3.1
  • matrix 0.4.2
  • prime 0.1.2
  • debug 1.4.0

See NEWS or commit logs for more details.

With those changes, 3124 files changed, 551760 insertions(+), 99167 deletions(-) since Ruby 3.0.0!

Merry Christmas, Happy Holidays, and enjoy programming with Ruby 3.1!

3.0.0

Posted by naruse on 25 Dec 2020

We are pleased to announce the release of Ruby . From 2015 we developed hard toward Ruby 3, whose goal is performance, concurrency, and Typing. Especially about performance, Matz stated “Ruby3 will be 3 times faster than Ruby2” a.k.a. Ruby 3x3.

With Optcarrot benchmark, which measures single thread performance based on NES’s game emulation workload, it achieved 3x faster performance than Ruby 2.0!

These were measured at the environment written in https://benchmark-driver.github.io/hardware.html. 8c510e4095 was used as Ruby 3.0. It may not be 3x faster depending on your environment or benchmark.

Ruby 3.0.0 covers those goals by

• Performance
  • MJIT
• Concurrency
  • Ractor
  • Fiber Scheduler
• Typing (Static Analysis)
  • RBS
  • TypeProf

With above performance improvement Ruby 3.0 introduces a number of new features described below.

Performance

When I first declared “Ruby3x3” in the conference keynote, many including members of the core team felt “Matz is a boaster”. In fact, I felt so too. But we did. I am honored to see the core team actually accomplished to make Ruby3.0 three times faster than Ruby2.0 (in some benchmarks). – Matz

MJIT

Many improvements were implemented in MJIT. See NEWS for details.

As of Ruby 3.0, JIT is supposed to give performance improvements in limited workloads, such as games (Optcarrot), AI (Rubykon), or whatever application that spends majority of time in calling a few methods many times.

Although Ruby 3.0 significantly decreased a size of JIT-ed code, it is still not ready for optimizing workloads like Rails, which often spend time on so many methods and therefore suffer from i-cache misses exacerbated by JIT. Stay tuned for Ruby 3.1 for further improvements on this issue.

Concurrency / Parallel

It’s multi-core age today. Concurrency is very important. With Ractor, along with Async Fiber, Ruby will be a real concurrent language. — Matz

Ractor (experimental)

Ractor is an Actor-model like concurrent abstraction designed to provide a parallel execution feature without thread-safety concerns.

You can make multiple ractors and you can run them in parallel. Ractor enables you to make thread-safe parallel programs because ractors can not share normal objects. Communication between ractors are supported by exchaning messages.

To limit sharing of objects, Ractor introduces several restrictions to the Ruby’s syntax (without multiple Ractors, there is no restriction).

The specification and implementation are not matured and may be changed in the future, so this feature is marked as experimental and show the “experimental feature” warning when the first Ractor.new.

The following small program measures the execution time of famous benchmark tak function (Tak (function) - Wikipedia), by executing it 4 times sequentially or 4 times in parallel with ractors.

def tarai(x, y, z) =
  x <= y ? y : tarai(tarai(x-1, y, z),
                     tarai(y-1, z, x),
                     tarai(z-1, x, y))
require 'benchmark'
Benchmark.bm do |x|
  # sequential version
  x.report('seq'){ 4.times{ tarai(14, 7, 0) } }
# parallel version

x.report('par'){

4.times.map do

Ractor.new { tarai(14, 7, 0) }

end.each(&:take)

}

end

Benchmark result:
          user     system      total        real
seq  64.560736   0.001101  64.561837 ( 64.562194)
par  66.422010   0.015999  66.438009 ( 16.685797)

The result was　measured on Ubuntu 20.04, Intel(R) Core(TM) i7-6700 (4 cores, 8 hardware threads). It shows that the parallel version is 3.87 times faster than the sequential version.

See doc/ractor.md for more details.

Fiber Scheduler

Fiber#scheduler is introduced for intercepting blocking operations. This allows for light-weight concurrency without changing existing code. Watch “Don’t Wait For Me, Scalable Concurrency for Ruby 3” for an overview of how it works.

Currently supported classes/methods:

• Mutex#lock, Mutex#unlock, Mutex#sleep
• ConditionVariable#wait
• Queue#pop, SizedQueue#push
• Thread#join
• Kernel#sleep
• Process.wait
• IO#wait, IO#read, IO#write and related methods (e.g. #wait_readable, #gets, #puts and so on).
• IO#select is not supported.

This example program will perform several HTTP requests concurrently:

require 'async'
require 'net/http'
require 'uri'
Async do

["ruby", "rails", "async"].each do |topic|

Async do

Net::HTTP.get(URI "https://www.google.com/search?q=#{topic}")

end

end

end

It uses async which provides the event loop. This event loop uses the Fiber#scheduler hooks to make Net::HTTP non-blocking. Other gems can use this interface to provide non-blocking execution for Ruby, and those gems can be compatible with other implementations of Ruby (e.g. JRuby, TruffleRuby) which can support the same non-blocking hooks.

Static Analysis

2010s were an age of statically type programming languages. Ruby seeks the future with static type checking, without type declaration, using abstract interpretation. RBS & TypeProf are the first step to the future. More steps to come. — Matz

RBS

RBS is a language to describe the types of Ruby programs.

Type checkers including TypeProf and other tools supporting RBS will understand Ruby programs much better with RBS definitions.

You can write down the definition of classes and modules: methods defined in the class, instance variables and their types, and inheritance/mix-in relations.

The goal of RBS is to support commonly seen patterns in Ruby programs and it allows writing advanced types including union types, method overloading, and generics. It also supports duck typing with interface types.

Ruby 3.0 ships with rbs gem, which allows parsing and processing type definitions written in RBS. The following is a small example of RBS with class, module, and constant definitions.

module ChatApp
  VERSION: String
  class Channel
    attr_reader name: String
    attr_reader messages: Array[Message]
    attr_reader users: Array[User | Bot]              # `|` means union types, `User` or `Bot`.
    def initialize: (String) -> void
    def post: (String, from: User | Bot) -> Message   # Method overloading is supported.
            | (File, from: User | Bot) -> Message
  end
end

See README of rbs gem for more detail.

TypeProf

TypeProf is a type analysis tool bundled in the Ruby package.

Currently, TypeProf serves as a kind of type inference.

It reads plain (non-type-annotated) Ruby code, analyzes what methods are defined and how they are used, and generates a prototype of type signature in RBS format.

Here is a simple demo of TypeProf.

An example input:

# test.rb
class User
  def initialize(name:, age:)
    @name, @age = name, age
  end
  attr_reader :name, :age
end
User.new(name: "John", age: 20)

An example output:

$ typeprof test.rb
# Classes
class User
  attr_reader name : String
  attr_reader age : Integer
  def initialize : (name: String, age: Integer) -> [String, Integer]
end

You can run TypeProf by saving the input as “test.rb” and invoke a command called “typeprof test.rb”.

You can also try TypeProf online. (It runs TypeProf on the server side, so sorry if it is out!)

See the documentation and demos for details.

TypeProf is experimental and not so mature yet; only a subset of the Ruby language is supported, and the detection of type errors is limited. But it is still growing rapidly to improve the coverage of language features, the analysis performance, and usability. Any feedback is very welcome.

Other Notable New Features

• One-line pattern matching is redesigned. (experimental)

  <ul>
    <li>
      <p><code class="language-plaintext highlighter-rouge">=&gt;</code> is added. It can be used as like rightward assignment.</p>
  
      <div class="language-ruby highlighter-rouge">
  

  0 => a
  p a #=> 0
  {b: 0, c: 1} => {b:}
  
  p b #=> 0
  
  

  
  


• 
  

  in is changed to return true or false.

      <div class="language-ruby highlighter-rouge">
  

  # version 3.0
  0 in 1 #=> false
  # version 2.7
  
  0 in 1 #=> raise NoMatchingPatternError
  
  

  
  

Find pattern is added. (experimental)

<div class="language-ruby highlighter-rouge">

case ["a", 1, "b", "c", 2, "d", "e", "f", 3]
in [*pre, String => x, String => y, *post]
  p pre  #=> ["a", 1]
  p x    #=> "b"
  p y    #=> "c"
  p post #=> [2, "d", "e", "f", 3]
end

Endless method definition is added.

<div class="language-ruby highlighter-rouge">

def square(x) = x * x

Hash#except is now built-in.

<div class="language-ruby highlighter-rouge">

h = { a: 1, b: 2, c: 3 }
p h.except(:a) #=> {:b=>2, :c=>3}

Memory view is added as an experimental feature

<ul>
  <li>This is a new C-API set to exchange a raw memory area, such as a numeric array and a bitmap image, between extension libraries.  The extension libraries can share also the metadata of the memory area that consists of the shape, the element format, and so on.  Using these kinds of metadata, the extension libraries can share even a multidimensional array appropriately.  This feature is designed by referring to Python’s buffer protocol.</li>
</ul>

Performance improvements

• Pasting long code to IRB is 53 times faster than bundled with Ruby 2.7.0. For example, the time required to paste this sample code goes from 11.7 seconds to 0.22 seconds.

• The measure command has been added to IRB. It allows simple execution time measurement.

  <div class="language-plaintext highlighter-rouge">
  

  irb(main):001:0> 3
  => 3
  irb(main):002:0> measure
  TIME is added.
  => nil
  irb(main):003:0> 3
  processing time: 0.000058s
  => 3
  irb(main):004:0> measure :off
  => nil
  irb(main):005:0> 3
  => 3
  

Other notable changes since 2.7

• Keyword arguments are separated from other arguments.
  • In principle, code that prints a warning on Ruby 2.7 won’t work. See the document in detail.

  • By the way, arguments forwarding now supports leading arguments.

        <div class="language-ruby highlighter-rouge">
    

    def method_missing(meth, ...)
      send(:"do_#{ meth }", ...)
    end
    

• Pattern matching (case/in) is no longer experimental.
• The $SAFE feature was completely removed; now it is a normal global variable.
• The order of backtrace had been reversed at Ruby 2.5, and is reverted. Now it behaves like Ruby 2.4; an error message and the line number where the exception occurs are printed first, and its callers are printed later.
• Some standard libraries are updated.
  • RubyGems 3.2.3
  • Bundler 2.2.3
  • IRB 1.2.6
  • Reline 0.1.5
  • Pysch 3.3.0
  • JSON 2.5.0
  • BigDecimal 3.0.0
  • CSV 3.1.9
  • Date 3.1.1
  • Digest 3.0.0
  • Fiddle 1.0.5
  • StringIO 3.0.0
  • StringScanner 3.0.0
  • etc.
• The following libraries are no longer bundled gems or standard libraries. Install the corresponding gems to use these features.
  • sdbm
  • webrick
  • net-telnet
  • xmlrpc
• The following default gems are now bundled gems.
  • rexml
  • rss
• The following stdlib files are now default gems and are published on rubygems.org.
  • English
  • abbrev
  • base64
  • drb
  • debug
  • erb
  • find
  • net-ftp
  • net-http
  • net-imap
  • net-protocol
  • open-uri
  • optparse
  • pp
  • prettyprint
  • resolv-replace
  • resolv
  • rinda
  • set
  • securerandom
  • shellwords
  • tempfile
  • tmpdir
  • time
  • tsort
  • un
  • weakref
  • digest
  • io-nonblock
  • io-wait
  • nkf
  • pathname
  • syslog
  • win32ole

See NEWS or commit logs for more details.

With those changes, 4028 files changed, 200058 insertions(+), 154063 deletions(-) since Ruby 2.7.0!

Ruby3.0 is a milestone. The language is evolved, keeping compatibility. But it’s not the end. Ruby will keep progressing, and become even greater. Stay tuned! — Matz

Merry Christmas, Happy Holidays, and enjoy programming with Ruby 3.0!

---

All Depfu comment commands

@​depfu refresh

Rebases against your default branch and redoes this update

@​depfu recreate

Recreates this PR, overwriting any edits that you've made to it

@​depfu merge

Merges this PR once your tests are passing and conflicts are resolved

@​depfu close

Closes this PR and deletes the branch

@​depfu reopen

Restores the branch and reopens this PR (if it's closed)

@​depfu pause

Pauses all engine updates and closes this PR