v0.11.0

Added

• Node#inspect is much improved.
• YARP::Pattern is introduced, which can construct procs to match against nodes.
• BlockLocalVariableNode is introduced to take the place of the locations array on BlockParametersNode.
• ParseResult#attach_comments! is now provided to attach comments to locations in the tree.
• MultiTargetNode is introduced as the target of multi writes and for loops.
• Node#comment_targets is introduced to return the list of objects that can have attached comments.

Changed

• BREAKING: GlobalVariable*Node#name now returns a symbol.
• BREAKING: Constant*Node#name now returns a symbol.
• BREAKING: BlockParameterNode, KeywordParameterNode, KeywordRestParameterNode, RestParameterNode, DefNode all have their name methods returning symbols now.
• BREAKING: ClassNode#name and ModuleNode#name now return symbols.
• BREAKING: Location#end_column is now exclusive instead of inclusive.
• Location#slice now returns a properly encoded string.
• CallNode#operator_loc is now CallNode#call_operator_loc.
• CallOperatorAndWriteNode is renamed to CallAndWriteNode and its structure has changed.
• CallOperatorOrWriteNode is renamed to CallOrWriteNode and its structure has changed.

v0.10.0

Added

• InstanceVariable*Node and ClassVariable*Node objects now have their name returning a Symbol. This is because they are now part of the constant pool.
• NumberedReferenceReadNode now has a number field, which returns an Integer.

Changed

• BREAKING: Various operator_id and constant_id fields have been renamed to operator and name, respectively. See 09d0a144 for details.
• %w, %W, %i, %I, %q, and %Q literals can now span around the contents of a heredoc.
• BREAKING: All of the public C APIs that accept the source string now accept const uint8_t * as opposed to const char *.

v0.9.0

Added

• Regular expressions can now be bound by \n, \r, and a combination of \r\n.
• Strings delimited by %, %q, and %Q can now be bound by \n, \r, and a combination of \r\n.
• IntegerNode#value now returns the value of the integer as a Ruby Integer.
• FloatNode#value now returns the value of the float as a Ruby Float.
• RationalNode#value now returns the value of the rational as a Ruby Rational.
• ImaginaryNode#value now returns the value of the imaginary as a Ruby Complex.
• ClassNode#name is now a string that returns the name of just the class, without the namespace.
• ModuleNode#name is now a string that returns the name of just the module, without the namespace.
• Regular expressions and strings found after a heredoc declaration but before the heredoc body are now parsed correctly.
• The serialization API now supports shared strings, which should help reduce the size of the serialized AST.
• *Node#copy is introduced, which returns a copy of the node with the given overrides.
• Location#copy is introduced, which returns a copy of the location with the given overrides.
• DesugarVisitor is introduced, which provides a simpler AST for use in tools that want to process fewer node types.
• {ClassVariable,Constant,ConstantPath,GlobalVariable,InstanceVariable,LocalVariable}TargetNode are introduced. These nodes represent the target of writes in locations where a value cannot be provided, like a multi write or a rescue reference.
• UntilNode#closing_loc and WhileNode#closing_loc are now provided.
• Location#join is now provided, which joins two locations together.
• YARP::parse_lex and YARP::parse_lex_file are introduced to parse and lex in one result.

Changed

• When there is a magic encoding comment, the encoding of the first token's source string is now properly reencoded.
• Constants followed by unary & are now properly parsed as a call with a passed block argument.
• Escaping multi-byte characters in a string literal will now properly escape the entire character.
• YARP.lex_compat now has more accurate behavior when a byte-order mark is present in the file.
• BREAKING: AndWriteNode, OrWriteNode, and OperatorWriteNode have been split back up into their 0.7.0 versions.
• We now properly support spaces between the encoding and =/: in a magic encoding comment.
• We now properly parse -> foo: bar do end.

v0.8.0

Added

• Some performance improvements when converting from the C AST to the Ruby AST.
• Two rust crates have been added: yarp-sys and yarp. They are as yet unpublished.

Changed

• Escaped newlines in strings and heredocs are now handled more correctly.
• Dedenting heredocs that result in empty string nodes will now drop those string nodes from the list.
• Beginless and endless ranges in conditional expressions now properly form a flip flop node.
• % at the end of files no longer crashes.
• Location information has been corrected for if/elsif chains that have no else.
• __END__ at the very end of the file was previously parsed as an identifier, but is now correct.
• BREAKING: Nodes that reference &&=, ||=, and other writing operators have been consolidated. Previously, they were separate individual nodes. Now they are a tree with the target being the left-hand side and the value being the right-hand side with a joining AndWriteNode, OrWriteNode, or OperatorWriteNode in the middle. This impacts all of the nodes that match this pattern: {ClassVariable,Constant,ConstantPath,GlobalVariable,InstanceVariable,LocalVariable}Operator{And,Or,}WriteNode.
• BREAKING: BlockParametersNode, ClassNode, DefNode, LambdaNode, ModuleNode, ParenthesesNode, and SingletonClassNode have had their statements field renamed to body to give a hint that it might not be a StatementsNode (it could also be a BeginNode).

v0.7.0

Added

• We now have an explicit FlipFlopNode. It has the same flags as RangeNode.
• We now have a syntax error when implicit and explicit blocks are passed to a method call.
• Node#slice is now implemented, for retrieving the slice of the source code corresponding to a node.
• We now support the utf8-mac encoding.
• Predicate methods have been added for nodes that have flags. For example CallNode#safe_navigation? and RangeNode#exclude_end?.
• The gem now functions on JRuby and TruffleRuby, thanks to a new FFI backend.
• Comments are now part of the serialization API.

Changed

• Autotools has been removed from the build system, so when the gem is installed it will no longer need to go through a configure step.
• The AST for foo = *bar has changed to have an explicit array on the right hand side, rather than a splat node. This is more consistent with how other parsers handle this.
• RangeNodeFlags has been renamed to RangeFlags.
• Unary minus on number literals is now parsed as part of the literal, rather than a call to a unary operator. This is more consistent with how other parsers handle this.

v0.6.0

First public release of the gem. 🎉

v0.5.0

On this release, all of our test targets are passing at 100%. The last feature was pattern matching.

v0.4.0

This release brings us very close to being able to parse most of our test targets. The remaining failures are tracked in #562. As of this tag we are at:

ruby/spec - 4662/4663 - 99.98%
rails/rails - 3081/3083 - 99.94%
discourse/discourse - 4423/4434 - 99.75%
ruby/ruby - 7823/7848 = 99.68%

The only syntax feature we have left is pattern matching. The rest of the issues are bug fixes. At this point we feel comfortable beginning the work of experimenting with integrating this parser into existing tools while we continue to finish the other pieces.

v0.3.0

New stuff

• We now have a MissingNode that is inserted when an expression is expected but not present
• We now keep a stack of contexts around as we're parsing. If a token is found that is unexpected that would close out a parent context, we put the parser into "recovering" mode and insert missing nodes until we get back up to the parent node
• We now expose CommentNode nodes in the parse result, along with handling __END__ and =begin..=end syntax
• We now support multiple encodings, and parse encoding magic comments to determine the encoding
• Much more documentation has been added
• Parse symbols in SymbolNode and InterpolatedSymbolNode
• There's now a pack parser baked into YARP
• We now template out Java classes based on the serialization
• We now parse regular expressions to get out locals generated from capture groups
• AliasNode and UndefNode
• RestParameterNode, KeywordParameterNode, BlockParameterNode, ForwardingParameterNode, NoKeywordsParameterNode
• BeginNode
• SClassNode
• ForNode
• MultiTargetNode which is the beginnings of parsing multiple assignment
• ParenthesesNode
• We now support providing the escaped version of a string

Changed stuff

• String lists with interpolation are now part of ArrayNode
• IfNode now also functions in the elsif context
• Now we parse else after unless
• Use 32 bits for locations and lengths to shorten serialized string
• We now support endless method definitions

Fixed stuff

• Handle more underscores in number literals
• Better handle identifiers that end in =
• Better handle \r\n line terminators
• Handle escaped terminators in strings

v0.2.0

Lots of updates since the last release 2 weeks ago, so I felt like it was time for another tag.

• I updated the codebase to try to enforce a little consistency in the function naming. Basically each struct should have an _alloc, _init, and _free variant. By separating out _alloc and _init it makes it a little more clear what's going on when some structs can be either nested or living on their own.
• I tried to simplify the config a bit to make it more maintainable going forward. The list of nodes is growing and it's getting more difficult to read at a glance. I also added comments to all of the nodes that we have so far to hopefully better communicate what they each mean semantically.
• I've added tokens that can be optional on nodes, which means we now have a YP_TOKEN_NOT_PROVIDED. This token type is meant to indicate that a token could be here but wasn't found in the source and it's not an error (think parentheses on method calls).
• I added some code to walk the tree in C and prettyprint the nodes/tokens/strings. It makes it much easier to debug what's going on when developing.
• We're now using rake to handle everything, which makes for a much streamlined development experience. Thanks @tenderlove!
• We now correctly parse superclasses, after fixing a binding power issue.
• I started work on encodings. We have ASCII and UTF-8 baked in now. I'm not 100% sure if this is the way I actually want to go, but it works for now. Lots more experimentation needed on this.

Now for the new nodes:

• BreakNode, NextNode, SuperNode, YieldNode
• ForwardingSuperNode
• StringListNode, StringNode, InterpolatedStringNode, StringInterpolatedNode
• ConstantPathWriteNode
• RangeNode
• DefinedNode
• ElseNode
• DefNode
• RequiredParameterNode, OptionalParameterNode, KeywordParameterNode

And a couple of changes:

• CharacterLiteralNode no longer exists, as it's just a StringNode.
• CallNode now works for methods with ? and !, operators of ::, and the .() shorthand
• CallNode now tracks its parentheses