Skip to content
  • Koichi ITO's avatar
    0f076fa5
    [ruby/prism] Fix an AST and token incompatibility for `Prism::Translation::Parser` · 0f076fa5
    Koichi ITO authored
    This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
    for empty xstring literal.
    
    ## Parser gem (Expected)
    
    ```console
    $ bundle exec ruby -Ilib -rparser/ruby33 -ve \
    'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
    [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
    ```
    
    ## `Prism::Translation::Parser` (Actual)
    
    Previously, the AST and tokens returned by the Parser gem were different:
    
    ```console
    $ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
    'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
    [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
    ```
    
    After this correction, the AST and tokens returned by the Parser gem are the same:
    
    ```console
    $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
    'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
    [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
    ```
    
    https://github.com/ruby/prism/commit/4ac89dcbb5
    0f076fa5
    [ruby/prism] Fix an AST and token incompatibility for `Prism::Translation::Parser`
    Koichi ITO authored
    This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
    for empty xstring literal.
    
    ## Parser gem (Expected)
    
    ```console
    $ bundle exec ruby -Ilib -rparser/ruby33 -ve \
    'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
    [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
    ```
    
    ## `Prism::Translation::Parser` (Actual)
    
    Previously, the AST and tokens returned by the Parser gem were different:
    
    ```console
    $ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
    'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
    [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
    ```
    
    After this correction, the AST and tokens returned by the Parser gem are the same:
    
    ```console
    $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
    'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
    [:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
    ```
    
    https://github.com/ruby/prism/commit/4ac89dcbb5
Loading