Skip to content
  • Koichi ITO's avatar
    c0b8dee9
    [ruby/prism] Fix an AST and token incompatibility for `Prism::Translation::Parser` · c0b8dee9
    Koichi ITO authored
    This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
    for dstring literal:
    
    ```ruby
    "foo
      #{bar}"
    ```
    
    ## Parser gem (Expected)
    
    ```console
    $ bundle exec ruby -Ilib -rparser/ruby33 -ve \
    'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:dstr,
      s(:str, "foo\n"),
      s(:str, "  "),
      s(:begin,
        s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
    [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
    [:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
    [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
    [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
    [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
    [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
    ```
    
    ## `Prism::Translation::Parser` (Actual)
    
    Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested:
    
    ```console
    $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
    'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:dstr,
      s(:dstr,
        s(:str, "foo\n"),
        s(:str, "  ")),
      s(:begin,
        s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
    [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
    [:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
    [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
    [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
    [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
    [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
    ```
    
    After this correction, the AST and tokens returned by the Parser gem are the same:
    
    ```console
    $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
    'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:dstr,
      s(:str, "foo\n"),
      s(:str, "  "),
      s(:begin,
        s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
    [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
    [:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
    [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
    [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
    [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
    [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
    ```
    
    https://github.com/ruby/prism/commit/c1652a9ee7
    c0b8dee9
    [ruby/prism] Fix an AST and token incompatibility for `Prism::Translation::Parser`
    Koichi ITO authored
    This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
    for dstring literal:
    
    ```ruby
    "foo
      #{bar}"
    ```
    
    ## Parser gem (Expected)
    
    ```console
    $ bundle exec ruby -Ilib -rparser/ruby33 -ve \
    'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:dstr,
      s(:str, "foo\n"),
      s(:str, "  "),
      s(:begin,
        s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
    [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
    [:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
    [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
    [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
    [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
    [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
    ```
    
    ## `Prism::Translation::Parser` (Actual)
    
    Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested:
    
    ```console
    $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
    'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:dstr,
      s(:dstr,
        s(:str, "foo\n"),
        s(:str, "  ")),
      s(:begin,
        s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
    [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
    [:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
    [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
    [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
    [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
    [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
    ```
    
    After this correction, the AST and tokens returned by the Parser gem are the same:
    
    ```console
    $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
    'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
    ruby 3.3.0 (2023-12-25 revision https://github.com/ruby/prism/commit/5124f9ac75) [x86_64-darwin22]
    [s(:dstr,
      s(:str, "foo\n"),
      s(:str, "  "),
      s(:begin,
        s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
    [:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
    [:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
    [:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
    [:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
    [:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
    [:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
    ```
    
    https://github.com/ruby/prism/commit/c1652a9ee7
Loading