Skip to content
  • Peter Zhu's avatar
    d7388f72
    Fix buffer overrun with auto-compact for shapes · d7388f72
    Peter Zhu authored
    The following script crashes:
    
    ```ruby
    GC.auto_compact = true
    GC.stress = true
    
    class Foo
      def initialize
        @a = @b = @c = 0
      end
    
      def add_ivars
        @d = @e = @f = 0
      end
    end
    
    ary = 1_000.times.map { Foo.new }
    ary.each { |f| f.add_ivars }
    ```
    
    This is because in rb_grow_iv_list, it first calls
    rb_ensure_iv_list_size to allocate the buffer (and also unsets the
    embed bit) then rb_shape_transition_shape_capa to get the new shape.
    However, auto-compact can trigger in rb_shape_transition_shape_capa
    which would re-embed the object since it doesn't have the new shape yet.
    This causes a crash as the object is now embedded but has a non-embed
    shape which would cause the object to have a buffer overrun.
    d7388f72
    Fix buffer overrun with auto-compact for shapes
    Peter Zhu authored
    The following script crashes:
    
    ```ruby
    GC.auto_compact = true
    GC.stress = true
    
    class Foo
      def initialize
        @a = @b = @c = 0
      end
    
      def add_ivars
        @d = @e = @f = 0
      end
    end
    
    ary = 1_000.times.map { Foo.new }
    ary.each { |f| f.add_ivars }
    ```
    
    This is because in rb_grow_iv_list, it first calls
    rb_ensure_iv_list_size to allocate the buffer (and also unsets the
    embed bit) then rb_shape_transition_shape_capa to get the new shape.
    However, auto-compact can trigger in rb_shape_transition_shape_capa
    which would re-embed the object since it doesn't have the new shape yet.
    This causes a crash as the object is now embedded but has a non-embed
    shape which would cause the object to have a buffer overrun.
Loading