Skip to content
  • NAITOH Jun's avatar
    a041a6c1
    [ruby/strscan] Add scan and search benchmark · a041a6c1
    NAITOH Jun authored
    (https://github.com/ruby/strscan/pull/111)
    
    # Why?
    To improve the parsing process, I would like to add benchmarks for all
    parsing processes.
    
    ## scan
    - scan_full(regexp, false, true) == StringScanner#check
    - scan_full(regexp, false, false) ==  StringScanner#match?
    
    ### CRuby
    
    ```
    $ benchmark-driver benchmark/scan.yaml
    Warming up --------------------------------------
              check(reg)    10.558M i/s -     10.848M times in 1.027445s (94.71ns/i)
              check(str)    13.368M i/s -     13.782M times in 1.030978s (74.80ns/i)
             match?(reg)    16.080M i/s -     16.247M times in 1.010340s (62.19ns/i)
             match?(str)    23.336M i/s -     23.501M times in 1.007088s (42.85ns/i)
    Calculating -------------------------------------
              check(reg)    11.601M i/s -     31.675M times in 2.730287s (86.20ns/i)
              check(str)    15.217M i/s -     40.104M times in 2.635475s (65.72ns/i)
             match?(reg)    18.781M i/s -     48.241M times in 2.568662s (53.25ns/i)
             match?(str)    29.441M i/s -     70.007M times in 2.377840s (33.97ns/i)
    
    Comparison:
             match?(str):  29441324.5 i/s
             match?(reg):  18780543.7 i/s - 1.57x  slower
              check(str):  15217130.1 i/s - 1.93x  slower
              check(reg):  11601371.2 i/s - 2.54x  slower
    ```
    ### JRuby
    
    ```
    $ benchmark-driver benchmark/scan.yaml
    Warming up --------------------------------------
              check(reg)     8.129M i/s -      8.090M times in 0.995222s (123.02ns/i)
              check(str)    16.691M i/s -     16.616M times in 0.995519s (59.91ns/i)
             match?(reg)     8.979M i/s -      9.001M times in 1.002440s (111.37ns/i)
             match?(str)    26.138M i/s -     26.011M times in 0.995150s (38.26ns/i)
    Calculating -------------------------------------
              check(reg)    11.808M i/s -     24.387M times in 2.065238s (84.69ns/i)
              check(str)    31.762M i/s -     50.072M times in 1.576495s (31.48ns/i)
             match?(reg)    13.944M i/s -     26.936M times in 1.931719s (71.71ns/i)
             match?(str)    50.872M i/s -     78.414M times in 1.541392s (19.66ns/i)
    
    Comparison:
             match?(str):  50872250.2 i/s
              check(str):  31761544.3 i/s - 1.60x  slower
             match?(reg):  13944219.6 i/s - 3.65x  slower
              check(reg):  11808244.1 i/s - 4.31x  slower
    ```
    
    ## search
    - search_full(regexp, false, true) == StringScanner#check_until
    - search_full(regexp, false, false) == StringScanner#exist?
    ```
    $ benchmark-driver benchmark/search.yaml
    Warming up --------------------------------------
        check_until(reg)     9.338M i/s -      9.456M times in 1.012573s (107.09ns/i)
        check_until(str)    11.385M i/s -     11.979M times in 1.052173s (87.83ns/i)
             exist?(reg)    13.416M i/s -     13.517M times in 1.007532s (74.54ns/i)
             exist?(str)    17.976M i/s -     18.677M times in 1.038981s (55.63ns/i)
    Calculating -------------------------------------
        check_until(reg)    10.297M i/s -     28.015M times in 2.720634s (97.11ns/i)
        check_until(str)    12.684M i/s -     34.156M times in 2.692853s (78.84ns/i)
             exist?(reg)    15.184M i/s -     40.249M times in 2.650786s (65.86ns/i)
             exist?(str)    21.426M i/s -     53.928M times in 2.517008s (46.67ns/i)
    
    Comparison:
             exist?(str):  21425527.1 i/s
             exist?(reg):  15183679.9 i/s - 1.41x  slower
        check_until(str):  12684053.7 i/s - 1.69x  slower
        check_until(reg):  10297134.8 i/s - 2.08x  slower
    ```
    
    ### JRuby
    ```
    $ benchmark-driver benchmark/search.yaml
    Warming up --------------------------------------
        check_until(reg)     7.646M i/s -      7.649M times in 1.000381s (130.78ns/i)
        check_until(str)    13.075M i/s -     13.010M times in 0.995048s (76.48ns/i)
             exist?(reg)     8.728M i/s -      8.684M times in 0.994921s (114.57ns/i)
             exist?(str)    20.609M i/s -     20.514M times in 0.995399s (48.52ns/i)
    Calculating -------------------------------------
        check_until(reg)     9.371M i/s -     22.939M times in 2.447900s (106.71ns/i)
        check_until(str)    22.760M i/s -     39.225M times in 1.723414s (43.94ns/i)
             exist?(reg)    11.758M i/s -     26.185M times in 2.226997s (85.05ns/i)
             exist?(str)    34.564M i/s -     61.827M times in 1.788749s (28.93ns/i)
    
    Comparison:
             exist?(str):  34564306.2 i/s
        check_until(str):  22759878.4 i/s - 1.52x  slower
             exist?(reg):  11757927.4 i/s - 2.94x  slower
        check_until(reg):   9371009.3 i/s - 3.69x  slower
    ```
    
    https://github.com/ruby/strscan/commit/81a80a176b
    a041a6c1
    [ruby/strscan] Add scan and search benchmark
    NAITOH Jun authored
    (https://github.com/ruby/strscan/pull/111)
    
    # Why?
    To improve the parsing process, I would like to add benchmarks for all
    parsing processes.
    
    ## scan
    - scan_full(regexp, false, true) == StringScanner#check
    - scan_full(regexp, false, false) ==  StringScanner#match?
    
    ### CRuby
    
    ```
    $ benchmark-driver benchmark/scan.yaml
    Warming up --------------------------------------
              check(reg)    10.558M i/s -     10.848M times in 1.027445s (94.71ns/i)
              check(str)    13.368M i/s -     13.782M times in 1.030978s (74.80ns/i)
             match?(reg)    16.080M i/s -     16.247M times in 1.010340s (62.19ns/i)
             match?(str)    23.336M i/s -     23.501M times in 1.007088s (42.85ns/i)
    Calculating -------------------------------------
              check(reg)    11.601M i/s -     31.675M times in 2.730287s (86.20ns/i)
              check(str)    15.217M i/s -     40.104M times in 2.635475s (65.72ns/i)
             match?(reg)    18.781M i/s -     48.241M times in 2.568662s (53.25ns/i)
             match?(str)    29.441M i/s -     70.007M times in 2.377840s (33.97ns/i)
    
    Comparison:
             match?(str):  29441324.5 i/s
             match?(reg):  18780543.7 i/s - 1.57x  slower
              check(str):  15217130.1 i/s - 1.93x  slower
              check(reg):  11601371.2 i/s - 2.54x  slower
    ```
    ### JRuby
    
    ```
    $ benchmark-driver benchmark/scan.yaml
    Warming up --------------------------------------
              check(reg)     8.129M i/s -      8.090M times in 0.995222s (123.02ns/i)
              check(str)    16.691M i/s -     16.616M times in 0.995519s (59.91ns/i)
             match?(reg)     8.979M i/s -      9.001M times in 1.002440s (111.37ns/i)
             match?(str)    26.138M i/s -     26.011M times in 0.995150s (38.26ns/i)
    Calculating -------------------------------------
              check(reg)    11.808M i/s -     24.387M times in 2.065238s (84.69ns/i)
              check(str)    31.762M i/s -     50.072M times in 1.576495s (31.48ns/i)
             match?(reg)    13.944M i/s -     26.936M times in 1.931719s (71.71ns/i)
             match?(str)    50.872M i/s -     78.414M times in 1.541392s (19.66ns/i)
    
    Comparison:
             match?(str):  50872250.2 i/s
              check(str):  31761544.3 i/s - 1.60x  slower
             match?(reg):  13944219.6 i/s - 3.65x  slower
              check(reg):  11808244.1 i/s - 4.31x  slower
    ```
    
    ## search
    - search_full(regexp, false, true) == StringScanner#check_until
    - search_full(regexp, false, false) == StringScanner#exist?
    ```
    $ benchmark-driver benchmark/search.yaml
    Warming up --------------------------------------
        check_until(reg)     9.338M i/s -      9.456M times in 1.012573s (107.09ns/i)
        check_until(str)    11.385M i/s -     11.979M times in 1.052173s (87.83ns/i)
             exist?(reg)    13.416M i/s -     13.517M times in 1.007532s (74.54ns/i)
             exist?(str)    17.976M i/s -     18.677M times in 1.038981s (55.63ns/i)
    Calculating -------------------------------------
        check_until(reg)    10.297M i/s -     28.015M times in 2.720634s (97.11ns/i)
        check_until(str)    12.684M i/s -     34.156M times in 2.692853s (78.84ns/i)
             exist?(reg)    15.184M i/s -     40.249M times in 2.650786s (65.86ns/i)
             exist?(str)    21.426M i/s -     53.928M times in 2.517008s (46.67ns/i)
    
    Comparison:
             exist?(str):  21425527.1 i/s
             exist?(reg):  15183679.9 i/s - 1.41x  slower
        check_until(str):  12684053.7 i/s - 1.69x  slower
        check_until(reg):  10297134.8 i/s - 2.08x  slower
    ```
    
    ### JRuby
    ```
    $ benchmark-driver benchmark/search.yaml
    Warming up --------------------------------------
        check_until(reg)     7.646M i/s -      7.649M times in 1.000381s (130.78ns/i)
        check_until(str)    13.075M i/s -     13.010M times in 0.995048s (76.48ns/i)
             exist?(reg)     8.728M i/s -      8.684M times in 0.994921s (114.57ns/i)
             exist?(str)    20.609M i/s -     20.514M times in 0.995399s (48.52ns/i)
    Calculating -------------------------------------
        check_until(reg)     9.371M i/s -     22.939M times in 2.447900s (106.71ns/i)
        check_until(str)    22.760M i/s -     39.225M times in 1.723414s (43.94ns/i)
             exist?(reg)    11.758M i/s -     26.185M times in 2.226997s (85.05ns/i)
             exist?(str)    34.564M i/s -     61.827M times in 1.788749s (28.93ns/i)
    
    Comparison:
             exist?(str):  34564306.2 i/s
        check_until(str):  22759878.4 i/s - 1.52x  slower
             exist?(reg):  11757927.4 i/s - 2.94x  slower
        check_until(reg):   9371009.3 i/s - 3.69x  slower
    ```
    
    https://github.com/ruby/strscan/commit/81a80a176b
Loading