-
Eugene Kenny authored
`Dir.glob` doesn't guarantee the order of its results: https://ruby-doc.org/core-2.6.5/Dir.html#method-c-glob > Case sensitivity depends on your system (File::FNM_CASEFOLD is > ignored), as does the order in which the results are returned. Minitest stores a list of all test cases in the order that they were defined; it shuffles them before they're run, but doesn't sort them: https://github.com/seattlerb/minitest/blob/v5.13.0/lib/minitest.rb#L1048 https://github.com/seattlerb/minitest/blob/v5.13.0/lib/minitest.rb#L156 This means that the order in which framework tests run is platform dependent, and running a test command that failed in CI locally won't necessarily reproduce the error, even when the same seed is provided. `Rake::FileList` resolves glob patterns to a sorted list of files: https://github.com/ruby/rake/blob/v13.0.1/lib/rake/file_list.rb#L408 By using `Rake::FileList` instead of `Dir.glob`, framework tests will always run in the same order when given the same seed, and reproducing order dependent CI failures will be easier.
Eugene Kenny authored`Dir.glob` doesn't guarantee the order of its results: https://ruby-doc.org/core-2.6.5/Dir.html#method-c-glob > Case sensitivity depends on your system (File::FNM_CASEFOLD is > ignored), as does the order in which the results are returned. Minitest stores a list of all test cases in the order that they were defined; it shuffles them before they're run, but doesn't sort them: https://github.com/seattlerb/minitest/blob/v5.13.0/lib/minitest.rb#L1048 https://github.com/seattlerb/minitest/blob/v5.13.0/lib/minitest.rb#L156 This means that the order in which framework tests run is platform dependent, and running a test command that failed in CI locally won't necessarily reproduce the error, even when the same seed is provided. `Rake::FileList` resolves glob patterns to a sorted list of files: https://github.com/ruby/rake/blob/v13.0.1/lib/rake/file_list.rb#L408 By using `Rake::FileList` instead of `Dir.glob`, framework tests will always run in the same order when given the same seed, and reproducing order dependent CI failures will be easier.
Loading