Skip to content
  • Sander Verdonschot's avatar
    9b62f88a
    Add `perform_all_later` to enqueue multiple jobs at once · 9b62f88a
    Sander Verdonschot authored
    Sidekiq has a useful optimisation called `push_bulk` that enqueues many jobs at
    once, eliminating the repeated Redis roundtrips. However, this feature is not
    exposed through Active Job, so it only works for `Sidekiq::Worker` jobs. This
    adds a barrier to Active Job adoption for apps that rely on this feature. It
    also makes it harder for other queue adapters to implement similar
    functionality, as they then have to take care of serialization, callbacks, etc.
    themselves.
    
    This commit adds `ActiveJob.perform_all_later(<job1>, <job2>)`, backed by
    Sidekiq's `push_bulk` and with a fallback to enqueuing serially if the queue
    adapter does not support bulk enqueue.
    
    The performance benefit for 1000 jobs can be more than an order of magnitude:
    
    | Enqueue type       | Serial time (ms) | Bulk time (ms) | Speedup |
    | ------------------ | ---------------- | -------------- | ------- |
    | Raw Sidekiq        |             2661 |            119 |     22x |
    | Active Job Sidekiq |             2853 |            208 |     14x |
    
    (Measured in a simple test app in our production environment.)
    
    Instrumentation for perform_all_later uses a new event `enqueue_all.active_job`
    9b62f88a
    Add `perform_all_later` to enqueue multiple jobs at once
    Sander Verdonschot authored
    Sidekiq has a useful optimisation called `push_bulk` that enqueues many jobs at
    once, eliminating the repeated Redis roundtrips. However, this feature is not
    exposed through Active Job, so it only works for `Sidekiq::Worker` jobs. This
    adds a barrier to Active Job adoption for apps that rely on this feature. It
    also makes it harder for other queue adapters to implement similar
    functionality, as they then have to take care of serialization, callbacks, etc.
    themselves.
    
    This commit adds `ActiveJob.perform_all_later(<job1>, <job2>)`, backed by
    Sidekiq's `push_bulk` and with a fallback to enqueuing serially if the queue
    adapter does not support bulk enqueue.
    
    The performance benefit for 1000 jobs can be more than an order of magnitude:
    
    | Enqueue type       | Serial time (ms) | Bulk time (ms) | Speedup |
    | ------------------ | ---------------- | -------------- | ------- |
    | Raw Sidekiq        |             2661 |            119 |     22x |
    | Active Job Sidekiq |             2853 |            208 |     14x |
    
    (Measured in a simple test app in our production environment.)
    
    Instrumentation for perform_all_later uses a new event `enqueue_all.active_job`
Loading