Thomas RibouletLead consultant
This article is aimed for Senior or Lead software engineers working on RubyOnRails applications and wondering about where to go for handling background jobs.
RubyOnRails has a great and vibrant ecosystem filled with many utilities for all kind of purposes. One of those purposes is background jobs. Following is a brief overview of the usual options RubyOnRails developers go with and of the less usual options.
From early on in RoR history we have relied on projects such as Resque, Sidekiq, DelayedJob. All of them are relying on Redis to store jobs. They are all a great solution to handle background jobs.
A lot of us favor Sidekiq lately, it’s a great solution : it’s easy to integrate (with or without ActiveJob), it’s easy to work with and it can scale to a decent level of business (and beyond). Yet, the other ones can be great to.
We have other solutions such as Que and Que Classic. As it relies on PostgreSQL it’s an easy addition to most teams who already rely on it for the main storage of their application.
Software engineers who have had another life in SE before working on RoR apps will often be surprised to find Redis or even PostgreSQL at the center of the background job processing setup.
For almost 10 years before RoR came up message brokers and pub/sub strategies were used to handle this.
One very common choice was and still is RabbitMQ. There are multiple libraries to use it with RubyOnRails, a popular one is Sneakers (https://github.com/jondot/sneakers).
Another way to use messages queues is to rely on AWS SQS and you can count on Shoryuken (https://github.com/ruby-shoryuken/shoryuken) for this. It’s widely influenced by Sidekiq and works well too.
Finally, we also should point out solutions using Kafka as backbone of the message handling are getting traction. There are several posts out there, here are two posts to get started :
Finally a handful of libraries are worth mentioning :
That question can be answered with such an article. You, as a software engineering professional, should look to your context to decide what your options are and which one is a good one for now.
You can have try outs, test each one of them, read up on them.
Sidekiq, Resque and a few others are great to start with because they gather several important points together :
Which ever solution you go with, keep in mind setting up and using (queuing jobs, treating jobs) isn’t the end of the story. On a day to day basis you will also have to know if the jobs are being handled properly, why they fail if they do, how to handle many workers to treat the jobs, and so on.
There are great news though : you are not the first to reach that point. Whether you are looking for a first background job processing solution or a replacement there has been many people going through that before. Search for articles online on the topic, read on all those different experiences.
We, at Imfiny, can also help with this. We have worked with many teams on this topic : helping with setting up good practices within teams, migrating from one to another, scaling the whole approach. Ping us, we can have a chat and see what we can do together.
A RubyOnRails consultancy based in the EU, we build your applications and services all over the world !