Slack started with the classic “one box, one crontab” approach for cron jobs. Initially, it worked fine, but as the platform grew, so did the number of scripts and their processing demands. This led to several issues:
- Reliability woes: A single node meant a single point of failure. Any problems with the box could bring down critical features.
- Scaling struggles: Adding more RAM and CPU became a temporary fix, not a long-term solution.
- Maintenance nightmares: Patching the growing complexity became increasingly burdensome.
Building a Better Way: Introducing Chronos
Facing these challenges, Slack opted for a custom solution: Chronos. Here’s a deeper look at its components:
Scheduled Job Conductor
Slack improved its cron system by using Golang for efficiency and sticking to a familiar cron format. For deployment, they chose Bedrock, a Kubernetes wrapper, making scaling a breeze. To enhance reliability, they implemented leader election, ensuring one “conductor” runs, with others ready to take over seamlessly. Smart scheduling was implemented to avoid unnecessary node downtime during peak execution times, optimizing performance.
Job Queue
For Slack, the smart move was using its powerful Job Queue platform to handle script execution efficiently. By running each script in its own queue, they made sure everything runs swiftly. This approach reduced the workload, handling processing and memory needs efficiently and making maintenance a breeze.
Vitess Database Table
In the Slack world, manual script locking (Flocks) is out, replaced by smart database checks to ensure smooth job execution. Now, every job’s progress is tracked, from being queued to done, offering clear transparency. Plus, users can conveniently keep an eye on their scripts and troubleshoot errors through a user-friendly web page.
Outcomes
Chronos implemented at Slack brought robust reliability with no single points of failure, seamless scalability to handle increased demands, and a user-friendly web interface for easy script monitoring and management.
Beyond Technical Benefits
Chronos at Slack embodies the essence of Slack’s engineering values. By utilizing platforms like Job Queue, it streamlines development and upkeep. Its scalability focus ensures it aligns with Slack’s ongoing success. The user-friendly interface prioritizes engineers’ ease in handling cron jobs effectively.
Conclusion
Chronos at Slack not only solved technical issues but also follows the company’s engineering principles, focusing on efficiency, scalability, and user-friendly design for ongoing success.
Reference
https://slack.engineering/executing-cron-scripts-reliably-at-scale/
