Manage Cronjobs Over Multiple Servers

Diglett is a cron management system that manages all your cronjobs without modifying crontab. Handles locking, logging, error emails, and more. This project was created from the need of managing hundreds of tasks ( scheduled tasks) over multiple servers.

Why Diglett ?

Let’s assume that you have the following projects :

  1. projectA : 3 Servers
  2. projectB : 2 Servers
  3. projectC : 4 Servers

And each one of these project has his own multiple cronjobs that only on one of its servers. And to make it more complicated, cronjob-X in porjectA should not be started until cronjob-Y in projectC is finished.

Imagine the complexity of solving dependencies using the standard crontab and bash scripts in Linux !

This is why we have built Diglett, it can easily move cronjobs from server to server, also adding a new server or project is easy as two click using the mouse.

Diglett Rules

  1. Each project has multiple servers, but only one server holds the active crontab.
  2. Jobs can depend on each other and the system alerts you whenever a job with a dependency fails or started before its dependency is over.
  3. Logs for all jobs; includes output logs, running status, running time and the last time it was executed.
  4. All jobs are executed on time, even if Diglett server is down or unreachable.

MongoDB Structure

All project data should be stored in the database, mainly in three collections :

Projects

{
 "name" : "projectA",
 "user" : "UserA",
 "hosts" : [
    "172.16.240.15:22",
    "172.16.241.147:22",
    "172.16.240.16:22"
    ],
 "active_host" : "172.16.241.147:22"
}

Crons

In this collections all commands are stored, each cronjob/command has a uniqe name per project. As shown in the following, it has the command itself, its time and the last time running status.

{
 "name" : "projectA.archive_command", 
 "project" : "projectA",
 "description" : "this is just a description",
 "depends_on" : "projectB.transfer_files",
 "command" : "/usr/bin/python /home/userA/path/archive.py",
 "time" : "*/5 * * * *",
 "active" : true,
 "last_run_at" : ISODate("2016-09-14T18:35:04.954Z"),
 "last_run_status" : "0"
}

History

Diglett inserts a document to this collection once it received a /started request from any job and updated with status code and logs once it’s done after receiving /finished request.

The running time for any job is the difference between the two signals, finish and start.

{
 "start_time" : 1466565121,
 "name" : "projectC.post_views_sync",
 "status_code" : "0",
 "log" : "all posts view are synced\nreal\t0m0.259s\nuser\t0m0.214s\nsys\t0m0.049s\n",
 "running_time" : 0.488843
}

Alerting

Diglett uses two methods of alerting for now, Email and Push Notifications.

Emails

Mainly depends on mail-utlis on linux system, therefore it should be installed on your Diglett Server.

Push Notification

For now, Diglett is using only one push notification backend called SimplePush. To use it, you should install the application on your Android and insert the key in config.ini file.

Also, you can disable the push notification feature from the same file.

Installation

Install Nginx as a webserver and Mongodb as DB :

$ sudo dnf update -y
$ sudo dnf install -y nginx git python-pip mongodb-server mongodb

Clone Diglett :

$ git clone https://github.com/OpenSooq/Diglett.git ~/diglett

Copy systemd service file and nginx configuration file from examples folder :

$ sudo cp ~/diglett/examples/diglett@.service /etc/systemd/system/
$ sudo cp ~/diglett/examples/nginx.conf /etc/nginx/conf.d/diglett.conf

Install required packages :

$ cat ~/diglett/requirements.txt | grep -v ^# | xargs sudo dnf install -y
$ sudo pip install pymongo==3.0.0

Change the UID and GID in uwsgi.ini and in diglett@.service :

$ vim ~/diglett/uwsgi.ini
...
uid = username
gid = username
...
$ sudo vim /etc/systemd/system/diglett@.service
...
User= username
Group= username
...

Change the admin user to a user that has root/sudo access to all servers :

$ vim ~/diglett/config.ini
...
[manager]
admin= username
manager_url= http://PROJECT_IP
...

Initialize the database

$ mongo diglett < examples/initialize_db.js

Start and enable services :

$ sudo systemctl enable nginx && sudo systemctl start nginx
$ sudo systemctl enable mongod && sudo systemctl start mongod
$ sudo systemctl start diglett@3030

Leave a Reply

Your email address will not be published.