At home I have various devices doing various things.. and it’s important to me that I know they are working. There are many tools out there designed to keep an eye on server health, but they don’t do everything that I want in the way that I want, and I’m a big believer that coding = creating, so there’s nothing wrong with reinventing the wheel just for the sake of it.
For me it started recently when I repurposed an old android phone as an IP webcam pointing out the front window. After that it was a natural progression to install the excelent linux app Motion to pick up movement from that feed, and store images and mpegs to disk – for security. Well no point in storing the images on the nice machine that is likely to be stolen in event of a break in.. so let’s store them on an old EEE-Box that’s running headless in a hidden corner of the basement 😉
Well there you have it. Definite needs for a frequent health check:
– is the camera working?
– is the EEE nfs mount mounted?
I wrote a ruby script and cron to solve that problem, but then it just grew. Now I have the system running on multiple machines:
– checking each other
– making sure all my websites are responding to ping
– validating that certain URLs are giving a 200 response
– making sure that disks aren’t filling up
– checking that certain processes / background applications are running
On top of that:
– my laptop knows if I’m at home or away and tests accordingly
– the system creates Desktop icons for each problem (and removes them when problems resolve)
– it can generate Ubuntu desktop notifications
– it can notify me of issues with a text message
– each machine manages hostname|issue style files on Dropbox so whereever I am my laptop can tell me what’s going on at home.
– it shuts up at night
– for every problem it finds, it checks to see if there are instructions to try to correct it
All this is achieved with:
– 200 lines of core ruby methods that perform all the tests
– 50 lines of code relating to specific issue resolution
– 100 lines of control code – one liners which are basically “If you are this machine, then do this test”
In Part II we’ll take the very basics of that code, and create a simple single ruby script to keep an eye on a machines own health.
Dropbox and Google Drive (with Insync) – let’s just call them “Netdrives” – are great ways to sync files across multiple linux (Ubuntu) machines.
- Utterly reliable sync
- Online login to get your files from anywhere
- Great mobile support
- Can be used headless on linux servers
- Cross platform (Mac, PC, Linux)
When it comes to using them as a repo for scripts though they have inherant issues. They don’t retain file attributes.. so you create a file on one machine, chmod +x and it gets synced to all other machines, but without the +x.
- Ubuntu One.. but every time I try it I walk away thinking it’s not really an option at all. Why can’t Ubuntu get this right!
- NFS mounts. A good option if everythings on the same Lan, but what if it’s not?
- sshfs mounts. I still love sshfs mounts for certain things, but not when I want persistency.
- Github.. a bit clunky.
So, here’s what I do to solve this issue. A BETTER way (and I’ll blog that soon) is to use incrond inotify, but here’s a quick and easy way that works well and is really painless.
1) Put all the files in the same place.
I use /Dropbox/config/bin for Bash files
2) Create two files that do the “work” for you.
You need two files. One of which you actually have to chmod +x yourself at some point, but it’s not a file that will EVER change.
I use /Dropbox/config/bin/worker as a file that has the job of making sure all other files in this tree are executable
chmod -R +x $HOME/Dropbox/config/bin
Note that it launches the file worker2
Here’s MY worker2
It just launches something that I want to run often. The reason for worker2 is that you can modify worker2 from any machine, and worker will make it executable just before executing it.
3) Use Cron to run worker as often as you like.
* * * * * export DISPLAY=:0.0 && export XAUTHORITY=/home/keith/.Xauthority && sudo -u keith /home/keith/Dropbox/config/bin/worker
I run it every minute.. because I want that pulse_check (makes sure all my stuff is healthy) to run every minute anyway. Note the overly complex way that this is called in cron. All that XAUTHORITY stuff on a Ubuntu box means that if notify-send (create desktop notifications) can be used in anything downstream launched by worker, and they show up on the desktop.
That’s it! It’s not rocket science, and in a way it’s a bit ghetto – but it works well.