News archives for general

It happened again. The GNU C Library broke, this time on Debian Etch. Not being able to track down the problem to either something I did or a Debian bug, I decided on adding another layer of protection. All services at tsampa.org are now running as virtual private servers (VPS) under OpenVZ. If any VPS dies, it does not hose the main system allowing me to restore within minutes instead of reinstalling and configuring everything from scratch. This also allows several auxiliary benefits, including easy migration to newer server hardware when that time comes and easy server resource management. Thumbs up for OpenVZ!

On February 20th, a routine upgrade of the server software using the Debian package manager failed horribly. The most central software library of them all, the GNU C Library, failed to upgrade correctly leaving the system largely non-functional. And to top it off, the Windows laptop followed suit with a prompt blue screen of death, thus killing the last open shell to the server. The web servers and databases were still happily up and running thanks to being isolated in their own self-containing jail environments, but most everything else, including logging in, was dead in the water. A reboot gave the expected result when the system failed to boot. Caputt.

I forked up the $50 for a 24 hour rental of a remote console with various rescue tools and operating system reloaders to be able to make a last full backup before reinstalling. While at it, I figured I would go ahead and upgrade to Debian Etch, since it is just around the corner to become the new stable release. But the server configuration, while flexible and hard drive failure resistant, proved to be a nightmare in this scenario where data had to be rescued off an unbootable system; the server was configured to mirror the data on two identical hard drives (RAID level 1) with most of the data residing on LVM volumes formatted with the ReiserFS filesystem. With a hardware card taking care of the mirroring, the two drives still transparently appeared as a single drive to the rescue disks. Alas, the rescue disks proved not have the software libraries needed to mount the LVM volumes AND read the ReiserFS format. Goodbye quick backup, goodbye.

Plan B was to have the RAID array broken up by removing the hardware card to allow a fresh install of the operating system with the requisite tools on one drive to access the original data on the second drive. This took a bit of constructive hard drive switching and bootloader configuration as the boot information and drive partition tables turned out to be stored on the hardware RAID card itself making the drive information unaccessible without it. In the end, fed up with facing yet another several hour wait to have a hard drive removed, I bravely installed the new operating system while having the original drive plugged in as secondary master with the feared result that I whacked up the partition table for the last original drive in the process. This was four days after the server crashed, two of which was spent waiting for the remote console to actually be hooked up and then determined not to work in my server rack due to a local issue necessitating the temporary relocation of my server. At this time, I had arrived at the tranquil conclusion that losing the last few months of e-mail and a few scripts would not be the end of the world. And it wasn’t. Alea iacta est and whatnot, I still had a reasonable fresh full backup of all the web data complemented by Google’s cache so things were reasonably dandy.

Armed with a new sense of how to reconfigure the server to better take this kind of disaster scenario into account, I had the techs remove the hardware RAID card for good and set out to install Debian Etch with software RAID. Things were not smooth sailing now either, as the installer first refused to install Etch due to a hard to track down Debootstrap Error, the server chassis needed to be replaced due to a faulty keyboard port and the techs made several mistakes (it did net me free access to the remote console for a few days and a free 512MB RAM addition for a current total of 2GB). In the end, it took an incredible eight days to go come to the point where I could actually do a 30 minute fresh install of the system and have the server moved back to its regular slot.

With this much downtime already behind me and no paying clients on the server, I took the time to upgrade all software, to rewrite the jailing scripts where needed and, more than anything else, to hack on a comeback to be reckoned with for my virtual powerlifting project (soon to be relaunched, tighten your socks!)… basically doing things that would have been needed to be done soon anyway. Apologies for the downtime, stay tuned for the pay off.

The final bits and pieces have now been restored here at tsampa.org, shoot me an e-mail if something is not working like it used to.

After two years of running tsampa.org on a virtual dedicated server over at Advantagecom (excellent hosting btw), we are now running on a dedicated 2.4 GHz Pentium 4 box with 1GB of RAM powered by Debian Sarge. The brunt of the migration effort is on the backside (my collected notes span 50 pages in OpenOffice), but I did take this opportunity to redesign and move the frontpage and Under the Bar from Movable Type to a hacked Wordpress 1.5. My design is, understandably enough, called One Line to Rule them All… There are still a few rough edges to sort out, but do e-mail me if anything strikes you as broken or, gulp, undesirable.

To fill a glaring gap in my bookshelf, I am looking for the following book. I am willing to pay a decent price if somebody can get me a copy (e-mail address at bottom of this page).

Edit 3 Dec 2005: Finally located a used copy. Nothing like finding a book you have been frantically searching for the last two years. Something tells me I should start thinking about writing a certain master’s thesis now…

Leslie, Donald Daniel: Islam in traditional China: a short history to 1800. Canberra College of Advanced Education, 1986.

Welcome to tsampa.org, which is going live today. It is about time; my first Tibetan learning program Tibword was released back in 1998-99, but has as far as I know never been publicly available on the web. I still have a lot of material waiting to be put online, so expect this site to keep growing throughout May. This frontpage is where new additions will be announced.

If you find yourself wondering what tsampa is, wonder no more. If you are into some form of weight training you might also be interested in my training blog Under the bar.

The design will also be refined later down the road, what matters now is that this site validates as XHTML 1.1 and is finally online.