Earlier today I got sent an email from the "server in the loft", porter in case you've forgotten...
The following warning/error was logged by the smartd daemon:
Device: /dev/hda, not capable of SMART self-check
For details see host's SYSLOG (default: /var/log/messages).
Not great... basically it meant that the computer was finding the drive hda so shafted that it couldn't even ask it if it was shafted or not.
Whoops... luckily I wasn't that bothered... ok so I've lost a drive, but there wasn't any data on it "Wooooo". It was however part of my LVM virtual group... (that the thing I set up so I could take snapshots for backups)
Luckily, it seems that LVM2 can cope with loosing a drive that doesn't contain any information that's currently part of the logical volumes...
"I know... I'll remove it from the virtual group, so that when I get home I'll be able to take it out of the machine".
Mistake.
Because the drive couldn't even be read LVM refused to remove it. Tits...
"No worries, there's a 'force' option," I thought as I applied '-f' to the command...
All went without a hitch (it seemed)... the physical volume had been succesfully removed!
So, I get home, turn off the box, unplug the knackered drive and reboot.
DISK BOOT FAILURE, INSERT SYSTEM DISK AND PRESS ENTER
Parp!
Now I know what you're thinking... he's removed hda, that'll be the first drive, the boot up drive... well no 'cause I'd set it in the BIOS to boot from the 3rd drive... "maybe I unplugged the wrong drive" I thought, so plugged it back in and rebooted...
Woo... it started to boot! I got the following message from LVM...
Inconsistent metadata copies found - updating to use version 201
Parp! Parp! Parp! Parp!
After plugging in the laptop directly into the modem I managed to find a helpful Google Groups post... which pointed me in the correct direction of using the command vgcfgrestore to restore the metadata to that of an earlier time - luckily for me each time the something happens with the LVM setup the data is backuped up and stored in /etc/lvm/archives... so I could recover it from there - it worked and my turtle's head popped back home.
Worryily... the drive that apparantly was broken has fixed itself... I shall see if it dies again shortly...
Damn this heat killing my IT kit!
This post has been brought to you with the help of the over use of the <pre></pre> tags :-)