Linux

I may have posted this before, but...late last year I realized my Debian server with circa 2009 hardware, with 4 gb of RAM and Core 2 Quad processor, was no longer up to the tasks I wanted it to perform; in particular, running a Home Assistant server. Back in 2018 or so, I added a software Linux RAID5 array with 5 active 3 TB drives and one hot spare, along with a "cold spare" that I've never actually used.

So, early this year, I bought hardware to upgrade my desktop machine, which was still plenty fast for me, and move the guts to my server. This is how my server usually gets upgraded. Upgrade the desktop machine, give it a few days or weeks to make sure it's stable, and then upgrade the server.

I installed the hardware without a problem, booted it up, and everything seemed okay, except that I ...couldn't access the RAID. At first it was like, well, I'm sure it's nothing serious, but then when mdadm could even FIND it, I started to get extremely worried. Fear set in.

Long story short: When I built the RAID, I followed directions that used the entire discs as the RAID, instead of making a partition on the disc and using that partition. The old motherboard didn't care, but the new one saw the bare discs and was like, "Hey, those are messed up, I'll fix the partition table for you!" Turns out, building Linux RAIDs by using the full discs like that is a VERY BAD IDEA for exactly this reason - but there are still guides out there showing that method and not mentioning the risks.

I was panicking. I spent days trying to figure out what to do and nothing was working. I was asking for help on the Linux-RAID list (and most of them were as helpful as they could be). Unfortunately my backups were NOT up to par (something I should have checked before starting), and I was at the point where I was like, well, I've lost x, y, and z.

I had basically given up and was just recreating the RAID using the "create command" then trying to see if I could mount the drive read-only. With 6 drives, there are quite a few possible combinations that could be the right one. If I remember correctly, I was able to figure out which drive was the spare, so I could limit my searches to the other 5, and knowing all 5 were in use, it was a matter of trying different orders. I think I got close one time and ext4 gave me weird read error, so after that I swapped two drives, and hit the right order.

Eventually .... I found it. I found the right combination and could reload it! Everything was there, untouched! As quickly as I could, I copied everything to a 10 TB drive I bought and installed into the desktop system. I saved the command, rebooted, and the same thing happened again - so it was definitely a motherboard problem - but this time I knew how to recreate it, and did so.

Since I now had a backup, I partitioned each drive and rebuilt the array using partitions...and I saved every piece of data I could think of about building the array, outputs of mdadm, outputs of /proc/mdstat, partition IDs, etc. Naturally, having that info likely means I'll never need it.

I was so relieved when I saw that mount command work without error. I spent close to a week worrying about it, and in that moment it was a huge rush.

New setup handles HA and other duties with aplomb and is very reliable, so in the end it was very worth it.

This is less "silly" and more "horrifying". Sorry.

I deleted my desktop environment during an apt upgrade, not once, but twice. Bad habit of not actually reading the messages that pop up properly - it did ask me if I wanted to delete it all, and I just said "yea lol lfg". There was some conflict with a third party PPA that caused this.

Didn't know that had happened to begin with. I was stuck on the session manager login screen and it just wouldn't proceed after entering password. First time I just reinstalled Linux, and the second time I found out how to reinstall it from tty. This is how I learned about tty as well.

I think I posted this before in some other thread, but one time back when I used to use Ubuntu, I opened my laptop and the screen was upside-down. Everything worked perfectly, but just upside-down. I went through every display setting I could find, trawled through forums for hours (on a different, non upside-down computer) and got absolutely nowhere. It was at the point where I was thinking I'll probably have to reformat and start over and this will forever be a mystery.

Then I accidentally solved it when my Playstation controller battery got low and I plugged it into the nearest USB port to charge, which was my laptop. As soon as I plugged it it, the screen flipped back the right way. As it turned out, Ubuntu was talking to the controller and had for some reason interpreted the gyroscope movement as 'rotate screen' the last time I charged it. After a couple of minutes of waving the controller around and watching the desktop spin while going "huh", I just unplugged it when it the right way round and crisis averted!

Just the other day I tried to remove pipewire from my system but didn't look at the list of packages to be removed... Turns out it removed gnome desktop and so booted into CLI 🤦‍♀️

I wanted to reinstall my Gentoo system. A SUSE (back before OpenSUSE) disc was the newest distro I had lying around. I thought it shouldn't matter from which system I do the install, Gentoo won't care.

So I repartitioned /dev/hda, installed the base system and went to set up my mount points. Only to discover that my data drive was gone. Stupid SUSE labeled the drives differently. /dev/hdb was my old system drive and I had repartitioned my old data drive.

Taught me to really check which drive was which. I wouldn't touch SUSE again for decades because of this.

One time while streaming I had someone convince me to install zsh. Almost bricked the thing live. No clue what I was doing that day.

I made alias q=exit for some reason and now I accidentally close the terminal when I press ;q and enter 😭

Short: I forgot the /etc/fstab mount entry

I'm not sure if the following counts as stupid, but here is one where I almost wiped my system and reinstalled everything. I have some entries in the /etc/fstab to bind certain directories to specific locations in my home, to keep it modular (doing this since over 10 years). One day I replaced one of the internal harddrives and then the system would no longer boot up, because the it tries to mount a non existent drive.

Due to my long years of experience and wisdom with Linux, I thought that either the new drive was broken or I something from my body sparked over the board. It took me several minutes until I realized what actually happened and then everything was fine.

BTW in EndeavourOS when this happens again (and it did) then while boot the system asks me to ignore that entry and continue. Which is soooo useful and don't know why this was never asked before (before I was on EndeavourOS).

I set up a cron task and it was meant to do a super scuffed sendmail if there was a problem, there was about 20GB on the spool before I noticed and the pi's SD card was full