From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Community icon by AlpΓ‘r-Etele MΓ©der, licensed under CC BY 3.0
Use cockpit by Red Hat. It gives you a GUI to make networking changes*, and will check if the connection still works before making the change. If the connection doesn't work (like the ip addresses changed), it will undo the change and then warn you. You can then either force the change through or leave it be.
*via NetworkManager only.
That's probably because of netplan, right? You should be able to get the same results with just netplan try.
Netplan is an abstraction layer, so it can go over systemd-networkd, NetworkManager, or iproute. I suppose it's better though, because it can be used with multiple backends.
Right, but the entirety of Cockpit is not necessarily required.
You don't need to install cockpit on the server being configured, you can use it as a gui to connect from other machines via the flatpak, over ssh.
Right.
My point is that a wrench was needed and a batmobile was recommended.
No. Netplan uses it's own yaml format, which people would have to learn and use. I don't want to do that, I would rather just configure my existing networkmanager setup, rather than learning another abstraction layer over what is already an abstraction layer.
I understand that cockpit (and similar type tools) are "the whole kitchen sink" of utilities, and it may seem like they come with more than you may need. But that doesn't change the fact that they get the job done, and in some usecases, are better than dedicated tools.
I still prefer net-tools and use ifconfig eth0 up That ip mess I'd rather do without, and those funky UU device/interface names I wish them out of my system
By the way, what system/init/svc manager are you using? With 50y in your back, cron job to check if it is up and resetting it while you are away. You can always remotely cancel the cronjob ... but it will be a new mistake not the old one :)
I started on Irix and ultrix if you remember those, what would I know :)
This is why IPMI is so important.
Not SysAdmin but about a year into my first software engineer job I was working on the live DB in SQL without using BEGIN TRAN ROLLBACK TRAN.
Suffice to say I broke the whole system my making an UPDATE without a WHERE clause. Luckily we have regular backups but it was a lot of debugging with the boss before I realised it was me that caused the issue the client was reporting.
It's not Unix, it's you.
For clarity, I have done it myself - plenty, but not just on Unix boxes.
A few months ago I accidentally dd'd ~3GiB to the beginning of one of the drives in a 4 drive array... That was fun to rebuild.
Like 3 weeks ago on my (testing) server I accidentally DD'd a Linux ISO to the first drive in my storage array (I had some kind of jank manual "LVM" bullshit I set up with odd mountpoints to act as a NAS, do not recommend), no Timeshift, no Btrfs snapshot. It gave me the kick in the pants I needed to stop trying to use a macbook air with 6 external hard drives as a server though. Also gave me the kick in the pants I needed to stop using volatile naming conventions in my fstab.
Your 4 drive raid5 array, right?
Right?!
I wish.
It was a bcachefs array with data replicas being a mix of 1,2 & 4 depending on what was most important, but thankfully I had the foresight to set metadata to be mirrored for all 4 drives.
I didn't get the good fortune of only having to do a resilver, but all I really had to do was fsck to remove references to non-existent nodes until the system would mount read-only, then back it up and rebuild it.
NixOS did save my bacon re: being able to get back to work on the same system by morning.
not RAID10 I hope...
I knew a guy who did this and had to fly to Germany to fix it because he didnβt want to admit what heβd done.
This hits....
At $DAYJOB, we're currently setting up basically a way to bridge an interface over the internet, so it transports everything that enters on an interface across the aether. Well, and you already guessed it, I accidentally configured it for eth0 and couldn't SSH in anymore.
Where it becomes fun, is that I actually was at work. I was setting it up on two raspis, which were connected to a router, everything placed right next to me. So, I figured, I'd just hook up another Ethernet cable, pick out the IP from the router's management interface and SSH in that way.
Except I couldn't reach the management interface anymore. Nothing in that network would respond.
Eventually, I saw that the router's activity lights were blinking like Christmas decoration. I'm guessing, I had built a loop and therefore something akin to a broadcast storm was overloading the router. Thankfully, the solution was then relatively straightforward, in that I had to unplug one of the raspis, SSH in via the second port, nuke our configuration and then repeat for the other raspi.
I formated an OS drive by mistake last night, thought it was my flash drive...
Almost did the same last night on a device that has its internal drive (flash) mounted as mmc and the USB drive was sda
That entire scenario scares me lol
Every network engineer must lock themselves out of a node at some point, it is a rite of passage.
Why don't you use chained commands, or better yet simply create an alias that chains down/up, then use the alias instead?
Because I plain forgot I was remote. It's as simple and as stupid as that.
We've all been there. If you do this stuff for a living, you've done that way more than once.
That is why you have KVMs..
Fair enough. I've done worse in my time as a keyboard jockey.
Or use some kind of molly guard. Or have an OOB management channel.
You'd think you'd learn from your mistakes after one or two of them, not fifty years' worth...
In my defense, I just installed the machine. I was configuring it from home after hours.
Youβd think youβd learn from your mistakes
Yes, that what you'd think. And then you'll sit with a blank terminal once again when you did some trivial mistake yet again.
A friend of mine developed a habit (working on a decent sized ISP 20+ years ago) to set up a scheduled reboot for everything in 30 minutes no matter what you're going to do. The hardware back then (I think it was mostly cisco) had a 'running conrfig' and 'stored config' which were two separate instances. Log in, set up scheduled reboot, do whatever you're planning to do and if you mess up and lock yourself out the system will restore to previous config in a while and then you can avoid the previous mistake. Rinse and repeat.
And, personally, I think that's the one of the best ways to differentiate actual professionals from 'move fast and break things' group. Once you've locked yourself out of the system literally half way across the globe too many times you'll eventually learn to think about the next step and failovers. I'm not that much of a network guy, but I have shot myself in the foot enough that whenever there's dd, mkfs or something similar on the root shell I automatically pause for a second to confirm the command before hitting enter.
And while you gain experience you also know how to avoid the pitfalls, the more important part (at least for myself) is to think ahead. The constant mindset of thinking about processes, connectivity, what you can actually do if you fuck up and so on becomes a part of your workflow. Accidents will happen, no matter how much experience you have. The really good admins just know that something will go wrong at some point in the process and build stuff to guarantee that when you fuck things up you still have availability to fix it instead of calling someone 6 timezones away in the middle of the night to clean up your mess.
Without repeating my other comment. This approach saved my life many times
Good to know that in another 30 years, I will still be doing the dumb shit I've been doing for the last 20.
Did this once on a router in a datacenter that was a flight away. Have remembered to set the reboot in future command since. As I typed the fatal command I remember part of my brain screaming not to hit enter as my finger approached the keyboard. π€¦ββοΈ
Have remembered to set the reboot in future command since
That's not a bad idea actually. I'll have to reuse that one. Thanks!
This.
Do it. This saved my life on more than one occasion.
You'll think βnah, it'll be fineβ and then at 11pm when your brain's fried on vending machine coffee you'll be glad that you did it.. 3 times over...
If it makes you feel any better, I did something just as infuriating a few years ago.
I had set up my home media server, and had finally moved it to my garage with just a power cable and ethernet cable plugged in. Everything was working perfectly, but I needed to check something with the network settings. Being quite new to Linux, I used a remote desktop tool to log in and do everything through a gui.
I accidentally clicked the wrong item in the menu and disconnected the network. I only had a spare ps/2 keyboard and mouse, and as the server was an old computer, it would crash if I plugged a ps/2 device in while it was running*.
The remote desktop stayed open but frozen, mocking me for my obvious mistake and lack of planning, with the remote mouse icon stuck in place on the disconnect menu.
*I can't remember if that was a ps/2 thing, or something specific to my server, but I didn't want to risk it
Old hardware used to get really upset before plug and play became common. I remember I was playing some old racing game with a joystick on a win95 box, and accidentally pulled the connector out, lost my entire game because the system flipped out.
Remember what Bruce Lee said:
I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.
There, but for the grace of god....