https://en.wikipedia.org/wiki/Serial_console
tl;dr:
Serial ports are (for example) commonly RS-232, although other types of ports exist. Imagine it to be a very slow Ethernet device. Because it's so slow (and the technology predates Ethernet and also has different requirements), it's usually attached directly to a device instead of to a network. But you could connect a modem to it and it becomes connected to a network device.
It could also be connected to a system console device. These are commonly called terminals. Such devices are often monochrome (especially older ones) because a serial connection is often bandwidth limited (eg, measured in kilobits per second instead of megabits or gigabits). Since it's so slow, it's not practical for video, so it's generally just text-only.
Note that your GPU might also output a system console but rendered on your display at very high resolution and with graphics-drawing capabilities. So a system console would be any console that connects to the system.
What is a console? Well, Wikipedia presents several valid articles and the common theme as far as computers go is that a "console" is typically something that a human and a computer use to interact with each other.
For serial consoles, you might find device files for them at /dev/tty*. But for general serial devices, it could be any of several different types of device files.
Wikipedia's article on /dev devices has a pretty decent listing of what kinds of devices you might find and several of them might be classified as a serial port. Any serial port might be connected to a serial console.
So in my case, a serial console is:
- 2x USB-to-RS-232 (USB is a serial protocol and is basically "just" another (Universal) (and perhaps high speed) Serial port (Bus), so conversion is super cheap)
- 1x RS-232 null modem cable
That's pretty much it in a nutshell. Then
- System 1 (the failing system) UEFI boots into repair system partition on a separately attached disk (eg, boot from CD or live USB) to get a local system console
- System 1 repair system mounts the failing system partition
- System 1 modifies failing system
grubconfiguration to enable a serial console on the attached USB-to-serial device file and saves changes, then unmounts failing system partition - Power off System 1
- Remove repair partition device
- Open terminal window on System 2 (recovery system)
- Connect System 2 terminal to the attached USB-to-serial device file using
screen(oh wow those were some old days) - Power on System 1
- System 1 boot enters grub recovery menu which allows fixing the system remotely
To be fair, a lot of that complexity could have been done by either reinstalling, or removing the hard drive and attaching it to another computer. But doing it this way allowed me to poke around and try different ways of solving the issue, rebooting, etc. It was a learning experience worth exploring.
It was years ago though and I think there was some complication with trying to understand what device file (or device number or something) needed to be to work on the correct serial device (there are often multiple)
I'll quote my current boss's boss's boss when he asked a question of me: