a small network switch is the right answer here. skips figuring out windows settings and doesn't rely on your PC being on.
something like this one should do just fine for ya. :)
https://www.amazon.com/Ethernet-Splitter-Optimization-Unmanaged-TL-SG105/dp/B00A128S24/?th=1
Tucking ethernet under/along baseboard trim can be a legit option depending on ages and preferences of others in the house.
If you own the place, you could also consider drilling a hole through the floor at the edge of the room as well if the space below is unfinished.
depending on what the upstairs connectivity requirements are moving all the wired connections to the basement and just using wifi for the rest is a very legit option. I've done that in the past when my office and gaming pc were in the basement and it was only the Roku and phones and tablet on wifi used elsewhere.
If it doesn't have an AP mode in the settings you can manually put it in AP mode.
https://www.cnet.com/home/internet/how-to-turn-an-old-wi-fi-router-into-an-access-point/
But since it sounds like you (OP) hasn't bought anything yet then probably easier to buy an Access Point device instead of another router.