Frankly yes if you’re doing it in a commercial setting, and probably if it’s a hobby. Frameworks like Rails or Django handle a lot of stuff you probably don’t want to and don’t know how to build right. You may want to learn how they work and jump in to futz with stuff to get it where you want it but you don’t want to build a lot of stuff yourself. There are security risks if you do that may be mitigated by the framework maintainers already, optimizations and edge case handling that you need to be aware of, and standards you might have to meet. I think at best you would end up rewriting a lot of the same functionality that’s in a framework.
A better or more sustainable option might be to learn how all these middleware and low level components work so you can form educated opinions on configuration and implementation