Aside from other stuff mentioned here about email. I always assumed I'd become a target for spam that I'd have a harder time filtering out to the point it stops being worth it to have a custom email address.
That and I can almost guarantee I would end up screwing up the backup of my inbox and losing everything rending the whole endeavour pointless.
It's an assumption, not based on evidence. I've never done it and not looked into the software solutions so I assume a FOSS selfhosted email solution wouldn't have the inbuilt antispam stuff that mainstream providers, say outlook, who host probably millions of email addresses would have.