Start by not really worrying about balance too much. CR is next to useless, even at very low levels. Just feel it out experimentally over a few battles, and if you go too big, withhold some of the nastier enemy abilities.
I do those big creative boss battles. I start by finding the most similar enemies to my custom enemies and running some basic numbers in Kobold Fight Club (maybe 5 minutes). Then I eyeball a rough difficulty multiplier based on their level, classes, and magic items.
My players are currently level 17 with minimal magic items and based on some experimental results I take whatever Kobold gives me for extremely deadly and then multiply the difficulty by about three to actually have a difficult battle. That might mean more enemies, or it might mean more damage, higher HP, higher AC, increased saves, etc. Right now a challenging battle for them is one in which I take a put maybe three CR 20-something monsters on the field against my party of five.