I switched from a big custom Ansible deployment to NixOS.
The system includes 8 managed machines, multiple VPNs and a custom certificate authority.
Downsides:
- rethinking of how to manage Certificates and VPN configs outside of Nix
- getting secrets to work took a bit until I found agenix
- deployments can take a while with deploy-rs
Still, I can only tell you how much more at ease I feel with the NixOS based system. Its just much easier to refactor, not having to take care of legacy cleanup and polluting the machines over time.
Once you wrap your head around it all more complex system architectures start to become manageable/maintainable.
IaC
You still need sth like Terraform on the side for your actual infrastructure provisioning.
Solutions to bridge this with the Nix ecosystem are evolving in the nix-community repos on Github, but I found it easier to manage that separately for the time being.
All in all I would recommend NixOS based systems for the heavy lifters in your setup. If you want to deploy a fleet of machines you are entering new territory. Exciting, but maybe too much of a time commitment for some.
Pixel Phone with GrapheneOS is simply the best solution available right now.