I don't use such a machine myself, but those are tools which I would turn to if I had :

• https://github.com/EmbroidePy/pyembroidery
• https://github.com/inkstitch/inkstitch

The first one is just a library (i.e. building blocks), while the second looks more like a complete solution - based on Linux' most used vector software Inkscape.

I don’t think this is exactly what you‘re looking for, but for cross stitch patterns, KXStitch can import images and create simplified designs with a reduced number of colours and cells/pixels. The KXStitch handbook gives a nice overview of all the features.