Another big problem with Signal is the fact that it's centralized with the server being located in US. Even if the protocol itself is secure with the server not having access user data, this presents a huge risk since US government can simply force Signal to shut down the service at any time. The server can also potentially collect metadata about the users providing US security agencies with user connection graphs.
I think that Matrix approach is much more sound, and would always recommend it over Signal.