I don't know what I did differently, but the error code changed to "Invalid signature for profile public key" so I set "enforce-secure-profile" to false since only my friend will be allowed to join anyway and then everything worked properly.
Here's the logs anyway, but I hope the problem doesn't come back. https://paste.ee/p/Ri44L