Ok I'm removing mods one by one from both client and server now, here's what I found out so far:
When I had the error and remove a mod from both sides, then connect to the server again, I still receive the error directly upon connecting.
When I delete the world on the server after removing a mod from both sides, it lets me connect to the server again and it takes about 5 minutes before I get the error again.