if using -Xms512M doesn't fix the issue, try adding -Xmn256M to the list of JVM arguments. I've noticed that with both modded server's I've set-up, and modded clients, that has prevented the server from running out of memory, and also, prevented clients and servers alike from crashing. Although, for vanilla, it runs a lot smoother.
-Xmn256M replaces -XX:MaxPermSize=xxM from what I understand