I've assembled my own modpack, but after some more testing I found out the server will crash everytime a client disconnects.
I couldn't get any usefull information from the log, but it doesn't mention any mod. Forge and PlayerAPI are the only JAR mods I have installed, so I suspect that Forge is the cause of this crash.
EDIT: This still happens with the latest version of forge for minecraft 1.4.7
Forge Log
Minecraft Crash Log