I'm having the same issue (OSX 10.8.5, "Mod system installer" window pops up with nothing but an OK button, OK button doesn't do anything, can't quit, have to force quit).
Tried to run the installer from the command line
java -jar minecraftforge-installer-1.6.4-9.11.1.916.jar
but there is no console output or log file created.
Any ideas? Thanks.