Typically, Forge itself has almost zero impact on the performance of minecraft. Some area its a little slower others it's massivly more efficient.

 

However, when it comes to adding mods in, it all depends on your computer. Some can handle it at 5,000fps, others can't

{My personal tests give be a 150% bump in FPS when Forge is installed}

The more mods you have installed, the less FPS you will have. Simply because the game is doing more.

 

If you really think this is something on Forge's end, which I highly doubt.

Go grab visualvm or jprofiler and start profiling minecraft to see where it's spending it's processing time.

Alright thanks for the tip, I will try both of those. Although when I install forge by itself with no mods I still get very low FPS. I'm guessing it's either an issue with my windows 8 or a hardware issue if not forge.