No, but there are a few disadvantages if you don't start a new world:
Most of the mods spawn blocks, like ores and trees, only in new chuncks. To find them, you have to go to a place where you've never been.
You cannot convert back to a non-forge server if you have mod blocks in your world (like ores, trees, etc.).