When I install the current latest (35.1.13) or recommended (35.1.4) version of Forge 1.16.4, my FPS drops from an average of ~150 to ~40, with regular drops below 30.
This is without any mods or resource packs.
Disabling "forgeLightPipelineEnabled" helped a little, the average rose to ~50 with regular spikes above 100, but the regular drops below 30 persisted.
Disabling "forgeCloudsEnabled" made the problem worse (as expected I presume).
Enabling "alwaysSetupTerrainOffThread" made the problem significantly worse, with an average of ~30.
Increasing memory allocation for Minecraft did not help (5GB).
Specs:
Intel Core i7-9750H
NVIDIA Quadro P2000
The log doesn't look too interesting, but here it is just in case :)