Minecraft Forge stuttering on high-end PC

[HappyChapi]

So I recently upgraded my PC's gpu from a GT 720 to a GTX 1660 Super. I've been able to play all the latest games like Modern Warfare, World War Z, and Forza Horizon 4 at max settings with 60+fps and plenty of headroom. However, Minecraft with Forge seems to be struggling. It runs really well when standing still, but whenever I move and the game loads chunks, it drops below 60fps to around 50-40 and is very annoying. Uncapped when standing still I get in the 100s. My ancient GT 720 card held a rock solid 60 fps at all times. I have Optifine installed, and I allocated 8GB of RAM to the game from the launcher and from the control panel. I only have a few mods installed. I think it has something to do with either the CPU or RAM. It only dips when the game is loading chunks. Applying shaders doesn't make the stutters any worse. I've seen a bunch of other people running Minecraft better with shaders, higher render distance, and no stutters with worse PCs than mine. I am quite puzzled.
PC Specs:
GeForce GTX 1660 Super
Driver version 446.14
Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
31.95 GB RAM
1920 x 1080 60Hz display

[DavidM]

Try removing all the mods. Does the stuttering still occur?

[HappyChapi]

Yes, and it's worse without Optifine.