Try the JVM argument in this post, just sub xmx/xms numbers for what you want/need allocated.
https://www.reddit.com/r/feedthebeast/s/lzmKUNZFrG
I'm having the same issue with a huge modpack using the 3 culprits to your issue.
Let me know if this helps!