I have noticed when travelling between nether and the overworld, the memory consumption slowly grows until the server eventually 1. becomes unresponsive or 2. terminates because it runs out of memory. Often times, the problem gets worse as the map ages. I created a brand new server and map with zero mods and was able to replicate the issue, making me think this is forge related.
This problem was recorded using forge-1.11.2-13.20.1.2454 and java 1.8.0_144. This problem was first discovered in 1.10.2 for me and my fellow players and may have existed long before then for all we know.
FML log file can be found here:
https://pastebin.com/QmCzcK0J
Regular (console?) log file can be found here:
https://pastebin.com/rHrjz408
Please also note, that even though we are using Multicraft to manage starting/stopping the JVM, this problem is present whether or not we use Multicraft or any server management software for that matter.