java.util.ArrayList.removeAll()

[Suterusu]

Although this may not be directly Forge related, I need someplace to start with asking for help.

 

I run some dedicated modded MC servers and have recently run into an issue where my CPU spikes, not randomly.  Minecraft 1.7.10

 

 

This is happening with the same frequency yet various intensity over all of my testing.  A modded server that has "been around" for a while will notice this spike at a much greater CPU value than that of a server with a fresh world.

 

I have been working to profile the server and found that this is not something caused by a ticking entity, title entity, etc. it's nothing that appears to be any way related to a particular mod itself (pending there is no heavy disk IO or some quiescing of MC).  I have this issue on modded servers with 0 forced chunks and 0 players online.

 

Using both Java 7_67 and 8_11 I have experienced this issue.  I've tested on several platforms with, like I said, various mod packs and noticed the same behavior.  This is straight Forge, however I have had the issue with Cauldron as well.  It's currently crushing one of my production MC servers with the dreaded "can't keep up!" every time the CPU spikes.

 

MCMA graph:

 

 

The best I've been able to get information-wise thus far is running Java VisualVM and sampling the cpw.mods.fml instance.  I've "timed" sampling and snapshotting of the CPU during the times of the CPU spikes to try and find something within the Server thread that could be the culprit.  The only thing I see active with high utilization during these snapshots is "java.util.ArrayList.removeAll()"

 

I really need to pin point the cause of these CPU spikes, if I am going down the wrong path, any direction would be greatly appreciated.

 

Server Specs:

Minecraft 1.7.10 (Mod Packs tested: Beyond Reality, Bevo Tech Pack)

E3 1245v2

SSDx2 in HW RAID 1

32GB RAM (30GB dedicated to MC, 16GB in testing)

[LexManos]

It would make sense that a lot of time is spent in ONE OF THE CORE FUNCTIONS OF JAVA!

Nice charts tho, they mean nothing. Unless you're going to post actual profiling information and let those who know what theya re doing dissect it these posts are useless.

 

Periodic CPU usage usually just means the GC is running, again basic java functionality.

[Suterusu]

The point was moreover the fact that these CPU spikes basically lock up the server.  If they are core Java functions that are working as intended, I am not accustom to them causing this severe level of performance degradation (even for MC).  Again, the CPU spike "trend" happens even on a freshly installed instance on a freshly generated world.  This is just stating information from what I've seen, not making a declaration of the problem.

 

Now, as you are correct that others would know way more what they are doing than I, how would you prefer all of the CPU sampling snapshots?  Dropbox link to a zip?

[LexManos]

Yes, any form of getting the snapshots.

To allow me to dissect it and try and figure out what is causing the issue.

As for it locking up your server, cant say anything about that, from your own logs its only taking ~24% cpu.

But again, can't really do whything with 'lag' or anything like that especially without real details.

[Suterusu]

I apologize, the CPU graphs were to show trending as I believe the Java Visual VM may have been averaging CPU load over all cores.  I won't post additional unneeded graphs of CPU usage, but the core that Java is running on does spike to 80-100% usage during the lag spikes.

 

I've attached what I believe to be importable CPU snapshots.  I've not had to export this data before so if it is incorrect, please tell me and I will research an alternative method.  I used Java Visual VM 8.

 

I've named the files based on the tests performed, if there is a question about testing condition, please let me know.  All of these test were performed with straight Forge on Java 1.8 Update 11.  I do NOT have extra Java options set, which may be part of my issue (other than -Xmx 16G).  I did find this thread regarding Java 7/8 options, however I've not tested them yet as there was debate to whether or not they did anything useful.  If anyone has clear direction on the best practice for additional Java options, I would gladly like to test them out.

 

http://www.minecraftforum.net/forums/support/server-support/server-administration/1937726-java-7-8-command-line-options-for-minecraft

 

Download for CPU sampling: https://www.dropbox.com/s/bd72honoj638lvk/minecraftcpu.zip?dl=0

 

[LexManos]

Looking at it, it looks like your are indeed tieing yourself up in the GC.

Churning about 1GB of memory every 20 seconds.

Your snapshots also dont include anything beyond just the overview so I can't tell you why, but this is more then JUST forge on that server, logistics pipes, IC2, OpenComputers, and god knows what else.

So, by the look of it it probably isn't Forge doing anything.

[Suterusu]

Lex, thanks for taking the time to look over the sampling.

 

Would you be able to point me in the direction of any useful resources regarding GC optimization?  I understand this is complex and no simple "fix" exists as it's the nature of the beast.  I at least want to know that I am following best practice.