[SOLVED][1.10+] Advice / Information on Safely Adding a Novel Biome Provider

[JaredBGreat]

Really looking more for general advice, info, or insight on this one...

 

I have a WIP biome placement mod (more of a biome zones mod than a biomes mod per se).  Basically it places biomes based on a quasi-realistic climate model, using a system I developed outside Minecraft in a tool I made for experimenting with procedural content generation (especially maps).  I've inserted my system into Minecraft as a custom world type.  The worlds it makes seems great at this point, but there is a serious problem I need to fix before I go further with things like configurability and updating:

 

I find the game using far more memory than I think it should, and exploring so that new terrain is generated causes the memory usage to increase without really going back down much (something I take as a sign memory is leaking).  Note that I don't use any genlayers at all, having completely replaced the vanilla system and simply returning biome arrays where needed.  Other than my own system (which I'm looking into, but I don't think has an internal problem), does anyone have any idea what might cause this?  Also, I see it producing save files for distant parts of the world, far from where a player have been (things like "r.16.-20.mca" when I've never been more than a couple thousand blocks from spawn), and suspect that may be a system of the same underlaying problem, probably based on not interfacing my biome provider correctly with the vanilla chunk provider and world in general.

 

I don't have a specific ease to nail down bug on this one, or specific code I can point to on this one, and don't really expect a specific answer, but if anyone has and potentially useful information or advice on doing things like this I'd be thankful to see it.

Edited December 29, 2017 by JaredBGreat

[daruskiy]

Without logs I can only take a guess, but I know from my experience one thing I had an issue with when modifying world generation was a "Cascading world generation" issue where each time you created a new chunk, regardless of it being in render distance, it would cause the next chunk after it to be created as well. When I was working on this issue for my mod this what was causing my lag during world generation and was fixed when I saw a logic hole in my code. Aside from that I can offer no guidance

[jabelar]

I don't know why people write code, watch it behave badly, and then start asking others what might be wrong when you can simply debug it yourself. Basically you simply need to trace the execution. I tend to do this by adding console statements (System.out.println()) statements at strategic locations of the code. I usually put such a statement in each method I want to confirm is being called (usually printing out the parameter values as well), and I also put statements inside any conditional statements to check that my if-statements and similar logic are executing as expected.

 

You can also use the debugger in Eclipse by setting breakpoints and such, but I personally find that less useful because much of what I want to watch is repeated operation during actual gameplay.

 

I'm pretty sure that by adding such traceability to your code it will be quickly apparent what is going wrong -- in this case you might see things being called more frequently than expected, or chunks being generated when not expected, and you can simply keep working backwards to find the source of the problem.

 

It is really hard to ask people on a forum to debug something like this because it is usually a very specific logic bug or typo/mistake that is really your responsibility to track down.

[JaredBGreat]

12 hours ago, daruskiy said:

Without logs I can only take a guess, but I know from my experience one thing I had an issue with when modifying world generation was a "Cascading world generation" issue where each time you created a new chunk, regardless of it being in render distance, it would cause the next chunk after it to be created as well. When I was working on this issue for my mod this what was causing my lag during world generation and was fixed when I saw a logic hole in my code. Aside from that I can offer no guidance

Thanks.  I don't think that exactly is the problem, as it doesn't become slow unless it runs out of memory completely and starts having lag spikes, and extra files are generally empty.  I'll take a look.

 

9 hours ago, jabelar said:

[...]

Thanks for accusing me of doing things I did not do, asking for favors I did not ask for, and telling me things I already know, and assuming I don't know how to (or won't) do something I've done many, many times.  I was simply asking if anyone had any good information on the sub-systems involved, not for anyone to debug anything.  Anyway, I think I'll pass on your suggestion to create a huge core mod that adds debugging code to a large portion of the game.

 

EDIT:  Nevermind -- I don't want to start a fight.  I was just hoping people who'd done similar things might have some general advice on pitfalls they may have encountered or know about, insights that might be useful.

 

I'm removing the link to the biome provider so no one will think I'm asking for someone to actually debug it -- for the curious its not hard to find.

Edited December 24, 2017 by JaredBGreat

[jabelar]

1 hour ago, JaredBGreat said:

Thanks for accusing me of doing things I did not do, asking for favors I did not ask for, and telling me things I already know, and assuming I don't know how to (or won't) do something I've done many, many times.  I was simply asking if anyone had any good information on the sub-systems involved, not for anyone to debug anything.  Anyway, I think I'll pass on your suggestion to create a huge core mod that adds debugging code to a large portion of the game.

Yes, it probably sounds accusatory (sorry) but was meant more as "tough love". Teaching you to learn things yourself. I'm only suggesting this because I feel you're ready for pushing to the next level of self-sufficiency.

 

There are two types of problems -- common ones in an area of general modding interest, and esoteric ones that are mostly personal interest. There are also two types of people asking questions -- those who are weak in programming and those that are strong. You have an esoteric interest and are a strong programmer so proper help is to teach the skills to debug. If you were asking a common question and were weak programmer I'd spoon feed you the answer...

 

No core mod is required. So I think you misunderstood my suggestion. Even though you don't suspect your own code, it is still most likely that the problem originates there (or in the way that it hooks into the vanilla code). There are certainly possible bugs in vanilla code (found one yesterday), but even then your code must have exposed it.

 

So start with instrumenting your own code. By the way, this is a good habit to get into anyway -- a proper code validation needs to confirm every code path. With your own classes confirm that they are (a) being called as expected (b) not taking excessive time in their own right.

 

If it is an actual memory leak, your IDE has ability to monitor that. Depending on your development environment google some instructions on debugging Java memory leaks. For example, Eclipse has the Memory Analyzer Tool. https://eclipsesource.com/blogs/2013/01/21/10-tips-for-using-the-eclipse-memory-analyzer/

 

 Lastly, if you really need to modify the vanilla classes you just copy them and add your stuff -- replacing chunk providers is a standard modding activity. This is just standard debug practice and worth learning, especially since tracing the execution of the vanilla code teaches a LOT about how Minecraft is architected. But you can also use the debugger in your IDE and set breakpoints.

 

Also, there is a built-in profiler that times some sections in the vanilla code which can help tell you if things are taking longer than expected.

 

So again sorry I sounded rude, but intended to be more like a sports coach yelling at you to push harder to get to next level. You can do it! I promise if you do the above suggestions you'll feel significant advancement in your programming prowess.

[JaredBGreat]

Well, I'd slept late and just rolled out of bed, so I was pretty grouchy myself.

 

Anyway, it may be that my own code is generating too many objects.  I started writing some memory management of my own in hopes it would be easier to profile by inserting debug code into that.  Then I realized I could add code to just count the objects, and got things like this:

 

[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:98]: There are...
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:99]:      180511 Basin nodes
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:100]:      9284 BiomeBasins
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:101]:      37139 ChunkTiles
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:102]:      3341 Regions
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:103]:      10031 Spatial noise generators

 

The numbers do go up and down a bit, so apparently its not a leak, they do get garbage collected.  But apparently I'm creating far more of these than I ever imagined (especially noise generators, which I thought I had only one in, and basin nodes -- well, especially everything I guess).  So, I need to figure out (1) where these are all coming from and (2) and good way to manage them.

 

Thanks.  Sorry for getting snarky earlier.

[jabelar]

1 hour ago, JaredBGreat said:

Well, I'd slept late and just rolled out of bed, so I was pretty grouchy myself.

 

Anyway, it may be that my own code is generating too many objects.  I started writing some memory management of my own in hopes it would be easier to profile by inserting debug code into that.  Then I realized I could add code to just count the objects, and got things like this:

 

[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:98]: There are...
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:99]:      180511 Basin nodes
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:100]:      9284 BiomeBasins
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:101]:      37139 ChunkTiles
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:102]:      3341 Regions
[16:11:25] [Server thread/INFO] [STDOUT]: [jaredbgreat.climaticbiome.generation.BetterBiomeProvider:cleanupCache:103]:      10031 Spatial noise generators

 

The numbers do go up and down a bit, so apparently its not a leak, they do get garbage collected.  But apparently I'm creating far more of these than I ever imagined (especially noise generators, which I thought I had only one in, and basin nodes -- well, especially everything I guess).  So, I need to figure out (1) where these are all coming from and (2) and good way to manage them.

 

Thanks.  Sorry for getting snarky earlier.

 

Cool. See that is progress! Once you have observability pretty much every programming problem is easy to solve.

 

Just keep working back and testing your assumptions on how things are working.

 

With regards to creating too many of something, just check out every where the constructor is called and ask whether it is really needed. You'll find that either you do need that many references but maybe instead of creating a new one you can retain one for reuse. Or you'll find that the whole section of code that is constructing it is being called unnecessarily.

 

Sounds like you're on track. But keep the thread posted. And if you get truly stuck of course we'll pitch in to help.

[JaredBGreat]

Well, I'm done with this for a couple days (I plan to visit my sister for Christmas tomorrow).

 

For now, I've added some memory management (caches with object pools for frequently used classes).  Most of my object are far less common than they were.  However, overall memory is still awfully high (not a big change), and I have found that some classes acquire new instances way too fast and don't let go of them -- notable the "Regions" (4096x4096 areas which each get landmass and climate nodes / attractors).  This leads me to think that my hunch that something is activating world gen way too far out or simply at incorrect coordinates.  I don't think my world gen is "rolling" and causing more on its own -- it only spreads while travelling and cause no lag (unless memory is depleted).  I'm not so much suspecting a bug in the vanilla code per se, as that I'm not interfacing with it correctly somewhere -- a case of playing with forces I don't understand.

 

Anyway, thanks for now.

 

EDIT: One of my experiments crashed with a null pointer in ChunkProviderServer, so maybe that's a good place to start looking for something that is asking my biome provider I wasn't expecting.

Edited December 25, 2017 by JaredBGreat

[JaredBGreat]

I've done more tests.  Applying some of the profiling out put in my original experimenting tool gives results like this when my biome system is run outside Minecraft:

 

Total memory usage: 
          USED MEMORY: 115 MB
     486 Basin nodes
     19449 BiomeBasins
     102781 ChunkTiles
     9 Regions
     4 Spatial noise generators

 

Where memory can spike to over 500 MB.  This has me confused as doing a few calculation on the objects it reports existing (based on the fields they contain and the size of the primitive types) only accounts for about 8 MB.  But it does appear my generation system may simply use far more memory than I realized.  Perhaps its too greedy for what it does, at least in the context of what most people would want and expect.

 

The I suspect the extra save files are from somewhere else.  I'm using a round-about way of getting data into the biomeCache, since its private in BiomeFinder; specifically ignoring the cache flag and always writing into the cache when it calls getBiomes(), since that is the only access I have (the way I wrote getBiomesForGeneration() requires this to work since it gets its data from the biomeCache instead of a genlayer like in vanilla).  What I suspect is happening is that something is calling this for distant locations (perhaps to check for possible structure locations of similar), and its not supposed to be cached when this happens.  Put differently, the problem comes from completely skipping the genlayers -- to fix this I'd need have my biome data written into a genlayer and accessed from there, so that I can leave the biomeCache to function as it was intended.  (Extra continental regions would also be a bi-product, as it would need the correct climate data for that location.)

 

I'm not sure I can get the memory usage down to an acceptable level -- not with this way of doing thing.  A system that was more memory efficient would simply not be at all the same system, and this system will never really be memory efficient; it just requires too much data (every chunk requires calculation on 48 fake chunks for one thing -- the calculation is fast but its a lot of extra proto-chunks being stored).

Edited December 27, 2017 by JaredBGreat

[jabelar]

If you're going to take a different approach, what about considering the ReplaceBiomeBlocks event? That event gets the world so you could try to access the biomes array and let the normal generation continue with the revised biomes. However, I just tried and it is tough because you'd need to use the following crazy code event.getWorld().getBiomeProvider().getBiomes(event.getWorld().getWorldInfo().getTerrainType().getChunkGenerator().biomesForGeneration, event.getX() * 16, event.getZ() * 16, 16, 16) and the biomesForGeneration is private so you'd have to use Java reflection. I think I'll put a pull request in to have the event updated to provide the biomes array as well, so in the future this will be easier.

 

Alternatively, you could copy the generation code (based on world type) into your event handler since the event also gets the ChunkPrimer and cancel the vanilla generation. 

 

Doing it this way should solve any problems with architectural issues on how you hook into the generation. After that it really depends on how efficient your algorithm is for coming up with the replacement biomes. If you still have memory problems at that point, it must be your algorithm.

Edited December 27, 2017 by jabelar

[JaredBGreat]

I've been away a few days, and haven't really looked at the event yet.  I often don't think of events for things like this.  I do know that getBiomesForGeneration() returns an array representing a 10x10 of locations at every four blocks which is used by the chunk provider to generate height maps that average across biome boundaries, and that getBiomes() returns and array of biomes within a chunk on a one block resolution and is used by the biome decorator to place things like trees and dirt.  I also have found there is something else (maybe several location) in accessing the biome cache directly without going through the biome provider, and that this is used for placing structures (writing to the biome array everything is treated as plains, so no temples, plains village in the deserts, etc.) -- I have no idea where that is so far.

 

I don't know if using the event would prevent the extra save files.  I do suspect that the memory usage will be high no mater what I do -- at least as long as I use this kind of large-scale land and climate model.

 

In the mean time I have been thinking of making a better caching system for my data -- well, that and having my mind wander back to my other project (possibly Doomlike Dungeons II or something along those lines).

Edited December 28, 2017 by JaredBGreat

[JaredBGreat]

I think the original question is solved -- even if some other issues aren't, so I'm marking it as such.  When and if the rest gets fixed will remain to be seen.  In the meantime, this topic is probably done for now.

[JaredBGreat]

One last update and then I'll let this die, since people were wondering what I'd figure out:

 

First, vanilla terrain gen in 1.10.x is more memory intensive than I realized at first.  The last time I looked at that it was for an older version (probably 1.6).  It seems my system may not actually use much if any more memory, and if a lot of terrain is generated quickly (e.g., flying around at high speed in spectator mode) with no modded world shows that vanilla will run out of memory and start to choke at least as fast (maybe faster, it seemed that way but I didn't time it).  My system also seems to generate terrain a bit faster than vanilla (though it has no rivers or beaches).  So maybe that isn't a problem.

 

Second, I think I've isolated the source of the extra save files, and know why they're so small.  It seems that the method areBiomesViable() in BiomeProvider is frequently run on random seeming, distant chunks.  The extra files also will appear without any modded world gen in 1.10, but do not appear in 1.12.  So I suspect these are being generated by an attempt to find a locations for large structures (woodland mansions?), and the generated region files is a side effect of generating these chunks which then get saved, a minor problem that was fixed in newer game versions --  and that it has nothing to do with the mod, and was never really an issue.

 

So that's it basically -- I was overly concerned about nothing, mistaking normal game behavior for a problem I'd caused; looking for solutions for problems that weren't really there because I didn't have a good understanding of how it was supposed to work.  If I get more done on this, the next post will probably be in a new thread in the mods section, and not too soon (I'm very busy again now).  Thanks everyone for all the help and advice.

[Daeruin]

Thanks for reporting back. The forum would be a TON more helpful if everyone did that instead of treating it like a chat room.