Chunks, Saving Data to, And So on

[Draco18s 2416]

So, I seem to have a strange problem saving data to the chunk its associated with when the chunk is saved and unloaded.

 

Basically:

 

[22:22:57] [server thread/INFO]: Saving and pausing game...

[22:22:57] [server thread/INFO]: Saving chunks for level 'Copy of ASDF'/Overworld

[22:23:02] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:saveData:75]: Saving data for chunk [dim0](-4,0,16)

[22:23:02] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:saveData:81]: Saving tree: Tree[-61,67,265] 693

[22:23:02] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:saveData:81]: Saving tree: Tree[-58,64,271] 974

[22:23:07] [server thread/INFO]: Saving chunks for level 'Copy of ASDF'/Nether

[22:23:07] [server thread/INFO]: Saving chunks for level 'Copy of ASDF'/The End

[22:23:23] [Client thread/INFO] [sTDOUT]: [com.draco18s.wildlife.WildlifeEventHandler:chunkUnload:693]: Unloading Chunk: -4,16

[22:23:45] [server thread/INFO]: Saving and pausing game...

[22:23:45] [server thread/INFO]: Saving chunks for level 'Copy of ASDF'/Overworld

[22:23:48] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:saveData:75]: Saving data for chunk [dim0](-4,0,16)

[22:23:54] [server thread/INFO]: Saving chunks for level 'Copy of ASDF'/Nether

[22:23:54] [server thread/INFO]: Saving chunks for level 'Copy of ASDF'/The End

 

The data is saved (22:23:02).

 

Then the data is unloaded (22:23:23) as the player moved out of range.

 

Then the data (which is now been removed from memory) is saved again (22:23:48), resulting in data loss.

 

WTH.  Why is an unloaded chunk receiving a ChunkSave event?  Why does

if(event.getChunk().isChunkLoaded)

still return true for this actually-unloaded chunk?

 

How the hell do I fix this?

[Awesome_Spider 64]

That's weird. What version is this?

[Draco18s 2416]

Forge v1291

 

1.7.10

[diesieben07 7705]

You need to ignore unload events from the client.

[Draco18s 2416]

...that's odd, I thought I made sure to account for that.  But on that specific event, its missing.

 

Lets try this again.

 

Ah yes, that did it.  Thanks Diesieben.

[Draco18s 2416]

You need to ignore unload events from the client.

 

Ok, this fixed it for SSP, but not SMP.

 

[23:16:22] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.WildlifeEventHandler:chunkSave:343]: Saved chunk [dim0](21,0,14)

[23:16:22] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:saveData:68]: Saved tree Tree[336,82,237] 956

[23:16:22] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:saveData:68]: Saved tree Tree[342,84,233] 144

[23:18:10] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:printChunkInfo:367]: Trees in chunk [dim0](21,0,14)

[23:18:10] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:printChunkInfo:370]:    Tree[336,82,237] 968

[23:18:10] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:printChunkInfo:370]:    Tree[342,84,233] 156

[23:20:23] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.WildlifeEventHandler:chunkUnload:366]: Unloaded chunk [dim0](21,0,14)

[23:20:23] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.WildlifeEventHandler:chunkSave:343]: Saved chunk [dim0](21,0,14)

[23:21:16] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:printChunkInfo:367]: Trees in chunk [dim0](21,0,14)

[23:21:16] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:printChunkInfo:374]:    [None]

 

Now what?

[Abastro 123]

I found that when it unloads chunks, it saves the data directly after unloading the chunks.

Maybe it is causing the issue.

[Draco18s 2416]

I found that when it unloads chunks, it saves the data directly after unloading the chunks.

Maybe it is causing the issue.

 

Ok, so how do I fix it?

[Abastro 123]

Maybe saving the unloaded chunks in a map in Chunk Unload Event?

[Draco18s 2416]

Clunky, but it should work.

[Draco18s 2416]

For the love of god.

 

[13:08:51] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.WildlifeEventHandler:chunkGen:423]: Generated chunk [dim0](-17,0,-3)

[13:08:51] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:addTree:146]: Tree[-262,66,-41] 1061 added to chunk [dim0](-17,0,-3)

[13:08:51] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:addTree:146]: Tree[-261,66,-41] 7993 added to chunk [dim0](-17,0,-3)

[13:08:51] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.util.TreeDataHooks:addTree:146]: Tree[-257,73,-48] 5064 added to chunk [dim0](-17,0,-3)

[13:11:17] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.WildlifeEventHandler:chunkUnload:433]: Unloaded chunk [dim0](-17,0,-3)

[13:11:22] [server thread/INFO] [sTDOUT]: [com.draco18s.wildlife.WildlifeEventHandler:chunkLoad:385]: Loaded chunk [dim0](-17,0,-3)

 

Now I have chunks that generate and never get a save call before they're unloaded.  This is just as bad as getting a save call after I purged the data from memory.

 

http://static.fjcdn.com/pictures/Rage_562aea_329374.jpg[/img]

 

Do none of these events make god damn sense?  I should not have to build a data structure to track what chunks are in what state so that I don't inadvertently trash data I still need or overwrite good data with bad.  The game already freaking does this, but it can't be arsed to inform me!

 

I'm about ready to store the data in an indestructible bedrock-level tileentity.

[Abastro 123]

... I don't know why those issues can ever happen, from this code.. (Note that mine is 1.7.10.1160 so it can be different)

    /**
     * Unloads chunks that are marked to be unloaded. This is not guaranteed to unload every such chunk.
     */
    public boolean unloadQueuedChunks()
    {
        if (!this.worldObj.levelSaving)
        {
            for (ChunkCoordIntPair forced : this.worldObj.getPersistentChunks().keySet())
            {
                this.chunksToUnload.remove(ChunkCoordIntPair.chunkXZ2Int(forced.chunkXPos, forced.chunkZPos));
            }

            for (int i = 0; i < 100; ++i)
            {
                if (!this.chunksToUnload.isEmpty())
                {
                    Long olong = (Long)this.chunksToUnload.iterator().next();
                    Chunk chunk = (Chunk)this.loadedChunkHashMap.getValueByKey(olong.longValue());

                    if (chunk != null)
                    {
                        chunk.onChunkUnload();
                        this.safeSaveChunk(chunk);
                        this.safeSaveExtraChunkData(chunk);
                        this.loadedChunks.remove(chunk);
                        ForgeChunkManager.putDormantChunk(ChunkCoordIntPair.chunkXZ2Int(chunk.xPosition, chunk.zPosition), chunk);
                        if(loadedChunks.size() == 0 && ForgeChunkManager.getPersistentChunksFor(this.worldObj).size() == 0 && !DimensionManager.shouldLoadSpawn(this.worldObj.provider.dimensionId)){
                            DimensionManager.unloadWorld(this.worldObj.provider.dimensionId);
                            return currentChunkProvider.unloadQueuedChunks();
                        }
                    }

                    this.chunksToUnload.remove(olong);
                    this.loadedChunkHashMap.remove(olong.longValue());
                }
            }

            if (this.currentChunkLoader != null)
            {
                this.currentChunkLoader.chunkTick();
            }
        }

        return this.currentChunkProvider.unloadQueuedChunks();
    }

So it surely unloads the chunk before it saves the chunk data..

  (Chunk#onChunkUnload surely marks the chunk as unloaded)

The only another way the ChunkSave event got called is WorldServer#saveAllChunks(), which is only called when world is saved..

[Draco18s 2416]

So it surely unloads the chunk before it saves the chunk data..

  (Chunk#onChunkUnload surely marks the chunk as unloaded)

The only another way the ChunkSave event got called is WorldServer#saveAllChunks(), which is only called when world is saved..

 

So, in other words, I need two events to happen for me to safely say "this data is no longer relevant."  First a chunkUnload, then a chunkSave (but watch out, saves can happen any time!)

 

Yeah.  While I'm sure that that makes sense, its a complete hassle from a "I want this to work" perspective.  E.g. "chunkUnload" should be the last thing that happens to a chunk before it's loaded again.  The event should be telling mods "this chunk is leaving, whatever cleanup you need to do, do it."

 

       loaded
          V
gen -> (active) <-> save
                     V
                  unloaded

 

Instead, the flow chart actually looks like this:

 

       loaded
          V
gen -> (active) <-> save
          V
       unloaded -> save

 

Note the two instances of "save."  They are not identical, but it is impossible to tell the difference between them without storing extra data as the events trigger.

[diesieben07 7705]

I'll just throw this in here:

"Just" use a WeakHashMap and don't even bother handling the unload.

[Draco18s 2416]

I'll just throw this in here:

"Just" use a WeakHashMap and don't even bother handling the unload.

 

Can't.  The save/load and tick operations happen on separate threads (that is, strait up using a HashMap causes concurrent modification exceptions) and there is no "ConcurrentWeakHashMap."

 

Not to mention that I tick the data, so its kind of important to know if the chunk the data is associated with is active.

[diesieben07 7705]

Ok. Here goes (not saying that this is ideal, this stuff needs to be fixed up. A lot of it was probably broken (because let's face it, this is broken) when the threaded chunk saving was introduced):

 

First, there is a "ConcurrentWeakHashMap", as always Guava to the rescue:

ConcurrentMap<Chunk, V> map = new MapMaker().weakKeys().concurrencyLevel(2).makeMap();

 

While exploring this path, I found the following: onChunkUnload sets

isChunkLoaded

to false. So in chunk unload you do nothing. Then in chunk save you check if isChunkLoaded is false and if so unload your data (you don't even need the WeakHashMap then).

[Draco18s 2416]

While exploring this path, I found the following: onChunkUnload sets

isChunkLoaded

to false. So in chunk unload you do nothing. Then in chunk save you check if isChunkLoaded is false and if so unload your data (you don't even need the WeakHashMap then).

 

Thanks for the digging, as always, diesieben07.

I've only managed to glean so much from the SRG'd vanilla source on my own.  Most of what I've figured out has been from observing the behavior of a single chunk as I interact with the system and with suggestions as I run into stumbling blocks.

 

There's a number of headaches going on with this code, and this is only one part of it.  It seems that even with a 3-chunk-deferred-radius for the generation of the data, it still doesn't find all of the trees, especially in a jungle.  Neither Keybounce nor I can figure out why.  But that's a problem for another thread.