Forge *needs* to save textures after stitching them

[tuskiomi]

It's not really a question at this point. It is ridiculous that forge does not save the workload and dynamically generates texture maps every time. Here is how this can be done easily:

When loading for the first time:

1) save a SHA checksum of all mods UUIDs (name+version) . This will be used to validate that the current mod's textures are up to date.

2) save all pngs on the drive with the checksum, and a map of item/block to the drive.

3) save a SHA checksum of all images + maps to the drive as well, to prevent user tampering.

When loading after the first time:

1) load all the mods + textures, and generate a SHA checksum (This is also done the first load, but it doesn't save the sum this time)

2) verify that the sum matches the current textureset.

3) load the textures +maps. take a checksum (this should be very fast)

4) if the sum does not match, make a new textureset as if it is the first time booting, Otherwise:

5) skip stitching, and load fast.

It's something that's been an annoyance for many years now, and it really should be fixed.

[DaemonUmbra]

I'm not sure the performance gain will be as massive as you make it out to be

[DavidM]

There is too much things that might go wrong with “saving the stitched results” of mods.

1. The config might change. While the mod ID and version stay the same, a mod might change results in TextureStitchEvent based on the config values.

2. Some mods can have Easter eggs based on current time (some mods even have varying registry entries during holidays like Christmas to add in Easter eggs items).

3. Some modders might accidentally label the version wrong (i.e. forgot to bump the version). While this have almost no effect as of now, with your proposed system such mistakes can make the entire game unplayable.

 

In general, a checksum of mod ID and version is not a definitive method of checking whether the stitched texture map has changed. Due to the flexibility of Forge, it is hard (if not impossible) to check if the stitched texture would change.

However in 1.14 the loading of mods is parallel, so the overall loading time would not be so long.

Edited March 25, 2020 by DavidM

[Alpvax]

Hmmm... I wonder what would happen if someone wanted to change the textures with a resource pack? Which can be done in-game, and now instead of just loading and stitching the textures, it has to hash and save. And then the player decides they didn't like that resource pack after all...

[LexManos]

You're not even scratching the surface of why this isn't gunna happen. There is no sane way to do this that doesn't waste more time then it saves. 

 

but by all means, if you can think of something,  write it up and add benchmarks to it.

[tuskiomi]

 

On 3/24/2020 at 3:58 PM, DaemonUmbra said:

I'm not sure the performance gain will be as massive as you make it out to be

 

See the benchmark below. I think you are very wrong.

 

On 3/25/2020 at 3:08 AM, DavidM said:

There is too much things that might go wrong with “saving the stitched results” of mods.

1. The config might change. While the mod ID and version stay the same, a mod might change results in TextureStitchEvent based on the config values.

2. Some mods can have Easter eggs based on current time (some mods even have varying registry entries during holidays like Christmas to add in Easter eggs items).

3. Some modders might accidentally label the version wrong (i.e. forgot to bump the version). While this have almost no effect as of now, with your proposed system such mistakes can make the entire game unplayable.

 

In general, a checksum of mod ID and version is not a definitive method of checking whether the stitched texture map has changed. Due to the flexibility of Forge, it is hard (if not impossible) to check if the stitched texture would change.

However in 1.14 the loading of mods is parallel, so the overall loading time would not be so long.

 

1.) That Is really not something that a mod should have access to.

2.) Like in Minecraft, the easter eggs are still in the game and loaded even when it is not present. Same concept flies here.

3.) That's why you hash the textures as well, not just the mod versions and names

 

On 3/30/2020 at 10:00 AM, Alpvax said:

Hmmm... I wonder what would happen if someone wanted to change the textures with a resource pack? Which can be done in-game, and now instead of just loading and stitching the textures, it has to hash and save. And then the player decides they didn't like that resource pack after all...

If the textures change, the jar changes, and the hash will be different.

 

On 3/30/2020 at 3:22 PM, LexManos said:

You're not even scratching the surface of why this isn't gunna happen. There is no sane way to do this that doesn't waste more time then it saves. 

 

but by all means, if you can think of something,  write it up and add benchmarks to it.

Ok!

Here's my script that can load, hash ,and unload mod files in 356ms(+/- 40ms) on my computer, using the mods for the Feed-The-Beast Continuum modpack, during which the texture stitching takes 34.5 seconds, with 500ms of  human error.

 

import java.io.IOException;

public class ShaHasher {

    public static void main(String[] args) {
        String cmd = "7z h -scrcSHA1 mods";
        String result = "";
        long timeStart = System.currentTimeMillis();
        try {
            result = execCmd(cmd);
        } catch (IOException e) {
            e.printStackTrace();
        }
        System.out.println(result);
        System.out.println("^^^CMD OUTPUT^^^");
        System.out.println("Extracted SHA1 for data only: "+ getSHAContents(result));
        System.out.println("Extracted SHA1 for data and names: "+ getSHAContentsAndNames(result));
        
        System.out.print("Time taken: ");
        long timeStop = System.currentTimeMillis();
        System.out.print(timeStop-timeStart);
        System.out.println(" ms.");
        
    }

    public static String getSHAContents(String toExtractFrom) {
        int loc = toExtractFrom.indexOf("SHA1   for data:");
        loc +=16;
        while(toExtractFrom.charAt(loc) == ' ') {
            loc++;
        }
        int end = loc;
        while(toExtractFrom.charAt(end) != '\n') {
            end++;
        }
        return toExtractFrom.substring(loc,end-1);
    }
    
    public static String getSHAContentsAndNames(String toExtractFrom) {
        int loc = toExtractFrom.indexOf("SHA1   for data and names:");
        loc +=26;
        while(toExtractFrom.charAt(loc) == ' ') {
            loc++;
        }
        int end = loc;
        while(toExtractFrom.charAt(end) != '\n') {
            end++;
        }
        return toExtractFrom.substring(loc,end-1);
    }
    
    public static String execCmd(String cmd) throws java.io.IOException {
        Process proc = Runtime.getRuntime().exec(cmd);
        java.io.InputStream is = proc.getInputStream();
        java.util.Scanner s = (new java.util.Scanner(is)).useDelimiter("\\A");
        String val = "";
        if (s.hasNext()) {
            val = s.next();
        }
        else {
            val = "";
        }
        s.close();
        return val;
    }
}

 

here is the output of the program.

https://pastebin.com/7mTLdmkq

 

What else can I benchark for ya?

 

[DaemonUmbra]

FTB Continuum is a 1.12.2 pack, which is out of support.

 

The supported versions, 1.14.4 and 1.15.2 have parallel mod loading, which speeds up loading SUBSTANTIALLY.

 

And are we going to get into the amount of disk space such a system would eventually build up to? I don't feel like periodically cleaning my disk of cached texture atlases, especially since different packs with the same mods would cause a new atlas set to be cached, and a resource pack change would cause a re-cache, and pack devs would see their test installation bloat with every mod added to the pack.

 

To be honest to me it sounds like effort that would save maybe a second or two of loading at the cost of disk space. I'd rather wait a couple seconds.

[tuskiomi]

45 minutes ago, DaemonUmbra said:

FTB Continuum is a 1.12.2 pack, which is out of support.

 

The supported versions, 1.14.4 and 1.15.2 have parallel mod loading, which speeds up loading SUBSTANTIALLY.

 

And are we going to get into the amount of disk space such a system would eventually build up to? I don't feel like periodically cleaning my disk of cached texture atlases, especially since different packs with the same mods would cause a new atlas set to be cached, and a resource pack change would cause a re-cache, and pack devs would see their test installation bloat with every mod added to the pack.

 

To be honest to me it sounds like effort that would save maybe a second or two of loading at the cost of disk space. I'd rather wait a couple seconds.

I like the sound of parallel loading, but I seriously doubt that textures can be stitched in parallel, but if that's been parallelized, good work, i'm impressed.

One thing I'm not sure of is this statement "I don't feel like periodically cleaning my disk of cached texture atlases, especially since different packs with the same mods would cause a new atlas set to be cached, and a resource pack change would cause a re-cache, and pack devs would see their test installation bloat with every mod added to the pack."

 

Let's think of set theory.  We have set M. set M contains all mods in your mod pack.  any mod 'm' is a set containing three items: 1: s, the space that the mod takes up. 2: a set C containing source files for the mod. 3: a set R, which contains all of the resources that a mod uses. Similarly, we have an Atlas set A, for which there is one A for every one M. Set A contains all R sets within M, as well as a D set, for Directory.  The A set also contains a unique element s, which is the space that A takes up. 

 

Then it follows that so long as set C is lesser in space than set D, then Mc will take up less space than Ac. In short, The atlas will always take up less space than the mods, considering that most worlds are bigger than the mods, then space won't be an issue, especially if you cull old atlases automatically.

Edited April 1, 2020 by tuskiomi

[Alpvax]

11 hours ago, tuskiomi said:

Let's think of set theory.  We have set M. set M contains all mods in your mod pack.  any mod 'm' is a set containing three items: 1: s, the space that the mod takes up. 2: a set C containing source files for the mod. 3: a set R, which contains all of the resources that a mod uses. Similarly, we have an Atlas set A, for which there is one A for every one M. Set A contains all R sets within M, as well as a D set, for Directory.  The A set also contains a unique element s, which is the space that A takes up. 

You still haven't taken into account user resource packs (i.e. those which are not in a mod jar). You would have to re-hash every time someone changed a resource pack, including in-game.

[tuskiomi]

9 minutes ago, Alpvax said:

You still haven't taken into account user resource packs (i.e. those which are not in a mod jar). You would have to re-hash every time someone changed a resource pack, including in-game.

Even if hashing took 10x longer, it would still be 10x shorter than stitching.

[LexManos]

Of coarse simple single use code is going to be faster then unrelated complex code.

The kicker is the integration with Minecraft itself, and taking into account the things you so nonchalantly ignored, user configs dynamic textures, or any of the plethora of other things mods do that screw up the cache.

Also, what parts of the texture stitching event actually takes time? What parts could be accelerated? The actual slot allocation shouldn't be that time consuming. But it'd be worth quantifying.

So, as I said before, write something up and add benchmarks. And i'm talking real world implementations not unrelated theoretical code.

[h121]

Sorry for resurrecting an old topic, but I just found this thread and I've been wondering about why nobody does something about this for years.

 

There were questions about newer version, and just for reference the 128x version of the PureBDCraft texture pack (which I've been using since 1.11 days) takes a frigging 7 minutes to stitch with my latest modpack, the Stacia Expert 1.8.2. The game is completely unresponsive for the entire time.

 

This is on MC 1.16.4, and a relatively beefy computer (AMD 5600X, 32GB of DDR3200 memory, a fast PCIE4 NVME drive)

The modpack has >200 mods in it, and without the texture pack the it loads in about one minute. Reasonable, considering the amount of mods. However the seven minutes stitching time is on top of that - the total game load time from launcher is thus roughly 8 minutes. Does this sound reasonable to you?

Here you go if you want to try it yourself:

• https://www.curseforge.com/minecraft/modpacks/stacia-expert
• https://bdcraft.net/downloads/purebdcraft-minecraft/

I used to be a Java developer for many years, and in my opinion this system is broken...

[DaemonUmbra]

To reiterate... if you think it's such a good idea, do it yourself and actually provide numbers.

Thread locked.