Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

Forge *needs* to save textures after stitching them


Recommended Posts

It's not really a question at this point. It is ridiculous that forge does not save the workload and dynamically generates texture maps every time. Here is how this can be done easily:

When loading for the first time:

1) save a SHA checksum of all mods UUIDs (name+version) . This will be used to validate that the current mod's textures are up to date.

2) save all pngs on the drive with the checksum, and a map of item/block to the drive.

3) save a SHA checksum of all images + maps to the drive as well, to prevent user tampering.

When loading after the first time:

1) load all the mods + textures, and generate a SHA checksum (This is also done the first load, but it doesn't save the sum this time)

2) verify that the sum matches the current textureset.

3) load the textures +maps. take a checksum (this should be very fast)

4) if the sum does not match, make a new textureset as if it is the first time booting, Otherwise:

5) skip stitching, and load fast.

It's something that's been an annoyance for many years now, and it really should be fixed.

Also, I should have you know that you are reading my signature.

Link to post
Share on other sites

I'm not sure the performance gain will be as massive as you make it out to be

This is my Forum Signature, I am currently attempting to transform it into a small guide for fixing easier issues using spoiler blocks to keep things tidy.

 

As the most common issue I feel I should put this outside the main bulk:

The only official source for Forge is https://files.minecraftforge.net, and the only site I trust for getting mods is CurseForge.

If you use any site other than these, please take a look at the StopModReposts project and install their browser extension, I would also advise running a virus scan.

 

For players asking for assistance with Forge please expand the spoiler below and read the appropriate section(s) in its/their entirety.

Spoiler

Logs (Most issues require logs to diagnose):

Spoiler

Please post logs using one of the following sites (Thank you Lumber Wizard for the list):

https://gist.github.com/100MB Requires member (Free)

https://pastebin.com/: 512KB as guest, 10MB as Pro ($$$)

https://hastebin.com/: 400KB

Do NOT use sites like Mediafire, Dropbox, OneDrive, Google Drive, or a site that has a countdown before offering downloads.

 

What to provide:

...for Crashes and Runtime issues:

Minecraft 1.14.4 and newer:

Post debug.log

Older versions:

Please update...

 

...for Installer Issues:

Post your installer log, found in the same place you ran the installer

This log will be called either installer.log or named the same as the installer but with .log on the end

Note for Windows users:

Windows hides file extensions by default so the installer may appear without the .jar extension then when the .log is added the log will appear with the .jar extension

 

Where to get it:

Mojang Launcher: When using the Mojang launcher debug.log is found in .minecraft\logs.

 

Curse/Overwolf: If you are using the Curse Launcher, their configurations break Forge's log settings, fortunately there is an easier workaround than I originally thought, this works even with Curse's installation of the Minecraft launcher as long as it is not launched THROUGH Twitch:

Spoiler
  1. Make sure you have the correct version of Forge installed (some packs are heavily dependent on one specific build of Forge)
  2. Make a launcher profile targeting this version of Forge.
  3. Set the launcher profile's GameDir property to the pack's instance folder (not the instances folder, the folder that has the pack's name on it).
  4. Now launch the pack through that profile and follow the "Mojang Launcher" instructions above.

Video:

Spoiler

 

 

 

or alternately, 

 

Fallback ("No logs are generated"):

If you don't see logs generated in the usual place, provide the launcher_log.txt from .minecraft

 

Server Not Starting:

Spoiler

If your server does not start or a command window appears and immediately goes away, run the jar manually and provide the output.

 

Reporting Illegal/Inappropriate Adfocus Ads:

Spoiler

Get a screenshot of the URL bar or copy/paste the whole URL into a thread on the General Discussion board with a description of the Ad.

Lex will need the Ad ID contained in that URL to report it to Adfocus' support team.

 

Posting your mod as a GitHub Repo:

Spoiler

When you have an issue with your mod the most helpful thing you can do when asking for help is to provide your code to those helping you. The most convenient way to do this is via GitHub or another source control hub.

When setting up a GitHub Repo it might seem easy to just upload everything, however this method has the potential for mistakes that could lead to trouble later on, it is recommended to use a Git client or to get comfortable with the Git command line. The following instructions will use the Git Command Line and as such they assume you already have it installed and that you have created a repository.

 

  1. Open a command prompt (CMD, Powershell, Terminal, etc).
  2. Navigate to the folder you extracted Forge’s MDK to (the one that had all the licenses in).
  3. Run the following commands:
    1. git init
    2. git remote add origin [Your Repository's URL]
      • In the case of GitHub it should look like: https://GitHub.com/[Your Username]/[Repo Name].git
    3. git fetch
    4. git checkout --track origin/master
    5. git stage *
    6. git commit -m "[Your commit message]"
    7. git push
  4. Navigate to GitHub and you should now see most of the files.
    • note that it is intentional that some are not synced with GitHub and this is done with the (hidden) .gitignore file that Forge’s MDK has provided (hence the strictness on which folder git init is run from)
  5. Now you can share your GitHub link with those who you are asking for help.

[Workaround line, please ignore]

 

Link to post
Share on other sites

There is too much things that might go wrong with “saving the stitched results” of mods.

1. The config might change. While the mod ID and version stay the same, a mod might change results in TextureStitchEvent based on the config values.

2. Some mods can have Easter eggs based on current time (some mods even have varying registry entries during holidays like Christmas to add in Easter eggs items).

3. Some modders might accidentally label the version wrong (i.e. forgot to bump the version). While this have almost no effect as of now, with your proposed system such mistakes can make the entire game unplayable.

 

In general, a checksum of mod ID and version is not a definitive method of checking whether the stitched texture map has changed. Due to the flexibility of Forge, it is hard (if not impossible) to check if the stitched texture would change.

However in 1.14 the loading of mods is parallel, so the overall loading time would not be so long.

Edited by DavidM

Some tips:

Spoiler

Modder Support:

Spoiler

1. Do not follow tutorials on YouTube, especially TechnoVision (previously called Loremaster) and HarryTalks, due to their promotion of bad practice and usage of outdated code.

2. Always post your code.

3. Never copy and paste code. You won't learn anything from doing that.

4. 

Quote

Programming via Eclipse's hotfixes will get you nowhere

5. Learn to use your IDE, especially the debugger.

6.

Quote

The "picture that's worth 1000 words" only works if there's an obvious problem or a freehand red circle around it.

Support & Bug Reports:

Spoiler

1. Read the EAQ before asking for help. Remember to provide the appropriate log(s).

2. Versions below 1.11 are no longer supported due to their age. Update to a modern version of Minecraft to receive support.

 

 

Link to post
Share on other sites

Hmmm... I wonder what would happen if someone wanted to change the textures with a resource pack? Which can be done in-game, and now instead of just loading and stitching the textures, it has to hash and save. And then the player decides they didn't like that resource pack after all...

Link to post
Share on other sites

You're not even scratching the surface of why this isn't gunna happen. There is no sane way to do this that doesn't waste more time then it saves. 

 

but by all means, if you can think of something,  write it up and add benchmarks to it.

  • Like 1

I do Forge for free, however the servers to run it arn't free, so anything is appreciated.
Patreon: http://www.patreon.com/lexmanos
Paypal: http://paypal.me/LexManos

BitCoin: 1Q8rWvUNMM2T1ZfDaFeeYQyVXtYoeT6tTn

Link to post
Share on other sites

 

On 3/24/2020 at 3:58 PM, DaemonUmbra said:

I'm not sure the performance gain will be as massive as you make it out to be

 

See the benchmark below. I think you are very wrong.

 

On 3/25/2020 at 3:08 AM, DavidM said:

There is too much things that might go wrong with “saving the stitched results” of mods.

1. The config might change. While the mod ID and version stay the same, a mod might change results in TextureStitchEvent based on the config values.

2. Some mods can have Easter eggs based on current time (some mods even have varying registry entries during holidays like Christmas to add in Easter eggs items).

3. Some modders might accidentally label the version wrong (i.e. forgot to bump the version). While this have almost no effect as of now, with your proposed system such mistakes can make the entire game unplayable.

 

In general, a checksum of mod ID and version is not a definitive method of checking whether the stitched texture map has changed. Due to the flexibility of Forge, it is hard (if not impossible) to check if the stitched texture would change.

However in 1.14 the loading of mods is parallel, so the overall loading time would not be so long.

 

1.) That Is really not something that a mod should have access to.

2.) Like in Minecraft, the easter eggs are still in the game and loaded even when it is not present. Same concept flies here.

3.) That's why you hash the textures as well, not just the mod versions and names

 

On 3/30/2020 at 10:00 AM, Alpvax said:

Hmmm... I wonder what would happen if someone wanted to change the textures with a resource pack? Which can be done in-game, and now instead of just loading and stitching the textures, it has to hash and save. And then the player decides they didn't like that resource pack after all...

If the textures change, the jar changes, and the hash will be different.

 

On 3/30/2020 at 3:22 PM, LexManos said:

You're not even scratching the surface of why this isn't gunna happen. There is no sane way to do this that doesn't waste more time then it saves. 

 

but by all means, if you can think of something,  write it up and add benchmarks to it.

Ok!

Here's my script that can load, hash ,and unload mod files in 356ms(+/- 40ms) on my computer, using the mods for the Feed-The-Beast Continuum modpack, during which the texture stitching takes 34.5 seconds, with 500ms of  human error.

 

import java.io.IOException;

public class ShaHasher {

    public static void main(String[] args) {
        String cmd = "7z h -scrcSHA1 mods";
        String result = "";
        long timeStart = System.currentTimeMillis();
        try {
            result = execCmd(cmd);
        } catch (IOException e) {
            e.printStackTrace();
        }
        System.out.println(result);
        System.out.println("^^^CMD OUTPUT^^^");
        System.out.println("Extracted SHA1 for data only: "+ getSHAContents(result));
        System.out.println("Extracted SHA1 for data and names: "+ getSHAContentsAndNames(result));
        
        System.out.print("Time taken: ");
        long timeStop = System.currentTimeMillis();
        System.out.print(timeStop-timeStart);
        System.out.println(" ms.");
        
    }

    public static String getSHAContents(String toExtractFrom) {
        int loc = toExtractFrom.indexOf("SHA1   for data:");
        loc +=16;
        while(toExtractFrom.charAt(loc) == ' ') {
            loc++;
        }
        int end = loc;
        while(toExtractFrom.charAt(end) != '\n') {
            end++;
        }
        return toExtractFrom.substring(loc,end-1);
    }
    
    public static String getSHAContentsAndNames(String toExtractFrom) {
        int loc = toExtractFrom.indexOf("SHA1   for data and names:");
        loc +=26;
        while(toExtractFrom.charAt(loc) == ' ') {
            loc++;
        }
        int end = loc;
        while(toExtractFrom.charAt(end) != '\n') {
            end++;
        }
        return toExtractFrom.substring(loc,end-1);
    }
    
    public static String execCmd(String cmd) throws java.io.IOException {
        Process proc = Runtime.getRuntime().exec(cmd);
        java.io.InputStream is = proc.getInputStream();
        java.util.Scanner s = (new java.util.Scanner(is)).useDelimiter("\\A");
        String val = "";
        if (s.hasNext()) {
            val = s.next();
        }
        else {
            val = "";
        }
        s.close();
        return val;
    }
}

 

here is the output of the program.

https://pastebin.com/7mTLdmkq

 

What else can I benchark for ya?

 

Also, I should have you know that you are reading my signature.

Link to post
Share on other sites

FTB Continuum is a 1.12.2 pack, which is out of support.

 

The supported versions, 1.14.4 and 1.15.2 have parallel mod loading, which speeds up loading SUBSTANTIALLY.

 

And are we going to get into the amount of disk space such a system would eventually build up to? I don't feel like periodically cleaning my disk of cached texture atlases, especially since different packs with the same mods would cause a new atlas set to be cached, and a resource pack change would cause a re-cache, and pack devs would see their test installation bloat with every mod added to the pack.

 

To be honest to me it sounds like effort that would save maybe a second or two of loading at the cost of disk space. I'd rather wait a couple seconds.

  • Like 1

This is my Forum Signature, I am currently attempting to transform it into a small guide for fixing easier issues using spoiler blocks to keep things tidy.

 

As the most common issue I feel I should put this outside the main bulk:

The only official source for Forge is https://files.minecraftforge.net, and the only site I trust for getting mods is CurseForge.

If you use any site other than these, please take a look at the StopModReposts project and install their browser extension, I would also advise running a virus scan.

 

For players asking for assistance with Forge please expand the spoiler below and read the appropriate section(s) in its/their entirety.

Spoiler

Logs (Most issues require logs to diagnose):

Spoiler

Please post logs using one of the following sites (Thank you Lumber Wizard for the list):

https://gist.github.com/100MB Requires member (Free)

https://pastebin.com/: 512KB as guest, 10MB as Pro ($$$)

https://hastebin.com/: 400KB

Do NOT use sites like Mediafire, Dropbox, OneDrive, Google Drive, or a site that has a countdown before offering downloads.

 

What to provide:

...for Crashes and Runtime issues:

Minecraft 1.14.4 and newer:

Post debug.log

Older versions:

Please update...

 

...for Installer Issues:

Post your installer log, found in the same place you ran the installer

This log will be called either installer.log or named the same as the installer but with .log on the end

Note for Windows users:

Windows hides file extensions by default so the installer may appear without the .jar extension then when the .log is added the log will appear with the .jar extension

 

Where to get it:

Mojang Launcher: When using the Mojang launcher debug.log is found in .minecraft\logs.

 

Curse/Overwolf: If you are using the Curse Launcher, their configurations break Forge's log settings, fortunately there is an easier workaround than I originally thought, this works even with Curse's installation of the Minecraft launcher as long as it is not launched THROUGH Twitch:

Spoiler
  1. Make sure you have the correct version of Forge installed (some packs are heavily dependent on one specific build of Forge)
  2. Make a launcher profile targeting this version of Forge.
  3. Set the launcher profile's GameDir property to the pack's instance folder (not the instances folder, the folder that has the pack's name on it).
  4. Now launch the pack through that profile and follow the "Mojang Launcher" instructions above.

Video:

Spoiler

 

 

 

or alternately, 

 

Fallback ("No logs are generated"):

If you don't see logs generated in the usual place, provide the launcher_log.txt from .minecraft

 

Server Not Starting:

Spoiler

If your server does not start or a command window appears and immediately goes away, run the jar manually and provide the output.

 

Reporting Illegal/Inappropriate Adfocus Ads:

Spoiler

Get a screenshot of the URL bar or copy/paste the whole URL into a thread on the General Discussion board with a description of the Ad.

Lex will need the Ad ID contained in that URL to report it to Adfocus' support team.

 

Posting your mod as a GitHub Repo:

Spoiler

When you have an issue with your mod the most helpful thing you can do when asking for help is to provide your code to those helping you. The most convenient way to do this is via GitHub or another source control hub.

When setting up a GitHub Repo it might seem easy to just upload everything, however this method has the potential for mistakes that could lead to trouble later on, it is recommended to use a Git client or to get comfortable with the Git command line. The following instructions will use the Git Command Line and as such they assume you already have it installed and that you have created a repository.

 

  1. Open a command prompt (CMD, Powershell, Terminal, etc).
  2. Navigate to the folder you extracted Forge’s MDK to (the one that had all the licenses in).
  3. Run the following commands:
    1. git init
    2. git remote add origin [Your Repository's URL]
      • In the case of GitHub it should look like: https://GitHub.com/[Your Username]/[Repo Name].git
    3. git fetch
    4. git checkout --track origin/master
    5. git stage *
    6. git commit -m "[Your commit message]"
    7. git push
  4. Navigate to GitHub and you should now see most of the files.
    • note that it is intentional that some are not synced with GitHub and this is done with the (hidden) .gitignore file that Forge’s MDK has provided (hence the strictness on which folder git init is run from)
  5. Now you can share your GitHub link with those who you are asking for help.

[Workaround line, please ignore]

 

Link to post
Share on other sites
45 minutes ago, DaemonUmbra said:

FTB Continuum is a 1.12.2 pack, which is out of support.

 

The supported versions, 1.14.4 and 1.15.2 have parallel mod loading, which speeds up loading SUBSTANTIALLY.

 

And are we going to get into the amount of disk space such a system would eventually build up to? I don't feel like periodically cleaning my disk of cached texture atlases, especially since different packs with the same mods would cause a new atlas set to be cached, and a resource pack change would cause a re-cache, and pack devs would see their test installation bloat with every mod added to the pack.

 

To be honest to me it sounds like effort that would save maybe a second or two of loading at the cost of disk space. I'd rather wait a couple seconds.

I like the sound of parallel loading, but I seriously doubt that textures can be stitched in parallel, but if that's been parallelized, good work, i'm impressed.

One thing I'm not sure of is this statement "I don't feel like periodically cleaning my disk of cached texture atlases, especially since different packs with the same mods would cause a new atlas set to be cached, and a resource pack change would cause a re-cache, and pack devs would see their test installation bloat with every mod added to the pack."

 

Let's think of set theory.  We have set M. set M contains all mods in your mod pack.  any mod 'm' is a set containing three items: 1: s, the space that the mod takes up. 2: a set C containing source files for the mod. 3: a set R, which contains all of the resources that a mod uses. Similarly, we have an Atlas set A, for which there is one A for every one M. Set A contains all R sets within M, as well as a D set, for Directory.  The A set also contains a unique element s, which is the space that A takes up. 

 

Then it follows that so long as set C is lesser in space than set D, then Mc will take up less space than Ac. In short, The atlas will always take up less space than the mods, considering that most worlds are bigger than the mods, then space won't be an issue, especially if you cull old atlases automatically.

Edited by tuskiomi

Also, I should have you know that you are reading my signature.

Link to post
Share on other sites
11 hours ago, tuskiomi said:

Let's think of set theory.  We have set M. set M contains all mods in your mod pack.  any mod 'm' is a set containing three items: 1: s, the space that the mod takes up. 2: a set C containing source files for the mod. 3: a set R, which contains all of the resources that a mod uses. Similarly, we have an Atlas set A, for which there is one A for every one M. Set A contains all R sets within M, as well as a D set, for Directory.  The A set also contains a unique element s, which is the space that A takes up. 

You still haven't taken into account user resource packs (i.e. those which are not in a mod jar). You would have to re-hash every time someone changed a resource pack, including in-game.

Link to post
Share on other sites
9 minutes ago, Alpvax said:

You still haven't taken into account user resource packs (i.e. those which are not in a mod jar). You would have to re-hash every time someone changed a resource pack, including in-game.

Even if hashing took 10x longer, it would still be 10x shorter than stitching.

Also, I should have you know that you are reading my signature.

Link to post
Share on other sites

Of coarse simple single use code is going to be faster then unrelated complex code.

The kicker is the integration with Minecraft itself, and taking into account the things you so nonchalantly ignored, user configs dynamic textures, or any of the plethora of other things mods do that screw up the cache.

Also, what parts of the texture stitching event actually takes time? What parts could be accelerated? The actual slot allocation shouldn't be that time consuming. But it'd be worth quantifying.

So, as I said before, write something up and add benchmarks. And i'm talking real world implementations not unrelated theoretical code.

I do Forge for free, however the servers to run it arn't free, so anything is appreciated.
Patreon: http://www.patreon.com/lexmanos
Paypal: http://paypal.me/LexManos

BitCoin: 1Q8rWvUNMM2T1ZfDaFeeYQyVXtYoeT6tTn

Link to post
Share on other sites
  • 9 months later...

Sorry for resurrecting an old topic, but I just found this thread and I've been wondering about why nobody does something about this for years.

 

There were questions about newer version, and just for reference the 128x version of the PureBDCraft texture pack (which I've been using since 1.11 days) takes a frigging 7 minutes to stitch with my latest modpack, the Stacia Expert 1.8.2. The game is completely unresponsive for the entire time.

 

This is on MC 1.16.4, and a relatively beefy computer (AMD 5600X, 32GB of DDR3200 memory, a fast PCIE4 NVME drive)

The modpack has >200 mods in it, and without the texture pack the it loads in about one minute. Reasonable, considering the amount of mods. However the seven minutes stitching time is on top of that - the total game load time from launcher is thus roughly 8 minutes. Does this sound reasonable to you?

Here you go if you want to try it yourself:

I used to be a Java developer for many years, and in my opinion this system is broken...

Link to post
Share on other sites

To reiterate... if you think it's such a good idea, do it yourself and actually provide numbers.

Thread locked.

  • Like 1

This is my Forum Signature, I am currently attempting to transform it into a small guide for fixing easier issues using spoiler blocks to keep things tidy.

 

As the most common issue I feel I should put this outside the main bulk:

The only official source for Forge is https://files.minecraftforge.net, and the only site I trust for getting mods is CurseForge.

If you use any site other than these, please take a look at the StopModReposts project and install their browser extension, I would also advise running a virus scan.

 

For players asking for assistance with Forge please expand the spoiler below and read the appropriate section(s) in its/their entirety.

Spoiler

Logs (Most issues require logs to diagnose):

Spoiler

Please post logs using one of the following sites (Thank you Lumber Wizard for the list):

https://gist.github.com/100MB Requires member (Free)

https://pastebin.com/: 512KB as guest, 10MB as Pro ($$$)

https://hastebin.com/: 400KB

Do NOT use sites like Mediafire, Dropbox, OneDrive, Google Drive, or a site that has a countdown before offering downloads.

 

What to provide:

...for Crashes and Runtime issues:

Minecraft 1.14.4 and newer:

Post debug.log

Older versions:

Please update...

 

...for Installer Issues:

Post your installer log, found in the same place you ran the installer

This log will be called either installer.log or named the same as the installer but with .log on the end

Note for Windows users:

Windows hides file extensions by default so the installer may appear without the .jar extension then when the .log is added the log will appear with the .jar extension

 

Where to get it:

Mojang Launcher: When using the Mojang launcher debug.log is found in .minecraft\logs.

 

Curse/Overwolf: If you are using the Curse Launcher, their configurations break Forge's log settings, fortunately there is an easier workaround than I originally thought, this works even with Curse's installation of the Minecraft launcher as long as it is not launched THROUGH Twitch:

Spoiler
  1. Make sure you have the correct version of Forge installed (some packs are heavily dependent on one specific build of Forge)
  2. Make a launcher profile targeting this version of Forge.
  3. Set the launcher profile's GameDir property to the pack's instance folder (not the instances folder, the folder that has the pack's name on it).
  4. Now launch the pack through that profile and follow the "Mojang Launcher" instructions above.

Video:

Spoiler

 

 

 

or alternately, 

 

Fallback ("No logs are generated"):

If you don't see logs generated in the usual place, provide the launcher_log.txt from .minecraft

 

Server Not Starting:

Spoiler

If your server does not start or a command window appears and immediately goes away, run the jar manually and provide the output.

 

Reporting Illegal/Inappropriate Adfocus Ads:

Spoiler

Get a screenshot of the URL bar or copy/paste the whole URL into a thread on the General Discussion board with a description of the Ad.

Lex will need the Ad ID contained in that URL to report it to Adfocus' support team.

 

Posting your mod as a GitHub Repo:

Spoiler

When you have an issue with your mod the most helpful thing you can do when asking for help is to provide your code to those helping you. The most convenient way to do this is via GitHub or another source control hub.

When setting up a GitHub Repo it might seem easy to just upload everything, however this method has the potential for mistakes that could lead to trouble later on, it is recommended to use a Git client or to get comfortable with the Git command line. The following instructions will use the Git Command Line and as such they assume you already have it installed and that you have created a repository.

 

  1. Open a command prompt (CMD, Powershell, Terminal, etc).
  2. Navigate to the folder you extracted Forge’s MDK to (the one that had all the licenses in).
  3. Run the following commands:
    1. git init
    2. git remote add origin [Your Repository's URL]
      • In the case of GitHub it should look like: https://GitHub.com/[Your Username]/[Repo Name].git
    3. git fetch
    4. git checkout --track origin/master
    5. git stage *
    6. git commit -m "[Your commit message]"
    7. git push
  4. Navigate to GitHub and you should now see most of the files.
    • note that it is intentional that some are not synced with GitHub and this is done with the (hidden) .gitignore file that Forge’s MDK has provided (hence the strictness on which folder git init is run from)
  5. Now you can share your GitHub link with those who you are asking for help.

[Workaround line, please ignore]

 

Link to post
Share on other sites
Guest
This topic is now closed to further replies.


×
×
  • Create New...

Important Information

By using this site, you agree to our Privacy Policy.