1.9 good method for syncing large amounts of information when connect to server?

[UberAffe]

Title?

I have a lot of information that needs to be synced between the player and the server.

It isn't essential to sync everything before the player loads. This is essentially a db of item models, I'm still working on a method to only require a full sync the first time a player connects and subsequent connections would only sync the changes since the last sync. But I still need to do the initial sync.

So far I have tried two methods of syncing using nbt the first one was just saving the information to nbt normally and the second way was by serializing the information into a byte array and then storing it as a string but both methods exceed the max packet size in more cases than I am willing to cut out.

 

This leads me to believe that I should revisit my method for handling the models. So question 2:

If anyone knows the in's out's of the json modeling system, maybe you can help me understand if it can handle what I need.

 

 

Edit:

I was definitely being dumb with the packets, When I sent them as byte arrays it fit into a packet size just fine. However I still would like to avoid syncing so much information so the second question still stands.

[Leviathan143]

I assume this is for Light Drafter?

What exactly are you sending, who is sending it and who receives it?

[coolAlias]

The server generally shouldn't have anything to do with models - those are really only to render things on the client.

 

Why can't you ship or load the models client-side, like vanilla JSON models do?

[UberAffe]

Yes it is.

 

Players design items using a list of parts and colors (or load a design from a file) and register it with the server. There will be items available by default also. When a player first connects to a server the server will send 1 packet per registered item to the player. After that any time a new item gets registered all connected players will get a packet to sync that item. I am working on a way to store all synced items on the client side so they wont need to be synced again, I still don't have everything figured out everything to make this safe.

 

The information that is getting sent in the sync packet is:

name of the item,

type of the item(pick,sword, etc)

partmap(hashmap<relativeLocationOfPart, part>)

  part information:

  color,

  type of part

 

The partmap is analyzed to determine the properties of the item as well as the model for the item.

[coolAlias]

Ah, I see. Well in that case, that's probably the best you can do - send a giant packet (or several) when the player connects to the server or joins the world, and then keep them updated with new or altered varieties as they come.

 

Will it be slow? Probably, especially as the number of items grows, but that's the price for breaking out of the framework of static model definitions loaded directly from the client computers. I honestly wouldn't worry too much about performance until it becomes an issue, though, as it might never become one.

[UberAffe]

I know I can't get away from syncing item information to the player but I'm wondering if the information could be simplified if I could use the model system. Knowing that each part fits within a 1/16th block size and every part has a specific model that is known during pre-postInit. Parts might be added through an api. Is the model system robust enough to handle conditions like this?

[coolAlias]

So assuming a full block, you have 16 x 16 x 16 = 4096 possible positions for each model, and an unknown number of model parts.

 

Will each model part be indexed somehow, e.g. as a hashmap containing a unique integer key to each model part definition?

 

That's probably your best-case scenario, and it's pretty bad: one integer to store the location and one integer to store the model part for each part of the whole, so if you had a model consisting of 20 parts, the minimum you could send would be 40 integers, with a maximum of 4096 x 2 or 8192 integers per complete model. That's not a trivial amount of data.

 

Now you could try to get fancy and try to figure out parts of models that don't need to be set, e.g. assuming that your 'parts' are actually pixel portions, then if there is a 4 x 4 x 4 cubic section where each face is fully covered with model parts, then you probably don't need to send the inner 3 x 3 x 3 section, but you'd probably need to come up with a fairly efficient algorithm to make it worthwhile.

 

I'm not really understanding your design with respect to how it is supposed to interact with the model system and how, exactly, you expect it to function in terms of the game, so the above is all just speculation on my part. If you explain in more detail, perhaps that will give people more of an idea on how to help you.

 

EDIT: On another note - why aren't you using the JSON model system? People are already accustomed to creating resource packs, and sending a compressed JSON file over the network would be trivial and usually pretty light-weight, though you'd have to figure a way to send textures as well. I don't know... the whole idea sounds like a performance nightmare in the making, but who knows, maybe someone will come up with a brilliant solution?

[UberAffe]

lets see if I can explain this in a coherent way

 

All usable parts get registered during initilization on both client and server side.

A Part has a SideOnly(Side.Client) method that returns a list of BakedQuads that describe the part.

An item (Draftable) has a hashmap of string (relative x,y,z) and part that describes the relative layout of parts.

The BakedQuad method in Part takes the relative x,y,z location and applies the offset before returning the BakedQuads.

When a Draftable gets registered on the client side a BakedModel gets generated from the compiled list of BakeQuads from the parts.

 

Each Draftable gets registered on the server and then updates the client if needed.

Each Draftable is an instance of a CoreType.

Each CoreType has a default instance that acts as the item registered with minecraft.

I maintain a registry of CoreTypes that a Draftable can be created with and the are all known during initilization.

CoreTypes are what house the item code (behave like sword/bow/etc).

 

I have customModelLoader that I assume is working since the items are not purple and black cubes. (On a strange side note when I had another player connect via lan and my item model changed as they walked around me, it ranged from a series of chickens with flapping wings to a giant red circle on the ground)

 

With my admittedly limited understanding of the JSON model system I didn't believe it would be able to handle all the possible combinations of parts + the potential addition of parts from add-ons.

 

Either way I will need to send sync the available Draftables between client and server. I'm also pretty sure that Pre-generating them is out of the question, that would increase load time by a ridiculous amount.

 

As for file size my serialization of items with {5, 16, 32, 64, 256, 512, 1024, 2048, 4096}# of parts is 202KB total so an item with 4096 parts should be around 103KB minus some overhead because that was calculated from a file that was a serialization of a hashmap of draftables.

Whereas the 8192 ints (assuming the ByteBufUtils is not compressing it) would be 262KB.