[SOLVED][1.7.2]Recommendations for packet payload content/format

[jabelar]

Okay, so I understand that you need packets to keep client in sync with the server (and vice versa).  And I understand how to send a packet in each direction.

 

But I'm interested in people's opinion on how they actually use the byte buffer payload of their packets.

 

My questions:

1)  Since most of my entities only have a couple custom fields, should I just create one packet that updates all of them (even if only one may have changed), or should I have a packet class for synching each field separately?

 

2) Do people tend to just come up with their own arbitrary byte stream?  Like just put your fields in whatever order you want and putInt(), putFloat() etc?  And then just read back in that order when received?

 

3) For Boolean fields, can you just use putInt() directly or do you have to convert the Boolean to int?  Or some other way?

 

4) I'm using FMLProxyPackets which extend Minecraft's Packet class.  That has a writeBlob() method which allows a length then a sub-stream of bytes.  Do people use those for strings?  Or what do people use for strings?  Just a loop of putChar()?

 

Basically I'm confident I can do the low level byte by byte stuff, but I'm wondering if I missing any methods that are more convenient.

[SanAndreaP]

1) I always only send a packet if something has changed, and then only the data I need. I tend to use "sub-IDs" where a Packet can set different values (e.g. I have a multi-page GUI, so I just use 1 Packet and send the page-ID and the data the page contains, which is always an int)

 

2) If you mean the order you put values into the byte stream, I do it logically (like if I send a packet with X/Y/Z coordinates, I put those (together, when there is more data than that) in this order), but you can pretty much do whatever you want.

 

3) You can use ByteBuf.writeBoolean

 

4) For Strings use ByteBufUtils.writeUTF8String(bytebuf, "a string"), respective ByteBufUtils.readUTF8String(bytes)

You can also write/read ItemStacks and NBTTags with the ByteBufUtils class

[TheGreyGhost]

Hi

 

1) I generally don't both economising on packet contents since unless the packet is biggish (more than say 1 K) or gets sent a lot, it's unlikely to make any noticeable difference.

2) Yes.  Typical code I use is (from 1.6.4)

 

  public Packet250CustomPayload getPacket250CustomPayload()
  {
    checkInvariants();
    Packet250CustomPayload retval = null;
    try {
      ByteArrayOutputStream bos = new ByteArrayOutputStream();
      DataOutputStream outputStream = new DataOutputStream(bos);
      outputStream.writeByte(Packet250Types.PACKET250_CLONE_TOOL_USE_ID.getPacketTypeID());
      outputStream.writeByte(commandToByte(command));
      outputStream.writeInt(toolID);
      outputStream.writeInt(sequenceNumber);
      outputStream.writeInt(actionToBeUndoneSequenceNumber);
      outputStream.writeInt(xpos);
      outputStream.writeInt(ypos);
      outputStream.writeInt(zpos);
      outputStream.writeByte(rotationCount);
      outputStream.writeBoolean(flipped);
      retval = new Packet250CustomPayload("speedytools",bos.toByteArray());
    } catch (IOException ioe) {
      ErrorLog.defaultLog().warning("Failed to getPacket250CustomPayload, due to exception " + ioe.toString());
      return null;
    }

    return retval;
  }

  /**
   * Creates a Packet250SpeedyToolUse from Packet250CustomPayload
   * @param sourcePacket250
   * @return the new packet for success, or null for failure
   */
  public static Packet250CloneToolUse createPacket250CloneToolUse(Packet250CustomPayload sourcePacket250)
  {
    try {
      DataInputStream inputStream = new DataInputStream(new ByteArrayInputStream(sourcePacket250.data));

      byte packetID = inputStream.readByte();
      if (packetID != Packet250Types.PACKET250_CLONE_TOOL_USE_ID.getPacketTypeID()) return null;

      byte commandValue = inputStream.readByte();
      Command command = byteToCommand(commandValue);
      if (command == null) return null;

      Packet250CloneToolUse newPacket = new Packet250CloneToolUse(command);
      newPacket.toolID = inputStream.readInt();
      newPacket.sequenceNumber = inputStream.readInt();
      newPacket.actionToBeUndoneSequenceNumber = inputStream.readInt();
      newPacket.xpos = inputStream.readInt();
      newPacket.ypos = inputStream.readInt();
      newPacket.zpos = inputStream.readInt();
      newPacket.rotationCount = inputStream.readByte();
      newPacket.flipped = inputStream.readBoolean();
      if (newPacket.checkInvariants()) return newPacket;
    } catch (IOException ioe) {
      ErrorLog.defaultLog().warning("Exception while reading Packet250SpeedyToolUse: " + ioe);
    }
    return null;
  }

 

 

 

3) DataOutputStream has a wide range of methods for writing data, including boolean.  You can even write objects if they are serialisable, eg a simple example for a BitSet

 

 

  /** serialise the VoxelSelection to a byte array
   * @return the serialised VoxelSelection, or null for failure
   */
  public ByteArrayOutputStream writeToBytes()
  {
    ByteArrayOutputStream bos = null;
    try {
      bos = new ByteArrayOutputStream();
      DataOutputStream outputStream = new DataOutputStream(bos);
      outputStream.writeByte(xsize);
      outputStream.writeInt(ysize);
      outputStream.writeInt(zsize);

      ObjectOutputStream objectOutputStream = new ObjectOutputStream(outputStream);
      objectOutputStream.writeObject(voxels);
      objectOutputStream.close();
    } catch (IOException ioe) {
      ErrorLog.defaultLog().warning("Exception while converting VoxelSelection toDataArray:" + ioe);
      bos = null;
    }
    return bos;
  }

  /** fill this VoxelSelection using the serialised VoxelSelection byte array
   * @param byteArrayInputStream the bytearray containing the serialised VoxelSelection
   * @return true for success, false for failure (leaves selection untouched)
   */
  public boolean readFromBytes(ByteArrayInputStream byteArrayInputStream) {
    try {
      DataInputStream inputStream = new DataInputStream(byteArrayInputStream);

      int newXsize = inputStream.readInt();
      int newYsize = inputStream.readInt();
      int newZsize = inputStream.readInt();
      if (newXsize < 1 || newXsize > MAX_X_SIZE || newYsize < 1 || newYsize > MAX_Y_SIZE || newZsize < 1 || newZsize > MAX_Z_SIZE) {
        return false;
      }

      ObjectInputStream objectInputStream = new ObjectInputStream(inputStream);
      Object newVoxels = objectInputStream.readObject();
      if (! (newVoxels instanceof BitSet)) return false;
      xsize = newXsize;
      ysize = newYsize;
      zsize = newZsize;
      voxels = (BitSet)newVoxels;
    } catch (ClassNotFoundException cnfe) {
      ErrorLog.defaultLog().warning("Exception while VoxelSelection.readFromDataArray: " + cnfe);
      return false;
    } catch (IOException ioe) {
      ErrorLog.defaultLog().warning("Exception while VoxelSelection.readFromDataArray: " + ioe);
      return false;
    }
    return true;
  }

 

 

4) I haven't used those before so I can't say. 

 

To be honest, I wouldn't worry too much about finding more convenient methods, the ones you already identified are probably fine and are easy to get right.

 

-TGG

 

[TheGreyGhost]

What TGG said is mostly right, but don't use Java serialization mechanics. Java Serialization is slow as heck, so if you use it to send objects over the network you'll a) use up much more bandwidth than you need do (it will send class-names, field-names, etc. as strings!) and b) definitely cause performance problems at some point.

Hi

 

As usual, I think the most-correct answer is "it depends".

You can cause a lot of bandwidth problems using Java serialisation if your class is full of stuff you don't actually need to send, or if you are sending packets multiple times a second.

But if your class contains only the data that needs sending, and you don't need to send it often, there's no problem.

The BitSet in my example uses an extra 76 bytes.  That's not significant unless you are sending many of these each second.

 

In many cases I think it's largely a matter of personal taste whether you initially implement the optimised hand-coding, or you just use the serialisation.  Java inbuilt serialisation is for sure less efficient.  It is also faster to code and reduces the chances of bugs when you refactor.

Personally I prefer to keep my code simple and easy to understand/refactor, then optimise it once I have some test data that shows me where I can get significant gains.

Others prefer to apply the obvious optimisations up-front.  The key point is to understand the tradeoffs.

 

-TGG

 

[jabelar]

Thanks, I respect all you guys as experts and will use some of what each of you said.

 

I will send packets only when there is a change, but I agree with TGG that economizing the packet size isn't probably worth it (especially in my case) -- a few dozen bytes isn't a big deal in modern "network" and there is overhead on the packets themselves.  So maybe I'll try the serialization.

 

Not sure how I missed the writeBoolean() method, but yeah I'll use that.

 

The ByteBufUtils is the kind of thing I was most looking for.  I figured someone must have gotten tired of using bytes for Minecraft related info.  The methods for NBT compounds are interesting, since it could let me merge my save/load code with the sync code.

 

TheGreyGhost, with serialization I understand how you would send the serialized object but not quite confident on how to read it such that it is actually applied to the entity.  So you can just set your object to equal the de-serialized object received in the packet?

[jabelar]

Okay after a marathon coding session, I've done my own grounds up netty packet handling and it seems to be working well, thanks to the tips to those above.

 

Just to describe my solution: I basically am using FMLProxyPacket class, using ByteBufOutputStream and ByteBufInputStream for their write and read methods.  I've subscribed to the ClientCustomPacket and ServerCustomPacket events.  I confirm that the channel is for my mod, that the side is correct, and then have methods in my custom packet class to handle each case.  I'm actually only using one packet class, but have packet type ID to switch to appropriate processing code.

 

I have another couple questions, but I'll start a new thread with the specific topics.  I consider this thread SOLVED.

 

For posterity to help others, here is some of the significant code I wrote.  Note that I was specifically trying to send variables from server to client to aid in entity render animations -- in this case I wanted my herd animal to "rear up" when attacked.

 

I register my networking channel with following method that I call in my common proxy:

 

public void registerNetworkingChannel()

{

WildAnimals.channel = NetworkRegistry.INSTANCE.newEventDrivenChannel(WildAnimals.networkChannelName);

    // when you want to send a packet elsewhere, use one of these methods (server or client):

//    WildAnimals.channel.sendToServer(FMLProxyPacket);

//    WildAnimals.channel.sendTo(FMLProxyPacket, EntityPlayerMP); for player-specific GUI interaction

//    WildAnimals.channel.sendToAll(FMLProxyPacket); for all player sync like entities

// and there are other sendToxxx methods to check out.

}

 

 

 

My server run configuration ServerPacketHandler class:

 

package wildanimals.networking;

 

import java.io.IOException;

 

import wildanimals.WildAnimals;

import wildanimals.network.entities.PacketWildAnimals;

import cpw.mods.fml.common.eventhandler.SubscribeEvent;

import cpw.mods.fml.common.network.FMLNetworkEvent.ServerCustomPacketEvent;

 

public class ServerPacketHandler

{

protected String channelName;

 

@SubscribeEvent

public void onServerPacket(ServerCustomPacketEvent event) throws IOException

{

channelName = event.packet.channel();

 

if (channelName == WildAnimals.networkChannelName)

{

PacketWildAnimals.processPacketOnServerSide(event.packet.payload(), event.packet.getTarget());

}

}

}

 

 

 

My "combined client" run configuration ClientPacketHandler class:

 

package wildanimals.networking;

 

import java.io.IOException;

 

import wildanimals.WildAnimals;

import wildanimals.network.entities.PacketWildAnimals;

import cpw.mods.fml.common.eventhandler.SubscribeEvent;

import cpw.mods.fml.common.network.FMLNetworkEvent.ClientCustomPacketEvent;

 

// Remember client run configuration includes server side too

public class ClientPacketHandler extends ServerPacketHandler

{

@SubscribeEvent

public void onClientPacket(ClientCustomPacketEvent event) throws IOException

{

channelName = event.packet.channel();

 

if (channelName == WildAnimals.networkChannelName)

{

PacketWildAnimals.processPacketOnClientSide(event.packet.payload(), event.packet.getTarget());

}

}

 

}

 

 

 

My registration of my server run config packet handler in the common proxy, plus a couple of override methods that seem to be necessary for the interface.  I call the registerServerPacketHandler() in my common proxy init() method:

 

protected void registerServerPacketHandler()

{

WildAnimals.channel.register(new ServerPacketHandler());

}

 

@Override

public Object getServerGuiElement(int ID, EntityPlayer player, World world,

int x, int y, int z) {

return null;

}

 

@Override

public Object getClientGuiElement(int ID, EntityPlayer player, World world,

int x, int y, int z) {

return null;

}

 

 

 

My registration of my combined client run configuration packet handler in the client proxy's init() method:

 

private void registerClientPacketHandler()

{

WildAnimals.channel.register(new ClientPacketHandler());

}

 

 

 

Then there is the class that actually makes packets to send, and decodes packets received.  I use one class for all packets, and have a packet type as first int to switch the processing and for now I'm processing entities, so next int is entityID and I use that to figure out which further fields to encode / decode:

 

package wildanimals.network.entities;

 

import io.netty.buffer.ByteBuf;

import io.netty.buffer.ByteBufInputStream;

import io.netty.buffer.ByteBufOutputStream;

import io.netty.buffer.Unpooled;

 

import java.io.IOException;

 

import net.minecraft.client.Minecraft;

import net.minecraft.entity.Entity;

import net.minecraft.world.World;

import wildanimals.WildAnimals;

import wildanimals.entities.bigcats.EntityBigCat;

import wildanimals.entities.herdanimals.EntityHerdAnimal;

import wildanimals.entities.serpents.EntitySerpent;

import cpw.mods.fml.common.network.internal.FMLProxyPacket;

import cpw.mods.fml.relauncher.Side;

 

// this class is intended to be sent from server to client to keep custom entities synced

public class PacketWildAnimals

{

// define IDs for custom packet types

public final static int packetTypeIDEntity = 1;

 

public PacketWildAnimals()

{

// don't need anything here

}

 

public static FMLProxyPacket createEntityPacket(Entity parEntity) throws IOException

{

ByteBufOutputStream bbos = new ByteBufOutputStream(Unpooled.buffer());

 

// create payload by writing to data stream

// first identity packet type

bbos.writeInt(packetTypeIDEntity);

 

// write entity instance id (not the class registry id!)

bbos.writeInt(parEntity.getEntityId());

 

// now write entity-specific custom fields

// process herd animals

if (parEntity instanceof EntityHerdAnimal)

{

EntityHerdAnimal entityHerdAnimal = (EntityHerdAnimal)parEntity;

bbos.writeFloat(entityHerdAnimal.getScaleFactor());

bbos.writeBoolean(entityHerdAnimal.isRearing());

}

// process serpents

else if (parEntity instanceof EntitySerpent)

{

EntitySerpent entitySerpent = (EntitySerpent)parEntity;

bbos.writeFloat(entitySerpent.getScaleFactor());

}

// process big cats

else if (parEntity instanceof EntityBigCat)

{

EntityBigCat entityBigCat = (EntityBigCat)parEntity;

bbos.writeFloat(entityBigCat.getScaleFactor());

}

 

// put payload into a packet

FMLProxyPacket thePacket = new FMLProxyPacket(bbos.buffer(), WildAnimals.networkChannelName);

 

// don't forget to close stream to avoid memory leak

bbos.close();

 

return thePacket;

}

 

public static void processPacketOnClientSide(ByteBuf parBB, Side parSide) throws IOException

{

if (parSide == Side.CLIENT)

{

World theWorld = Minecraft.getMinecraft().theWorld;

ByteBufInputStream bbis = new ByteBufInputStream(parBB);

 

// process data stream

// first read packet type

int packetTypeID = bbis.readInt();

 

switch (packetTypeID)

{

case packetTypeIDEntity:  // a packet sent from server to sync entity custom fields

{

// find entity instance

int entityID = bbis.readInt();

// DEBUG

System.out.println("Entity ID = "+entityID);

Entity foundEntity = getEntityByID(entityID, theWorld);

// DEBUG

System.out.println("Entity Class Name = "+foundEntity.getClass().getSimpleName());

 

// process based on type of entity class

// process herd animals

if (foundEntity instanceof EntityHerdAnimal)

{

EntityHerdAnimal foundEntityHerdAnimal = (EntityHerdAnimal)foundEntity;

// apply custom fields to entity instance

foundEntityHerdAnimal.setScaleFactor(bbis.readFloat());

foundEntityHerdAnimal.setRearing(bbis.readBoolean());

// DEBUG

System.out.println("Is rearing = "+foundEntityHerdAnimal.isRearing());

}

// process serpents

else if (foundEntity instanceof EntitySerpent)

{

EntitySerpent foundEntitySerpent = (EntitySerpent)foundEntity;

// apply custom fields to entity instance

foundEntitySerpent.setScaleFactor(bbis.readFloat());

}

// process big cats

else if (foundEntity instanceof EntityBigCat)

{

EntityBigCat foundEntityBigCat = (EntityBigCat)foundEntity;

// apply custom fields to entity instance

foundEntityBigCat.setScaleFactor(bbis.readFloat());

}

break;

}

}

 

// don't forget to close stream to avoid memory leak

bbis.close();

}

}

 

public static void processPacketOnServerSide(ByteBuf payload, Side parSide)

{

if (parSide == Side.SERVER)

{

// currently haven't defined any packets from client

 

}

}

 

// some helper functions

public static Entity getEntityByID(int entityID, World world)       

{       

for(Object o: world.getLoadedEntityList())               

{                       

if(((Entity)o).getEntityId() == entityID)                       

{                               

System.out.println("Found the entity");                               

return ((Entity)o);                       

}               

}               

return null;       

}

}

 

 

And with all the above set up, whenever I want to sync client to server I simply have to do:

 

 

When I want to send an sync packet (usually on one of my "setter" methods that change value of a custom field):

 

    // send sync packet to client, if on server side

    if (!this.worldObj.isRemote)

    {

        try

        {

    WildAnimals.channel.sendToAll(PacketWildAnimals.createEntityPacket(this));

    }

        catch (IOException e)

        {

    e.printStackTrace();

    }

    }

 

 

 

[coolAlias]

Now that you've done all that work, have you seen the wiki Netty Packet Handling code? It's quite nice and easy to use. I'm impressed you coded your own, though

[jabelar]

Now that you've done all that work, have you seen the wiki Netty Packet Handling code? It's quite nice and easy to use. I'm impressed you coded your own, though

 

I had looked at that one, but frankly it seemed overkill for my purpose.  At least right now I'm not doing complex GUIs, but rather just doing special animations for some custom entities, so I just need some state variables synced on the client to control the rendering.  So to simply transfer a Boolean across I didn't figure I needed pipelines, discriminators, and various handlers.

 

I was able to get by with a fairly compact set of hooks into FML (The FMLProxyPacket, the ClientCustomPacket event, the ServerCustomPacket event) and avoided needing all the other network handlers.

 

But yeah, after going through this I can see there would be a lot of different ways to approach it.