[1.12.2] Extended continuous streaming of packets results in network lag

[CosmoConsole]

I'm encountering a weird issue when attempting to stream packets with a Forge SimpleNetworkWrapper. The use case in question involves streaming audio to each individual player that has their "pocket radio" turned on, and sends a constant stream of packets (the audio quality being relatively low to decrease the network usage, yet at the same time add realism to a "pocket radio").

 

"Are you insane???" Yes, I am, but the system works well enough... be it for one issue I've encountered.

 

The system consists of a thread sending running every 200 ms, that sends audio data split over several packets (used to be a single large packet in my attempt to fix the issue I'm about to describe) and a handler on the client side to actually get the audio data and feed it to a SourceDataLine (which does the actual playback).

 

The weird issue I've noticed is that if you have the pocket radio on for long enough, it'll eventually start causing some network lag - entity movements start stuttering and there's a delay when placing/breaking blocks or trying to open containers. The lag usually starts after a few minutes and gets increasingly worse. Addressing it is sometimes as simple as turning off the pocket radio for a few seconds and turning it back on again, but it seems that relogging will also clear any lag that might've started to happen. My only guess is some sort of adaptive QoS or priority system, but I couldn't find any hint to this being the case (and there seems to be no way whatsoever to assign a priority to the packets: if such a system existed, I would set the audio packets to have a low priority). The lag is not caused by server slowdown: any players not using the radio will not be affected, but they will of course see players with radios eventually start to stutter.  

 

Packet file (PhotonicRadioPacket.java)

package email.com.gmail.cosmoconsole.forge.photoniccraft;

import email.com.gmail.cosmoconsole.forge.photoniccraft.client.PhotonicClientProxy;
import io.netty.buffer.ByteBuf;
import net.minecraft.client.Minecraft;
import net.minecraft.client.audio.SoundHandler;
import net.minecraft.client.audio.SoundManager;
import net.minecraftforge.fml.common.ObfuscationReflectionHelper;
import net.minecraftforge.fml.common.network.simpleimpl.IMessage;
import net.minecraftforge.fml.common.network.simpleimpl.IMessageHandler;
import net.minecraftforge.fml.common.network.simpleimpl.MessageContext;
import paulscode.sound.SoundSystem;

public class PhotonicRadioPacket implements IMessage {
	public static final int PACKET_SPLIT = 5;
	public static final int PACKET_SIZE = 2205 / PACKET_SPLIT;
	public byte[] audio;
	private int length;
	private int start;
	private int end;
	public double strength;
	public PhotonicRadioPacket() {
		this.strength = 0;
		this.length = 0;
		this.audio = new byte [0];
		this.start = this.end = 0;
	}
	public PhotonicRadioPacket(double strength, byte[] audio, int start, int end) {
		this.strength = strength;
		this.length = end - start;
		this.start = start;
		this.end = end;
		this.audio = audio;
	}
	@Override
	public void fromBytes(ByteBuf buf) {
		strength = buf.readDouble();
		length = buf.readShort();
		start = 0;
		end = length;
		this.audio = new byte[length];
		buf.readBytes(this.audio);
	}
	@Override
	public void toBytes(ByteBuf buf) {
		buf.writeDouble(strength);
		buf.writeShort(length);
		buf.writeBytes(audio, start, end - start);
	}
	public static class Handler implements IMessageHandler<PhotonicRadioPacket, IMessage> {
		@Override
		public IMessage onMessage(PhotonicRadioPacket message, MessageContext ctx) {
			if (ModPhotonicCraft.sourceDataLine == null) return null;
			double strength = message.strength;
			if (strength < 0) {
				PhotonicAPI.shouldNotBePlaying();
				PhotonicAPI.stopPlayingRadio();
				return null;
			}
			byte[] audio = message.audio;
			if (audio.length != PACKET_SIZE) {
				return null;
			}
			byte[] noise = new byte[PACKET_SIZE];
			byte[] finalaudio = new byte[PACKET_SIZE*2];
			
			float scv = 1.0F;
			try {
				scv = ((float)Minecraft.getMinecraft().gameSettings.getSoundLevel(ModPhotonicCraft.radioCategory));
			} catch (Exception ex) {}
            // extremely ugly and there's probably a better way, but works for now
			double volumeToPlay = ((SoundSystem)ObfuscationReflectionHelper.getPrivateValue(SoundManager.class, 
					(SoundManager)ObfuscationReflectionHelper.getPrivateValue(SoundHandler.class, Minecraft.getMinecraft().getSoundHandler(), PhotonicReflectionHelper.sndManager), 
					PhotonicReflectionHelper.sndSystem)).getMasterVolume()
					* scv;
			if (volumeToPlay <= 0) return null;
			PhotonicClientProxy.setVolume(volumeToPlay);
			volumeToPlay = 1.0;
			
			double istrength = 1 - strength;
			PhotonicAPI.silentNoiseGenerate(noise);
			for (int i = 0; i < PACKET_SIZE; i++) {
				if (strength > 0) {
					finalaudio[2*i] = (byte) scaleVolume(audio[i], noise[i], strength, volumeToPlay);
				} else {
					finalaudio[2*i] = (byte) scaleVolume((byte)0, noise[i], 0.0, volumeToPlay);
				}
				finalaudio[2*i+1] = (byte) (finalaudio[2*i] ^ 0x80);
			}
			long now = System.currentTimeMillis();
			if ((now - ModPhotonicCraft.lastReceive) > 1500L) {
				return null;
			}
			long start = System.currentTimeMillis();
			ModPhotonicCraft.sourceDataLine.write(finalaudio, 0, 2*PACKET_SIZE);
			return null;
		}
	}
	private static double scaleVolume(byte d, byte n, double s, double v) {
		return (((double)unsign(qualityRuin(d,v)) - 128) * s + ((double)unsign(n) - 128) * (1 - s)) * v + 128;
	}
	private static byte qualityRuin(byte val, double q) {
		if (q >= 0.9) return val;
		if (q >= 0.75) { return (byte) ((val & 0xFE) | ((val & 0x80) >> 7)); }
		if (q >= 0.6) { return (byte) ((val & 0xFC) | ((val & 0xC0) >> 6)); }
		if (q >= 0.45) { return (byte) ((val & 0xF8) | ((val & 0xE0) >> 5)); }
		if (q >= 0.3) { return (byte) ((val & 0xF0) | ((val & 0xF0) >> 4)); }
		if (q >= 0.2) { return (byte) ((val & 0xE0) | ((val & 0xF0) >> 3)); }
		if (q >= 0.1) { return (byte) ((val & 0xC0) | ((val & 0xF0) >> 2)); }
		return 0;
	}
	public static double unsign(byte b) {
		if (b < 0)
			return b + 256;
		return b;
	}
}

 

The thread that takes care of sending data via the network handler (ModPhotonicCraft.network)

        serverTimer.scheduleAtFixedRate(new TimerTask() {
			@Override
			public void run() {
				if (PhotonicAPI.midSend.get()) return;
				PhotonicAPI.midSend.set(true);
				try {
					MinecraftServer srvr = FMLCommonHandler.instance().getMinecraftServerInstance();
					radioHandled.clear();
					recvData.clear();
					recvPower.clear();
					recvLoc.clear();
					PhotonicAPI.cannotModify.set(true);
					sendLock.lock();
					try {
						dataSentLastTick.clear();
						for (int channel: sendCache.keySet()) {
							recvData.put(channel, sendCache.get(channel));
							dataSentLastTick.put(channel, sendCache.get(channel));
							recvPower.put(channel, sendPower.get(channel));
							recvLoc.put(channel, sendLoc.get(channel));
						}
						sendCache.clear();
						sendPower.clear();
						sendLoc.clear();
					} finally {
						sendLock.unlock();
					}
					PhotonicAPI.cannotModify.set(false);
					long now = System.currentTimeMillis();
					byte[] b = new byte[2205];
					Arrays.fill(b, (byte)0);
					ArrayList<UUID> removeKeys = new ArrayList<UUID>();
					byte[] d = new byte[2205];
					int m = 0;
					for (UUID u: radioPlayers.keySet()) {
						if (m >= ModPhotonicCraft.maximumRadio) {
							break;
						}
						if ((now - radioPlayers.get(u)) >= 1000L) {
							removeKeys.add(u);
						} else {
							int ch = radioChannels.get(u);
							double ampl = 0.0;
							EntityPlayer plr = getPlayerFromUUID(srvr, u);
							if (plr instanceof EntityPlayerMP) {
								EntityPlayerMP p = (EntityPlayerMP)plr;
								if (recvData.containsKey(ch)) {
									d = recvData.get(ch);
									ampl = calculateAmplitude(recvLoc.get(ch).distanceSq(PhotonicLocation.fromPlayer(p)), recvPower.get(ch));
								} else
									Arrays.fill(d, (byte)0);
								byte[] ba = ampl <= 0.0 ? b : PhotonicAPI.cloneByteArray(d);
								for (int i = 0; i < PhotonicRadioPacket.PACKET_SPLIT; ++i) {
									ModPhotonicCraft.network.sendTo(new PhotonicRadioPacket(ampl, ba, i * PhotonicRadioPacket.PACKET_SIZE, (i + 1) * PhotonicRadioPacket.PACKET_SIZE), p);
								}
							} else if (plr != null) {
								if (recvData.containsKey(ch)) {
									d = recvData.get(ch);
									ampl = calculateAmplitude(recvLoc.get(ch).distanceSq(PhotonicLocation.fromPlayer(plr)), recvPower.get(ch));
								} else
									Arrays.fill(d, (byte)0);
								byte[] ba = ampl <= 0.0 ? b : PhotonicAPI.cloneByteArray(d);
								for (int i = 0; i < PhotonicRadioPacket.PACKET_SPLIT; ++i) {
									ModPhotonicCraft.network.sendToAll(new PhotonicRadioPacket(ampl, ba, i * PhotonicRadioPacket.PACKET_SIZE, (i + 1) * PhotonicRadioPacket.PACKET_SIZE));
								}
							}
						}
					}
					for (UUID u: removeKeys)
						radioPlayers.remove(u);
				} catch (Exception ex) {
					PhotonicAPI.debugMessage("Radio server fatal error");
					ex.printStackTrace();
				} finally {
					PhotonicAPI.flipState.set(!PhotonicAPI.flipState.get());
					PhotonicAPI.cannotModify.set(false);
					PhotonicAPI.midSend.set(false);
				}
			}
		}, 10L, 200L);

 

There are several possible solutions to this, but I'd have to be certain of the underlying cause to try anything. Are there any tips, or perhaps, is there a system that is better designed for direct streaming of data (maybe even in Netty itself)?

 

update 1: "In addition, I've also noticed that there is immense connection lag should a player join with the radio already turned on, which also seems to point at an issue that can be fixed with something like QoS or packet priority."

update 2: To clarify, the radio stream remains completely smooth when everything else starts to lag (even in the case of what happens in update 1).

Edited January 20, 2018 by CosmoConsole
add more info

[jabelar]

Maybe it is memory leak on the client. Implementing networking often leads to memory leaks if you don't manage the buffers or keep creating buffers without releasing them. Have you checked the memory usage over time?

Edited January 20, 2018 by jabelar

[CosmoConsole]

1 hour ago, jabelar said:

Maybe it is memory leak on the client. Implementing networking often leads to memory leaks if you don't manage the buffers or keep creating buffers without releasing them. Have you checked the memory usage over time?

No, I haven't still properly tested for memory leaks myself (and if I will, I'm planning to use something more informative than a Task Manager memory usage meter), but that's because this doesn't seem to be a memory leak for three reasons:

1. Any of the other testers have said the client wasn't either using its full heap nor increasing its heap size any faster than usual.

2. I can't think of anything that would leak memory this slow - if it is the SourceDataLine, it'd certainly cause the memory usage to increase a lot quicker.
3. It's unlikely that the only side effect of a memory leak would be connection lag - I have not seen or heard of FPS drops or freezes caused by this issue.

 

In addition, I've also noticed that there is immense connection lag should a player join with the radio already turned on, which also seems to point at an issue that can be fixed with something like QoS or packet priority.

Edited January 20, 2018 by CosmoConsole
grammar

[CosmoConsole]

Nobody else will probably try what I tried, but I'm posting my current solution anyway. I don't like it that much and don't want to consider a permanent one for that reason, but it seems to do its job for now. I did three changes in total:

 

1. Disable the radio on all players when they join - them being on causes massive connection lag (despite the radio stream being smooth). 

2. Restart the SourceDataLine if the remaining number of samples in it goes over a certain limit, in case samples arrive just slightly too quickly (even though the buffer size is restricted). This is done by adding the number of samples I'm writing and subtracting the difference between getLongFramePosition() values between such writes.

3. If the ping goes over a certain limit (currently set at 150 ms), instruct the server to skip the next radio packet every few seconds. (Since ping seems to update pretty slowly with a new value every 10-15 seconds if not less often, I've also added a check that the ping must not be equal to the ping during the previous skip). 

 

With these changes (which add a few chances for small skips in the audio due to 2 and 3) I haven't noticed any issues like originally described, or at least to the point they would be noticeable anymore. I'm still open to more permanent solutions though.

Edited January 23, 2018 by CosmoConsole