Take screenshot on client and send to server

[The Box]

How do I take a screenshot on the client when an item is right clicked and then send that image over to the server in a packet and use it there?

Edited April 17, 2019 by The Box

[DavidM]

What are you going to do with the image?

Keep in mind that the client can always lie and send a fake image to the server.

[Cadiboo]

This is continued from discord. @The Box is trying to make a multiplayer-compatible camera mod.

Quote

I'm basically trying to make a camera mod. I'm stuck where I'm getting an error stating OpenGL is missing. Here is the function having the problem.

```java
  @Override
  public ActionResult<ItemStack> onItemRightClick(World worldIn, EntityPlayer playerIn, EnumHand handIn)
  {
  if (!worldIn.isRemote)
  {
  ItemStack stack = new ItemStack(ModItems.PICTURE);

  NBTTagCompound nbt;
  if (stack.hasTagCompound())
  {
  nbt = stack.getTagCompound();
  } else
  {
  nbt = new NBTTagCompound();
  }

  BufferedImage screenshot = new ScreenShotHelper().createScreenshot(Main.mc.displayWidth,
  Main.mc.displayHeight, Main.mc.getFramebuffer());
  byte[] bytes = null;
  try
  {
  ByteArrayOutputStream baos = new ByteArrayOutputStream();
  boolean foundWriter;
  foundWriter = ImageIO.write(screenshot, "jpg", baos);
  assert foundWriter;
  bytes = baos.toByteArray();
  } catch (IOException e)
  {
  e.printStackTrace();
  }

  nbt.setByteArray("Picture", bytes);

  stack.setTagCompound(nbt);
  }
  
  return super.onItemRightClick(worldIn, playerIn, handIn);
  }
```
This is where an item is supposed to be given with the pictures stored in the nbt. But on the line taking the screenshot with the helper I get this error:

java.util.concurrent.ExecutionException: java.lang.RuntimeException: No OpenGL context found in the current thread.

I said that this was because the code was being run on the server thread, and that I couldn’t think of a way to do what they wanted, so they should ask on the forums. The easiest solution would to just run everything on the client, but this will cause desyncs even in singleplayer.

Edited April 17, 2019 by Cadiboo

[V0idWa1k3r]

21 minutes ago, Cadiboo said:

The easiest solution would to just run everything on the client, but this will cause desyncs even in singleplayer.

Client thread isn't nececcairly the same as the render thread, and all rendering-related stuff must be done on the rendering thread unless you are using vulkan. 

What they need to do is enqueue the action on the client, aka have a static field changed to true on the onItemRightClick method, then in any kind of render event(RenderTickEvent, RenderWorldLastEvent, etc) check for that field being true and if it is do the gl related things and create an image there. Then yeah, a packet to send it to the server is needed, I *think* that if the packet is too big it will be split into multiple smaller packets, but I am not sure, you might just crash right away because of the packet size limit, in that case you would need to compress the image before sending it. And unfortunately clients will be able to abuse this packet and send extremely long byte arrays to the server regardless, and I don't think much can be done about it.

 

24 minutes ago, Cadiboo said:

foundWriter = ImageIO.write(screenshot, "jpg", baos);
  assert foundWriter;

You don't need this assertion. This is not the usecase for the statement.

 

25 minutes ago, Cadiboo said:

but this will cause desyncs even in singleplayer

I am not sure what do you mean here?

[Cadiboo]

1 hour ago, V0idWa1k3r said:

I am not sure what do you mean here?

I was thinking of only updating the data on the logical client which obviously wouldn’t work well