[1.19] Screen point (x,y) to world coordinates (x,y,z)

[than00ber1]

 Hi there

I am attempting to make a gui where the player will be able to click whatever block is in front of them.

I've tried a few approaches and none of them have worked. (Bare with me as I have absolutely zero knowledge in 3D rendering.)

 

I've looked online for a bit and found this on stackoverflow using OpenGL.

Here is my 1:1 attempt in Java on my container screen:

    @Override
    public void render(@NotNull PoseStack pose, int mouseX, int mouseY, float partialTick) {
        // https://stackoverflow.com/a/7702895/13844398
        // STEP 1: get mouse coords. (mouseX, mouseY)
        
        // STEP 2
        Matrix4f modelviewMatrix = RenderSystem.getModelViewMatrix();
        Matrix4f projectionMatrix = RenderSystem.getProjectionMatrix();

        // STEP 3
        modelviewMatrix.multiply(projectionMatrix);

        // STEP 4
        modelviewMatrix.invert();

        // STEP 5
        float x = Mth.clampedMap(mouseX, 0.0F, width, -1.0F, 1.0F);
        float y = Mth.clampedMap(mouseY, 0.0F, height, -1.0F, 1.0F);
        float z = 1.0F; // the depth value ?
        float w = 1.0F;

        Vector4f vector4f = new Vector4f(x, y, z, w);

        // STEP 6
        vector4f.transform(modelviewMatrix);

        // STEP 7
        vector4f.mul(1 / vector4f.w());

        // presumably results in a direction vector ??
        Vec3 vec3 = new Vec3(vector4f.x(), vector4f.y(), vector4f.z());

        getMinecraft().font.draw(pose, vec3.toString(), 10.0F, 10.0F, 14737632);
    }

My screen prints something that resembles this: (239.75..., 126.00..., -2003000.125). Which simply looks like my cursor's coordinates (with an extra`z` value) on screen an nothing more.

 

Am I heading towards the right direction? Is there an easier way?

Edited July 22, 2022 by than00ber1
added missing link to post

[warjort]

I have zero knowledge about this stuff either. 🙂 I do know you don't want to modify those internal matrices like you are.

You should at least clone them. But I don't think they are what you want anyway.

 

Take what is below with a pinch of salt!

I would guess the matrix values you want are those used in RenderTarget._blitToScreen()? 

But even then this isn't going to help you. You don't know the depth of the block. The examples in the link assume you know the value or use z=1 (a point on the far plane).

You probably really need to do some kind of depth test on the buffer at the clicked "pixel". I have no idea how to do that.

 

Your better bet would to ask on the discord channel of one of the graphics optimisation mods. They will have a lot more knowledge about this stuff.