[1.16][solved] 3d to 2d conversion

[Ravenwolf397]

I need to take a 3d point in the world (eg. "The cow is at (12, 63, -19)") and project it to 2d screen coordinates (eg. "The cow is 243 pixels from the left of the screen, and 87 pixels from the top of the screen").

 

I found this: https://forums.minecraftforge.net/topic/75508-how-do-i-convert-a-3d-point-to-screen-coordinates/ but it is non-functional.  I tried figuring out the rotations on my own, but that also didn't work.  Both produce incorrect projections, that don't seem to correlate with the correct point in any noticable way.

 

My math (`MiscUtil.projectToPlayerView`):

public static Vector2f projectToPlayerView(Vector3d target) {
	double fov = 70;

	ActiveRenderInfo ari = Minecraft.getMinecraft().gameRenderer.getActiveRenderInfo();
	Vector3d camera_pos = ari.getProjectedView();
	double x = camera_pos.x, y = camera_pos.y, z = camera_pos.z;
	double yaw = Math.toRadians(ari.getYaw()), pitch = Math.toRadians(ari.getPitch());

	x = target.x - x;
	y = target.y - y;
	z = target.z - z;

	// my rotations (doesn't work)
	// rotate so camera is on yz-plane
	// x = (x * Math.cos(yaw) - z * Math.sin(yaw));
	// z = (x * Math.sin(yaw) + z * Math.cos(yaw));
	// rotate so camera is on z axis
	// y = (y * Math.cos(pitch) - z * Math.sin(pitch));
	// z = (y * Math.sin(pitch) + z * Math.cos(pitch));

	// rotations given in linked post
	z = z * Math.cos(yaw) * Math.cos(pitch);
	x = x * Math.sin(yaw);
	y = y * Math.sin(pitch);

	// apply fov
	x = x / z * fov;
	y = y / z * fov;

	return new Vector2f((float) x, (float) y);
}

 

Edited July 18, 2020 by Ravenwolf397
solved

[Dzuchun]

5 hours ago, Ravenwolf397 said:

I need to take a 3d point in the world (eg. "The cow is at (12, 63, -19)") and project it to 2d screen coordinates (eg. "The cow is 243 pixels from the left of the screen, and 87 pixels from the top of the screen").

 

I found this: https://forums.minecraftforge.net/topic/75508-how-do-i-convert-a-3d-point-to-screen-coordinates/ but it is non-functional.  I tried figuring out the rotations on my own, but that also didn't work.  Both produce incorrect projections, that don't seem to correlate with the correct point in any noticable way.

 

My math (`MiscUtil.projectToPlayerView`):

public static Vector2f projectToPlayerView(Vector3d target) {
	double fov = 70;

	ActiveRenderInfo ari = Minecraft.getMinecraft().gameRenderer.getActiveRenderInfo();
	Vector3d camera_pos = ari.getProjectedView();
	double x = camera_pos.x, y = camera_pos.y, z = camera_pos.z;
	double yaw = Math.toRadians(ari.getYaw()), pitch = Math.toRadians(ari.getPitch());

	x = target.x - x;
	y = target.y - y;
	z = target.z - z;

	// my rotations (doesn't work)
	// rotate so camera is on yz-plane
	// x = (x * Math.cos(yaw) - z * Math.sin(yaw));
	// z = (x * Math.sin(yaw) + z * Math.cos(yaw));
	// rotate so camera is on z axis
	// y = (y * Math.cos(pitch) - z * Math.sin(pitch));
	// z = (y * Math.sin(pitch) + z * Math.cos(pitch));

	// rotations given in linked post
	z = z * Math.cos(yaw) * Math.cos(pitch);
	x = x * Math.sin(yaw);
	y = y * Math.sin(pitch);

	// apply fov
	x = x / z * fov;
	y = y / z * fov;

	return new Vector2f((float) x, (float) y);
}

 

Why would you even need that? You need to draw something near a cow? You may use RenderLivingEvent (Post subclass, if you need to draw over the cow).

 

About your own math - check if you're rotating your vector in correct directions. Actually, you need to turn -yaw and -pitch (I see, you did it ok), because initially your vector points target, but you need it to point your looking direction. Also your yaw and pitch should be actually a difference between camera yaw and entity relative yaw, I can't see you calculating it.

 

At last, you may use Quaternions to make you code more readable - they are at package net.minecraft.util.math.vector.Quaternion, and Vector3f has useful methods, such as rotation(float) gives you a Quternion to rotate around axis collinear to vector and transform(Quternion) to transform current vector with specified Quaternion.

Happy math-ing!

Edited July 17, 2020 by Dzuchun

[Ravenwolf397]

Thank you!

 

Quaternions were the answer; I just apply the ActiveRenderInfo::getRotation() as a transformation to the relative position of the point and the camera, then scale by some factor `scale_factor` (180 works well at the default fov of 70).

 

Through a bit of finagling, I even managed to compensate for the effects of view bobbing.  

 

However, I now face a new but related conundrum: changing the fov away from 70 makes the calculated point become scaled incorrectly.  Specifically, if the fov is high then it is too far from the screen center, and if the fov is low then it is too close to the screen center.  

 

The problem seems pretty clearly to do with my scaling factor of 180 above.  I tried multiplying `scale_factor` by (70 / fov), but that didn't change it enough; the result was still too close to center at low fov, and too far from center at high fov.  It was closer to what it should be, but still not correct.

 

Since the function appeared to be trigonometric, I attempted multiplying `scale_factor` by `-1 / Math.tan(fov / 2)` (essentially treating it as focal length), but that produced very odd behavior with no discernable pattern.

 

How does Minecraft's fov affect this situation?  I would have though that `scaling_factor` was just the focal length, but the multiplier above involving tangent should have worked in that case, so I'm not sure.

 

Updated code:

    public static Vector2f projectToPlayerView(Vector3d target, float partialTicks) {
        /* The (centered) location on the screen of the given 3d point in the world. */
        float scale_factor = 180;

        ActiveRenderInfo ari = getActiveRenderInfo();
        Vector3d camera_pos = ari.getProjectedView();
        Quaternion camera_rotation_conj = ari.getRotation().copy();
        camera_rotation_conj.conjugate();

        Vector3f result3f = new Vector3f((float) (camera_pos.x - target.x),
                (float) (camera_pos.y - target.y),
                (float) (camera_pos.z - target.z));
        result3f.transform(camera_rotation_conj);

        // compensate for view bobbing (if active)
        // this isn't relevant to the question, I just put it here in case anyone wants it
        // the following code adapted from GameRenderer::applyBobbing (to invert it)
        Minecraft mc = getMinecraft();
        if (mc.gameSettings.viewBobbing) {
            Entity renderViewEntity = mc.getRenderViewEntity();
            if (renderViewEntity instanceof PlayerEntity) {
                PlayerEntity playerentity = (PlayerEntity) renderViewEntity;
                float distwalked_modified = playerentity.distanceWalkedModified;

                float f = distwalked_modified - playerentity.prevDistanceWalkedModified;
                float f1 = -(distwalked_modified + f * partialTicks);
                float f2 = MathHelper.lerp(partialTicks, playerentity.prevCameraYaw, playerentity.cameraYaw);
                Quaternion q2 = new Quaternion(Vector3f.XP, Math.abs(MathHelper.cos(f1 * (float) Math.PI - 0.2F) * f2) * 5.0F, true);
                q2.conjugate();
                result3f.transform(q2);

                Quaternion q1 = new Quaternion(Vector3f.ZP, MathHelper.sin(f1 * (float) Math.PI) * f2 * 3.0F, true);
                q1.conjugate();
                result3f.transform(q1);

                Vector3f bob_translation = new Vector3f((MathHelper.sin(f1 * (float) Math.PI) * f2 * 0.5F), (-Math.abs(MathHelper.cos(f1 * (float) Math.PI) * f2)), 0.0f);
                bob_translation.setY(-bob_translation.getY());  // this is weird but hey, if it works
                result3f.add(bob_translation);
            }
        }

        // handle alteration due to fov
        double fov = mc.gameSettings.fov;
        // neither of these are correct:
        // scale_factor *= (70 / fov);
        // scale_factor *= -1 / Math.tan(fov / 2);

        // todo include fov modifier from sprinting, speed, etc.

        Vector2f result = new Vector2f(-result3f.getX(), result3f.getY());
        result = new Vector2f(scale_factor * result.x / result3f.getZ(), scale_factor * result.y / result3f.getZ());

        return result;
    }

 

Edited July 17, 2020 by Ravenwolf397

[Dzuchun]

Actually, I have no certain idea how FoV works in Minecraft. But I if we assume, that it works like an optical camera, I may help you.

Here you can see how world is projected to the screen (or matrix in camera).

So, to negate FoV deformations you should (using spherical trigonometry) calculate angle distance to center of the screen and take a tangent of it - now you have pure coordinate - distance to target in radius of some sphere. To find a pure coordinate of screen border, you may take tan(FoV/2). Then using screen resolution calculate distance in pixels on screen between center and target. Then again using spherical trigonometry calculate angle between Center-Target and, for example, horizon line, understand that it is unchanged after projection, and get your result vector by this angle and that distance.

But please,

don't do that!

I'm sure, more forge-like way exists to do what you want.

For example, this:

11 hours ago, Dzuchun said:

You may use RenderLivingEvent (Post subclass, if you need to draw over the cow).

or net.minecraftforge.client.event.RenderWorldLastEvent.

Edited July 17, 2020 by Dzuchun

[Draco18s]

7 minutes ago, Dzuchun said:

Actually, I have no certain idea how FoV works in Minecraft. But I if we assume, that it works like an optical camera, I may help you.

Here you can see how world is projected to the screen (or matrix in camera).

So, to negate FoV deformations you should (using spherical trigonometry) calculate angle distance to center of the screen and take a tangent of it - now you have pure coordinate - distance to target in radius of some sphere. To find a pure coordinate of screen border, you may take tan(FoV/2). Then using screen resolution calculate distance in pixels on screen between center and target. Then again using spherical trigonometry calculate angle between Center-Target and, for example, horizon line, understand that it is unchanged after projection, and get your result vector by this angle and that distance.

 

 

 

This actually helps me on a completely unrelated project. The (builtin) WorldToScreen method I have doesn't (seem to) generate useful values when the position isn't inside the view frustum (so something that was off-screen-leftish would show a direction of off-screen-upish).

[Dzuchun]

2 minutes ago, Draco18s said:

 

This actually helps me on a completely unrelated project. The (builtin) WorldToScreen method I have doesn't (seem to) generate useful values when the position isn't inside the view frustum (so something that was off-screen-leftish would show a direction of off-screen-upish).

Please, double-check everything writen in my post (because of my signachure).

[Draco18s]

57 minutes ago, Dzuchun said:

Please, double-check everything writen in my post (because of my signachure).

👍

Sure, I was just saying that because the builtin was being bad, that gave me something to work with to try and write the function myself. A starting point as it were.

 

And by bad, I mean this object is in the lower left and the coordinate popping out is in the upper right.

:

But I move it over here:

Suddenly its correct again.

 

Could be related to absolute distance from the camera (I can get values of about -20,000,000 before it goes positive), but when I've tried mathing the world coordinates down towards the camera (this being very easy, the stuff in the center is at (0,0,0), so dividing by a fixed value should work) I still get nonsense.

 

Success! Calculating the angle myself (ok, I used a builtin) based on the object's position and the camera's position (with the same height to avoid some oddities with things being directly ahead/behind) works perfectly.

Edited July 17, 2020 by Draco18s

[Ravenwolf397]

Okay, I got it working 100%.  Compensates for screen resolution, view bobbing, fov changes, and even dynamic fov like from sprinting and speed potions.

 

public static FinalTriple<Float, Float, Boolean> projectToPlayerView(double target_x, double target_y, double target_z, float partialTicks) {
        /* The (centered) location on the screen of the given 3d point in the world.
         * Result is <dist right of center screen, dist up from center screen, is target in front of viewing plane> */
        ActiveRenderInfo ari = getActiveRenderInfo();
        Vector3d camera_pos = ari.getProjectedView();
        Quaternion camera_rotation_conj = ari.getRotation().copy();
        camera_rotation_conj.conjugate();

        Vector3f result3f = new Vector3f((float) (camera_pos.x - target_x),
                (float) (camera_pos.y - target_y),
                (float) (camera_pos.z - target_z));
        result3f.transform(camera_rotation_conj);

        // ----- compensate for view bobbing (if active) -----
        // the following code adapted from GameRenderer::applyBobbing (to invert it)
        Minecraft mc = getMinecraft();
        if (mc.gameSettings.viewBobbing) {
            Entity renderViewEntity = mc.getRenderViewEntity();
            if (renderViewEntity instanceof PlayerEntity) {
                PlayerEntity playerentity = (PlayerEntity) renderViewEntity;
                float distwalked_modified = playerentity.distanceWalkedModified;

                float f = distwalked_modified - playerentity.prevDistanceWalkedModified;
                float f1 = -(distwalked_modified + f * partialTicks);
                float f2 = MathHelper.lerp(partialTicks, playerentity.prevCameraYaw, playerentity.cameraYaw);
                Quaternion q2 = new Quaternion(Vector3f.XP, Math.abs(MathHelper.cos(f1 * (float) Math.PI - 0.2F) * f2) * 5.0F, true);
                q2.conjugate();
                result3f.transform(q2);

                Quaternion q1 = new Quaternion(Vector3f.ZP, MathHelper.sin(f1 * (float) Math.PI) * f2 * 3.0F, true);
                q1.conjugate();
                result3f.transform(q1);

                Vector3f bob_translation = new Vector3f((MathHelper.sin(f1 * (float) Math.PI) * f2 * 0.5F), (-Math.abs(MathHelper.cos(f1 * (float) Math.PI) * f2)), 0.0f);
                bob_translation.setY(-bob_translation.getY());  // this is weird but hey, if it works
                result3f.add(bob_translation);
            }
        }

        // ----- adjust for fov -----
        Method m;
        float fov;
        GameRenderer gameRenderer = mc.gameRenderer;
        try {
            m = gameRenderer.getClass().getDeclaredMethod("getFOVModifier", ActiveRenderInfo.class, float.class, boolean.class);
        } catch (NoSuchMethodException e) {
            LOGGER.error(e);
            throw new Error("getFOVModifier method not present on GameRenderer class; cannot project to player screen.", e);
        }
        m.setAccessible(true);
        try {
            fov = ((Double) m.invoke(gameRenderer, ari, partialTicks, true)).floatValue();
        } catch (IllegalAccessException | InvocationTargetException e) {
            LOGGER.error(e);
            throw new Error("getFOVModifier invocation caused error.", e);
        }

        float half_height = (float) mc.getMainWindow().getScaledHeight() / 2;
        float scale_factor = half_height / (result3f.getZ() * (float) Math.tan(Math.toRadians(fov / 2)));
        return new FinalTriple<>(-result3f.getX() * scale_factor, result3f.getY() * scale_factor, result3f.getZ() < 0);
    }

 

There are couple of self-explanatory methods (getActiveRenderInfo, getMinecraft) and a class (FinalTriple) that should be obvious to implement yourself.