Directly accessing the JVM is not a good idea. This will break the game when there are other mods using ASM. Moreover, looking up fields and methods during runtime via reflection (yes, even through JVM) is much, much slower than writing your code in Java, as the latter is compiled to bytecode and have direct access.
Therefore, you should not be writing mods in languages other than Java, as it is unnecessarily expensive.
As for your question, the easiest way would be via GLM. Simply apply the camera's transform onto the 3D point you want to transform. You can look up the specific calling on GLM's documentation.
The transformation is relative to the transform of the camera. If I understood your intention correctly, you need to offset the position of your 3D points by the position of the player.
Also:
1. Please don't use raw pointers.
2. There is no need to cast the ptrs to a float array and copy it into a double array. gluProject takes in a void pointer; you can simply throw ptr into it without any explicit conversions.
3. Using GLdouble to store rendering data like model and projection is slow and unnecessary. Float should suffice, as the precision difference is not enough to be displayed on a pixelated viewport.