I have literally done this a few hours ago. Here's what to do.
There's a bit of linear algebra involved, so make sure to look that up. Check out 3Blue1Brown's videos. Especially the ones on 3d. They're a 10-minute watch away.
After that, you have two steps to take into consideration:
1. Rotation, which is a simple linear transform (this should be represented by a 3x3 matrix).
2. Displacement, which is an "offset" so-to-speak which you must apply to your vectors, to make up displaced quads. This is necessary because rotating quads using this method will rotate them around the 0 axis of the block (be it in the direction of x, y, or z. It depends on what you need)
Here's a pic of how I messed up the displacement, so you can understand why it's necessary:
The transparent-blueish spots are where the block is, and as you can see the quads are displaced incorrectly. This was however easily fixed once I went over the maths one more time.
I don't think the mojang Vector3d has a transform function, you'll need to write that yourself. But here's my take on it, so it's easier for you:
// The CustomVector3D class extends mojang's Vector3d, so the class fields x, y, z are available
public CustomVector3D linearTransform(double[][] mat) {
if (mat.length != 3
|| mat[0].length != 3
|| mat[1].length != 3
|| mat[2].length != 3) {
throw new IllegalArgumentException("Transformation matrix needs to be 3x3x3 in size.");
}
double xx = mat[0][0] * this.x + mat[0][1] * this.y + mat[0][2] * this.z;
double yy = mat[1][0] * this.x + mat[1][1] * this.y + mat[1][2] * this.z;
double zz = mat[2][0] * this.x + mat[2][1] * this.y + mat[2][2] * this.z;
return new CustomVector3D(xx, yy, zz);
}
There are some rotation methods, but I haven't taken a good look at them because I feel more comfortable (at the moment) with doing my own maths.
If you have any questions, please let me know.