looks like you're applying yaw and pitch swapped during partial ticks. That is, you're applying a large yaw change to the pitch angle and a small pitch change to the yaw angle.
As for how the matrix works, you'll want to read up on projection matricies:
https://jsantell.com/3d-projection/