It has nothing to do with randomness and more about how specific each step has to be. In most cases in the Minecraft world there are several paths between two points, but in your case there is exactly one correct next step.
Then if you think about the general problem of finding the best path when there is only one exact path that works you can see how the problem gets extremely hard. Think about actually trying to code it. What should the first step be? You might think that stepping in the direction of the target is best, but in general case that isn't guaranteed. Maybe you should go sideways a bit, or backwards. Your example only goes upwards but what about a case where you have to go down a bit and then back up? No matter how you code your pathfinding you can always build a case that is bad for that algorithm.
In the end the only way to guarantee finding a path is by brute force, meaning try every possible valid move. But that is still hard. In your example, if you spawned the entity right under the target block the correct block requires about 30 exact steps to get to. So you would need an algorithm that finds every combination of 30 steps to ensure you find the correct one. Sometimes it will be found early and sometimes at the end. With smarter algorithms you can make sure you don't include trying paths that cover ground you've already been on. But it still adds up. If you just do the first level of math, in each step you have 4 directions you can go. So the number of actual paths is 4 raised to the power of 30! That is 1.15x10^18 paths -- or 1.15 million million million! You can cut that down by assuming you don't go back to the space you just came from, but that is still 3^30.
Anyway, the number of possible paths is extremely large. The only way to really make it manageable is to limit the number of steps that the pathfinding can look for and hope that getting closer to the target is helpful overall (not always true, but often is).
The interesting thing is this shows how good humans are at pathfinding. We can look at a picture and almost instantly understand the path, probably because our brain can take advantage of some massively parallel processing that isn't available on a PC running Minecraft.
Again, there is a lot of research into this -- it is important for robotics and gaming and is interesting from a general math and computer science theory. But even the best algorithms are pretty limited.
EDIT: If you really want the entity to find the path using vanilla pathfinding I think it should be possible by adding more "waypoints" -- basically intermediate targets within the range of the pathfinding.