How would I detect and hide enclosed space from a 3d map and only render open space?

[Kirdow]

I know this isn't really a topic directed at forge or minecraft, but since it's a forge mod I figured this is the right place.

Having a game called Cube World finally out to the public, me and a friend of mine figured we would make the map/minimap a thing in Minecraft, that is the 3d map you can rotate, zoom and pan around in a 3d view rather than top down 2d.

I got the rendering of this sorted out, but there's one slight thing that might be major that we want sorted out before initial release.

 

Before we go into more details I want to mention that the mapping is done by generating a mesh when we find the chunk, together with a checksum to check for future changes. Firstly chunk is copied to a custom chunk format, then on a separate thread, generate a mesh out of this data, and lastly call the main thread to do any opengl calls necessary. Worth mentioning this rendering uses the vanilla map colours, with each axis having its own tint to make some depth, and the mesh itself is generated using greedy meshing, that is scanning left to right, top to bottom for rectangles with similar look, and creating 2 triangles from that rectangle, which result in much fewer vertices for the same result/outcome.

 

But back to the problem, the task is to not render enclosed space, would it be caves, or even a bunker or a secret base in a mountain. As you may know, a 3d map you can view in actual 3d would cause more problems than a normal 2d top down maps in the sense of revealing stuff, like where ores are, or where someones base yet to find is located. On well over 50% of servers I'm been on, using map mods requires you to turn off caves in the map settings, as it could fall into the use of an X-Ray mod/hack.
 

Because of this, we want to get some sort of algorithm to implement a type of system that would render what would be seen from someone flying above surface, this would include the stuff below trees to the most part and the overhangs on mountains (as much as we can do that is, nothing in programming is 100% perfect), but not caves below the surface, and if the cave is connected to the surface, it would render it, but cut off at some distance into the cave. This sounds good and easy on paper but in practice it's not that too easy if you've never done similar stuff before.
 

At the current state of the code, we are currently looking for specific blocks like stone, dirt, grass etc, and when that is found, we increment a counter (how much depend on the type of block) and when an upper bound is reached, stop render the tiles (I want to point out this is done starting at y 255 down to 0, and does this for each xz coordinate per chunk).

 

What I had in mind, was to follow the 8,8 coordinate in each chunk (or the average highest median coordinate), down until it reaches surface level. Then, from that point, if any emptiness is found nearby or below, use something like A* to find the distance and deviation from the straight line between those points, render stuff with little deviation, and ignore/blacken stuff with much deviation. The distance variable used in A* would also play a huge role and would also have priority over other rules in use if it would get too long. This is the best I could think of myself, but I'm not fully sure how this would pay out in practice, thus I come here to ask if there's anything I could do in any other way, to give a good result, while still not going too complex.

 

Note: I'm not asking for direct code, just what algorithms I should use, in what way and stuff like that, relative to the ideas I already have, and what is currently in use.