[SOLVED... KIND OF]Stencil buffer cuts everything

[Lucius Q. User]

EDIT: I switched to using scissor test.

I am trying to set up stencil buffer to cut everything that gets drawn outside of a specific rectangle in the gui. Instead, everything that is drawn after the stencil buffer is set up gets cut;

Here is the stencil set up code:

public static void setUpStencil(int left, int top, int right, int bottom) {
	if (left < right) {
		int i = left;
		left = right;
		right = i;
	}

	if (top < bottom) {
		int j = top;
		top = bottom;
		bottom = j;
	}
	Tessellator tessellator = Tessellator.getInstance();
	VertexBuffer vertexbuffer = tessellator.getBuffer();
	GL11.glEnable(GL11.GL_STENCIL_TEST);
	GL11.glStencilFunc(GL11.GL_ALWAYS, 1, 0xFF);
	GL11.glStencilOp(GL11.GL_KEEP, GL11.GL_KEEP, GL11.GL_REPLACE);
	GL11.glStencilMask(0xFF);
	GL11.glDepthMask(false);
	GL11.glClear(GL11.GL_STENCIL_BUFFER_BIT);
	vertexbuffer.begin(GL11.GL_QUADS, DefaultVertexFormats.POSITION);
	vertexbuffer.pos(left, bottom, 0.0D).endVertex();
	vertexbuffer.pos(right, bottom, 0.0D).endVertex();
	vertexbuffer.pos(right, top, 0.0D).endVertex();
	vertexbuffer.pos(left, top, 0.0D).endVertex();
	tessellator.draw();
	GL11.glStencilFunc(GL11.GL_EQUAL, 1, 0xFF); 
	GL11.glStencilMask(0x00);
	GL11.glDepthMask(true);
}

Here is the actual drawing code:

drawDefaultBackground();
setUpStencil(20, 20, width - 20, height - 20);
drawRect(0, 0, 50, 65); //Some drawRect calls here

 

Edited July 1, 2017 by Lucius Q. User

[Draco18s]

You would have to reset the stencil buffer.

[Lucius Q. User]

Sorry, what? I have just set it up and it cuts everything drawn while it is active.

[Elix_x]

Make sure that you enabled (created) stencil buffer. By default, MC does not create one.

Alternatively, because you're in 2D, use scissors.

Edited July 1, 2017 by Elix_x
Added link to stencil buffer creation with Forge

[Lucius Q. User]

I did enable it. The problem is not that it is not cutting anything, but rather that it cuts too much.

 

EDIT:

The problem is as follows:

drawRect(...)// <- gets drawn

setUpStencil(...)

drawRect(...) // <- gets cut entirely (should get cut only partially)

disableStencil()

drawRect(...) // <- gets drawn

Edited July 1, 2017 by Lucius Q. User

[Elix_x]

When stencils don't work for me right away, i usually enable the color write and draw the stencil'ed region with colors, then i go into debug mode and play with it until i get it to work.