Oh gosh looks like I tunnel visioned onto the model and layer themselves and it never clicked on me that it was the render event all along. Made a set to store all players that I've added the layer to, now it's smooth like butter. Thanks for the help dude, and also for the grain interaction thing. Scrap that, someone pointed out to me that the renderer is shared for all entities, so yeah, only once at startup.
I'm trying to make some custom elytra-like items and have them render when worn. Got the rendering to work somewhat but there are many problems. When starting to fly in midair the wing opening animation is smooth like vanilla one, but when they are closing when landing they just snap to closed position and leaves a kind of "trail" effect i.e. like if there were a dozen of them closing, same thing happens when crotching to make the wings spread a little. The enchantment glow effect is kinda broken, looks like the glow just keeps getting stronger until it gets to full white. There is kinda of a fps drop when the model is getting rendered. Definitely there is something wrong in there or something that I forgot to put in, but I was unable to find a solution after digging thru docs and google search results and source codes of other similar mods so I'm kinda at wit's end now, hope if you know what is going on here. The classes used for the rendering are in the following package: https://github.com/LocusAzzurro/icaruswings_mod/tree/main/src/main/java/org/mineplugin/locusazzurro/icaruswings/render