Forge 1.14.4 Server Does not allow clients to join with client side mods not loaded onto Server.

I am running a forge server for Minecraft 1.14.4 and I have a fairly small mod pack that does not require a lot of recourses. Normally when I have a forge server I only have to put mods on it that are required to have on the server, but this time when I try to join with things like optifine, Enhanced Visuals, Damage Indicators, Waila and Mini Map mods that you’d think are client side, my server does not allow the player to join unless I add those mods to the server itself. It says many things along with Mismatched mods list. I put all of the client mods I can on the server which works and it’s fine with me to do that but the problem is when certain client mods don’t work with the server so I can’t use them. And I can’t just have it running in the client because for some reason the server doesn’t ignore them. Somebody please help, server logs won’t help with this bc whenever it disconnects the player it says in the console Player left due to disconnect. Please help.