First of all: Thanks for responding that fast.
Server: It's a VPS, 16 GB RAM, 4 CPU Cores, 300 GB memory, 5 TB traffic, 100 MBit/s Uplink, 1x IPv4 & IPv6. The forge server itself has 15 GB RAM. The traffic has been used for under 1%. There is no lag (especially if I'm alone on the server).
Client: I can't use Github to upload my debug.log, it seems to be too big. I uploaded it to Google drive (https://drive.google.com/file/d/10M9_bon1mQrESKf1KETZ5YmhlEBFwK76/view?usp=sharing). I noticed some other things that might help: I seem to disconnect either entering or leaving the nether( haven't tested other dimensions.) and I have to connect 5 times until I can actually join. I and my friend get those disconnects, trying to join ( IDK if he gets disconnected with that Nether thing.). I don't know if this helps but if for example, my friend joins I get his join message and his skin/player appears, I can hit him, but he gets no damage, then he disconnects and he is justing seeing that dirt screen and then he gets that Disconnected / Timeout message.
If you need something else or if you have a way I can upload that debug log to Github, message me.
EDIT: Forgot to mention the TPS is usually between 18 and 20 and I have a ping between 10 and 40.