LapisLazuli

I downloaded forge and all the mods for this server, but I can't join the server? But when I played a singleplayer server with all the mods, I was able to load in?? Either gives a normal Failed to Connect to the Server or a io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: Its so odd to me that I can load it in on singleplayer but not on multiplayer? If anyone has a fix let me know, thanks!