Generally everyone will need to have every mod, there are some server-only mods and some client-only mods, provided they are properly written they will auto-disable if they are on the wrong side, if they aren't then they'll crash and the crash report should make it obvious.
Custom maps should work fine as long as you have at least the mods from the save.