Please always make backups of your worlds regardless if you plan to open them with mods or not. It's also probably good practice to separate vanilla and modded instances of the game. Regardless, my friend posted this in another server I am in that might help you out:
1) First thing's first, you'll need to install a program called NBTExplorer https://github.com/jaquadro/NBTExplorer/releases
Direct download link to the latest version: https://github.com/jaquadro/NBTExplorer/releases/download/v2.8.0-win/NBTExplorer-2.8.0.zip
If you downloaded the .zip file, all you need to do is extract the contents into their own folder. In Windows, you should just be able to right click the .zip folder and click the option that says "Extract All...". Follow the prompts and you should be in business.
2) Make a copy of the world you want to modify. Save it somewhere to your desktop or any other location where it's out of the way.
3) Open NBTExplorer. You should see the saves in Minecraft's default directory. If you use custom directories (manually or through a third party launcher) you'll need to go to File -> Open Folder and navigate to where your saves are stored. Get to the world save you want to modify, and open it up.
4) In the world save, there's a file called level.dat. Using NBTExplorer, open that file. (You can also set NBTExplorer as the default program to open .dat files through Windows, allowing you to simply go to the file normally and open it)
5) In the level.dat go to Data -> WorldGenSettings -> dimensions. You should see something similar to the image I posted with this.
6) Once you're there, click one of the modded dimensions you want to remove, and hit the Delete Tag button (with the red X). Do this for each dimension. Once all the dimensions you wanted to remove are gone, hit the save button.
7) If all went well, you should now be able to join your world! If not, that's why we made a backup in step 2.