What I am wondering is how to do something like 1.9 Luck mechanic making it more common to find rarer items. So I have a custom fishing rod that can find rare fish, but what I want to know is if I can do something like if say a player has a potion effect active, it makes the rarer fish more common.

I suggest looking at Forge's patch to

EntityFishHook#getFishingResult

to see how it uses the Lure and Luck of the Sea enchantment levels to determine which loot category to select from.

I know about the different loot categories, but what I mean is, is there a way to say have a "Treasure" category, and when you have a higher "luck" then you have a higher chance of getting the loot with the lower weight.

Within a given loot table, there is not any way of which I am aware to affect what is chosen - thus the use of different loot tables when fishing.

 

It might be possible, however, to change the weights of the items in the table before selecting an item, but unless you made a copy of the table to do so (possibly slow) or make sure to change it back (possibly affects other loot using the same table or even potentially risks ConcurrentModificationException), then it's probably not advisable.