Within World.getClosestVunerablePlayer there is an invisibility check (if mobs can't see you they won't attack) which also calls EntityPlayer.getArmourVisibility:

    /**
     * When searching for vulnerable players, if a player is invisible, the return value of this is the chance of seeing
     * them anyway.
     */
    public float getArmorVisibility()
    {
        int i = 0;
        ItemStack[] aitemstack = this.inventory.armorInventory;
        int j = aitemstack.length;

        for (int k = 0; k < j; ++k)
        {
            ItemStack itemstack = aitemstack[k];

            if (itemstack != null)
            {
                ++i;
            }
        }

        return (float)i / (float)this.inventory.armorInventory.length;
    }

I think there should be some check for the armour being invisible (mmmPowersuits comes to mind), or semi-visible.

 

    /**
     * When searching for vulnerable players, if a player is invisible, the return value of this is the chance of seeing
     * them anyway.
     */
    public float getArmorVisibility()
    {
        float f = 0;
        ItemStack[] aitemstack = this.inventory.armorInventory;
        int j = aitemstack.length;

        for (int k = 0; k < j; ++k)
        {
            ItemStack itemstack = aitemstack[k];

            if (itemstack != null)
            {
                 //new function that returns the opacity of the ItemStack. default implementation would return 1.0F (i.e. fully opaque).
                f += itemstack.getItem().getVisibility(itemstack);
            }
        }

        return f / (float)this.inventory.armorInventory.length;
    }

I would have suggested adding this check to ItemArmor, but due to the fact that any item can be armour, it would have to be a general method.

Alternatively it could be part of a new interface IArmor, and the isValidArmor method would return if the item was an instanceof IArmor

public interface IArmor
{
public ItemArmor.ArmorMaterial getArmorMaterial(); //various other armor-related methods could be moved across
public float getArmorOpacity(ItemStack stack);
}

and the check would become:

Item item = itemstack.getItem();
f += item instanceof IArmor ? ((IArmor)item).getArmorOpacity(itemstack) : 1.0F;

I have not checked this recently, but it would be sensible to also check the held item being in the hand (I imagine many mobs could see a floating sword or battale axe pretty well). An empty hand would add no visibility, whereas a full opaque sword would or hammer would.

I did not even think of that, I was just basing it purely off the vanilla code, but you have a good point. maybe it should just be a new method in Item, that way it could be called for any "visible" Item.