How can I make a custom sword unbreakable?

If can't how could I set the durability of the sword?

this.setMaxDamage(-1);

It should work.

another option is to make your own tool material which essentially dose the same thing except it also allows you to set the damage, enchantability and if you also want to use it for tools mining level and mining speed.

To create a custom tool material use the following.

public static ToolMaterial MY_MATERIAL= EnumHelper.addToolMaterial("MY_MATERIAL", 4, -1, 12.0F, 11.0F, 35);
                                                              //("Name", Mining level, durability, mining speed, attack damage, enchantability)