Irritation: When your colocation provider shuts off port 22 for the whole network, temporarily (of undisclosed time), while you're in the middle of working.
Serious Irritation: Knowing you could have installed your firewall/VPN box last week.
Denouement: Realizing you aren't paying for it, and you have no room to really say a thing.
