When Things Go Wrong

External facing Linux server running BIND.

Zone updated after a long time but Linux permissions resulted in zone file being locked and non-functional.

Zone included MX records that were then unavailable.

Caused Email outage.

Linux admin team decided that they didn't want DNS service maintenance under their role so moved it to Infoblox.

NIOS Hidden Primary.

Third Party DNS provider zone transferred to publicly host zones.

Third Party acquired by another company.

Other company migrates systems.

Everything works.

Other company deletes migration systems.

Turns out, a few customers had their TSIG keys tied up in the migration system.

TSIG keys no longer existed.

Zone transfers failed.

Alerts for zone transfer failure… failed.

After a week, the secondary servers dropped the zones that they could not longer update.

Result - outage of critical website until customer updated public DNS to “unhide” their NIOS boxes.