13.1 Summary

After the project was marked complete, the live site (https://cicd-demo.qossim005.online) went down unexpectedly following an unrelated infrastructure change on a separate project sharing the same parent domain (qossim005.online).

13.2 Root Cause

The parent domain’s nameservers were intentionally updated at the registrar (Namecheap) to point to a new, Terraform-managed Route 53 hosted zone, as part of work on a separate project. This is a legitimate and expected change for that project - however, it silently broke DNS resolution for this project’s subdomain, because the NS delegation record for cicd-demo (originally created in the old hosted zone during Day 18-20) only existed in the old, now-bypassed zone. The new zone had no record telling it to hand off authority for cicd-demo to this project’s hosted zone, so requests for cicd-demo.qossim005.online resolved to nothing.

Note: Key mechanism: delegation is a one-way pointer stored as an NS record in the parent zone. Changing which zone is authoritative for the parent domain does not automatically carry over delegations that were only recorded in the old zone - they must be recreated in whatever zone is newly authoritative.

13.3 Diagnosis

13.4 Resolution

13.5 Key Learning

Subdomain delegation via NS records is resilient to changes within either zone individually, but not to a change in which zone is authoritative for the parent domain - that specific change requires re-adding the delegation record in the new parent zone. This is an easy failure mode when a domain is shared across multiple unrelated projects/accounts, and is worth checking any time a parent domain’s nameservers are modified for any reason.

This also reinforced the value of the monitoring set up in Day 21-22: while DNS-level outages aren’t caught by the CloudFront 5xxErrorRate alarm (the request never reaches CloudFront at all), the incident was caught quickly through direct manual verification of the live URL - a good argument for adding an external uptime check (e.g. a simple periodic HTTP check from outside AWS) as a future improvement.

13.6 Incident Summary