Hexagon Systems Journal

Build reversible migrations

2026-11-03T00:00:00Z

Infrastructure changes become safer when the replacement can be prepared and tested before the previous path is removed.

Separate preparation from activation

Build the new service independently. Inspect the host before installing anything, preserve unrelated workloads, and verify local dependencies. Activate one public dependency at a time: address assignment, DNS, control-plane registration, and finally user traffic.

This sequence keeps failures attributable. If several dependencies change simultaneously, a rollback becomes guesswork.

Define evidence and rollback

Write the success checks before the change. They should cover process state, listening sockets, firewall behavior, public DNS, certificates, management connectivity, and a real client request. Define the observation that will trigger rollback and the exact operation required.

Backups are useful only when their scope and restoration procedure are known. A small copy of the previous configuration beside a tested restore command is often more valuable than an unexamined full-system archive.

Record the active state

Documentation should describe object relationships, routing intent, public exposure, and verification results without storing credentials. Future operators need to understand why the system is shaped this way, not merely repeat the last command that happened to work.

Certificates are a lifecycle

2026-10-29T00:00:00Z

A valid certificate is not a finished task. It begins an operational lifecycle involving challenge reachability, scheduled renewal, safe reloads, expiry monitoring, and recovery from failed automation.

Validate the prerequisites

Before issuance, compare the public DNS answer with the intended endpoint. Confirm that the challenge path is reachable from outside and that an unrelated redirect or access rule does not intercept it. Automation should stop when these assumptions are false.

After issuance, inspect the subject alternative names and validity period from the public endpoint. A file on disk is not evidence that the active service loaded it.

Exercise renewal early

A renewal dry run demonstrates that credentials, challenge routing, and reload hooks still agree. Schedule it while the current certificate has ample lifetime remaining, not during the final hours before expiry.

Keep the previous valid material until the replacement is complete and the service has passed an external TLS check. A deployment should never replace working credentials with an incomplete result.

Monitor what visitors see

Expiry alerts should inspect the public service, not only a certificate directory. This catches stale processes, incorrect virtual hosts, and load balancers that continue to present an older certificate.

Routing policy that can be read

2026-10-20T00:00:00Z

Routing policy is executable documentation. Every rule should have a narrow match, a clear purpose, and an obvious place in the order of evaluation.

Put restrictive rules first

Private destinations and traffic that must never leave a trust boundary should be handled before broad regional or service rules. The final catch-all must be deliberate: operators should know whether unmatched traffic is sent directly, through a controlled exit, or blocked.

Names matter. Tags such as regional-direct, selected-service, and blocked-private communicate intent better than numbered labels whose meaning lives only in someone's memory.

Test branches independently

One successful request cannot validate a policy with several exits. Test regional domains, regional IP addresses, international destinations, blocked networks, and every special proxy branch separately. Confirm the observed egress where possible.

Health checks also need to use representative destinations. A probe can report that an outbound is alive while the service people actually need remains unreachable.

Prefer a small policy

Complex routing is sometimes necessary, but complexity should be earned by a measurable requirement. Remove dead outbounds, stale selectors, and duplicate conditions. A smaller policy produces fewer ambiguous failures and makes rollback safer.

Measure the whole path

2026-10-12T00:00:00Z

A user does not experience DNS, routing, transport encryption, and an application as separate systems. A request either completes or it does not. Monitoring should reflect that reality.

Begin with the public name

Start a check with the same hostname and protocol a visitor uses. Confirm answers from more than one resolver, inspect the selected address, negotiate TLS, and request a real application path. Each stage should record latency and the evidence needed to explain a failure.

A local process check remains valuable, but it proves only that a process exists. It does not prove that public DNS is correct, a firewall permits the request, the certificate matches, or the return path reaches the client.

Treat latency as a distribution

A single average hides the variance users feel as stalled pages and unstable sessions. Track percentiles over time and compare observations from several networks. When a route changes, the median may remain acceptable while the tail becomes dramatically worse.

Traceroute can offer clues, but intermediate routers may handle diagnostic packets differently from application traffic. Always finish the investigation using the actual hostname, port, and protocol.

Keep the check understandable

An end-to-end probe should be simple enough that an operator can reproduce it manually. Record the resolved address, TLS name, HTTP status, response time, and observed endpoint. Good evidence shortens incidents because each result eliminates a class of possible causes.