Multi-CDN failover sent users to a non-HTTPS origin, causing mixed content errors and the failover health check that enforced TLS – WP Newsify

As the Internet becomes more complex and performance expectations rise, using multiple Content Delivery Networks (CDNs) has become a common strategy to increase availability, improve load times, and guard against outages. However, if multi-CDN installations are not implemented carefully, new issues can arise that unexpectedly impact the user experience and compromise security. One such incident involved a failover scenario in a multi-CDN configuration where users were directed to an HTTP origin despite the primary site being HTTPS only. This led to widespread errors with mixed content and failing assets on the client side.

TL; DR

In a multi-CDN configuration, when a health check-triggered failover switched traffic to a secondary CDN, users were redirected to a non-HTTPS origin (HTTP only). This caused browsers to reject unsafe content, triggering mixed content warnings and breaking critical site functionality. Surprisingly, the failover health check criteria only validated TLS from the CDN to the origin, and not the end user’s HTTPS access. Addressing this issue required an overhaul of origin health monitoring, stricter TLS enforcement, and content validation rules.

The rise of multi-CDN architectures

Deploying one multi-CDN architecture has become a best practice in today’s global internet landscape. Organizations use two or more CDNs to improve:

• Availability: Failover mechanisms kick in when a CDN provider experiences an outage.
• Performance: Traffic is routed to the CDN with the lowest latency based on geographic proximity and current load.
• Resistance: Protection against DDoS attacks or region-specific errors.

This kind of approach works well in theory, but its complexity leaves room for misconfiguration. If one of your CDNs is misaligned (especially in the way it communicates with the source or serves secure content), you can unknowingly expose your users to errors and even security risks.

The incident: when failover meets HTTP

During a routine failover initiated due to high latency detected on CDN1, a global e-commerce platform began routing traffic through CDN2. However, without thoroughly testing how CDN2 retrieved resources from the origin, something crucial was overlooked: CDN2 served assets through a non-HTTPS origin.

This wouldn’t have been a problem for standalone HTTP sites, but in this case the platform’s primary domain enforced HTTPS on all pages using a strict Content Security Policy (CSP). As soon as CDN2’s HTTP-based resources were served in an HTTPS environment, browsers started throwing mixed content errors and blocking essential content such as JavaScript files, CSS stylesheets, and fonts.

The result? Users were unable to interact with the platform: buttons were unresponsive, dropdowns did not work, and payment pages could not load.

Understanding mixed content errors

Mixed content occurs when a secure website (loaded over HTTPS) loads assets (such as scripts, images, or videos) over an insecure HTTP connection. Most modern browsers cancel such requests to protect users, leading to glitches in the way pages display or function.

There are two types of mixed content:

1. Active mixed content: Contains scripts, iframes, or other code that can interact with the document. These are completely blocked.
2. Passive mixed content: Contains media elements such as images or videos. Browsers can load these, but often display warnings.

In this case, it was heavy with active mixed content: everything from React bundles to authentication scripts failed to load, essentially turning a robust ecommerce platform into a static HTML tombstone.

The Health Check that misses the mark

Most CDNs use health checks to determine if an origin is healthy and can serve traffic. These health checks typically include HTTP requests at regular intervals to ensure an expected response is received. But in this scenario, an important nuance was overlooked: the failover was triggered by latency statistics from CDN1, and once routed to CDN2, CDN2’s health checks confirmed that the origin was “healthy”. But how?

It turns out that the health check validated the origin by accessing a TLS-terminated endpoint, but only between the CDN and the origin server. It did not validate how the origin responded to user-facing URLs and whether HTTPS was configured correctly for content accessed by real end users.

The main misconfiguration issues included:

• No authentication if the origin served identical HTTPS content as the primary CDN.
• The health check did not check for content mismatches or HTTP response headers such as Strict-Transport-Security or Content-Security-Policy.
• No TLS validation on the host headers used by the fallback CDN.

This monitoring led to a situation where the origin passed the health check but served HTTP content that was incompatible with the site’s security requirements.

Detect and diagnose the problem

The problem did not manifest itself immediately. Initially, CDN2 gave the green light: the origin connections were successful, the responses were timely, and there were no internal CDN alarms. However, within minutes the error reports started to increase:

• JavaScript CDNs failed to load core logic.
• Fonts and stylesheets were blocked due to insecure sourcing.
• Customer support tickets poured in, complaining that the checkout page was “broken.”

Teams quickly surveyed the browser developer tools and noticed a deluge Mixed content: The page was loaded over HTTPS, but requested an insecure script mistakes. The logs, rich with 403 and 404 responses for asset files, indicated that HTTP origins were being forcibly entered into HTTPS pages.

Mitigation and the applied solutions

Solving the problem required multiple tactical and strategic solutions.

Tactical solutions:

• Manually disabled CDN2 routing and sent traffic back to CDN1 once latency returned to normal.
• Temporarily updated the DNS failover weights to prevent automatic failbacks.
• Pushed hotfix patches to change base URLs for critical JavaScript to enforce HTTPS paths.

Strategic changes:

• Introduced TLS validation in all health check layers – including CDN to origin and end-to-end authentication using emulated browser requests via headless Chrome settings.
• Enforced SSL redirection at the origin server level so that all HTTP connections are robustly upgraded to HTTPS.
• Comprehensive content security policy to restrict asset loading to domains that only use HTTPS.
• Tested all CDN provider origin paths to ensure consistency in configuration and behavior regardless of access point.

What can we learn from this?

This type of failover fiasco highlights the dangerous assumptions often made in distributed environments:

• Those health checks confirm the quality of the user experience, but that is not the case.
• That origin has the same exact configuration for all access points – maybe not.
• That HTTP can be upgraded silently without side effects, not when CSPs and modern browsers enforce strict rules.

Takeaway: Every component in a failover chain should be treated as live production and tested accordingly, including security configurations, protocols, and content delivery behavior. CDN redundancy is valuable, but only if all components meet the same expectations.

Final thoughts

Failover systems are intended to improve stability, not compromise it. In this story, a seemingly intelligent, automated system created chaos because origin validation failed to take HTTPS and user security into account. By refining health check criteria and enforcing stricter protocols, companies can avoid this pitfall and deliver the reliable, secure experience users expect.

The battle for high availability doesn’t end with adding more nodes; it ends by ensuring that every node, path, and byte respects your application’s security standards, especially when hostile conditions arise.

Latest messages from Editorial Staff (see all)