2019 — From 95% to 99% Availability in Production

What Happened That Year

In 2019, our first-generation proxy product went live. HTTP and SOCKS5 services opened to enterprise users. The product evolved from "it works" to "people use it."

Users voted with their wallets. They would not pay for features alone; they paid because it worked reliably. And reliable meant stable.

That year, we formally committed to an availability metric. The gap between 95% (internal testing) and 99% (production target) took us the entire year.

Product Launch Background and Preparation

2019 was a pivotal year for us. The previous two years had been spent on technical validation and architecture building, but there was still a layer between the product and users -- users accessed through customized integrations rather than a standardized product interface.

Before the official launch, we spent nearly three months on final preparations. The core goal was ensuring that user growth would not cause stability degradation after launch.

Stress testing. We built a stress testing environment simulating real user behavior, covering the entire workflow from registration, activation, and configuration to daily usage. The stress tests exposed several issues that had not been discovered before: database connection pool performance bottlenecks when concurrency exceeded 200, API request timeouts under high concurrency, and monitoring system write performance lagging under high traffic.

Emergency response plans. We wrote detailed emergency response plans for every possible failure scenario. Each plan included failure characteristics, possible causes, handling steps, responsible personnel, and expected recovery time. These plans proved effective in numerous subsequent incidents.

Gradual rollout. Instead of launching the product to all users at once, we first invited a group of long-term test users to use the new system. During this gradual rollout, we discovered and fixed multiple issues before they could affect the full user base.

One typical issue uncovered during rollout: test users and production users operate in different network environments. Test users were mostly on good network conditions, while production users included those in complex environments -- enterprise intranets, mobile networks, and overseas connections. These special network environments exposed protocol compatibility issues we had not previously covered.

Another important finding was that users understood the product workflow differently than we expected. The activation process seemed simple to us internally, but over 30% of rollout users encountered confusion at some step. We optimized the process based on feedback, and the final version achieved over 95% activation success rate.

Most of the issues we anticipated before launch turned out to be well-founded.

Unexpected Discoveries After Launch

In the first month after launch, we encountered problems that had not appeared in the test environment at all.

Users used the product differently than we expected. When designing the product, we assumed users primarily did web data collection, making HTTP proxy the main use case. But after launch, we found many users using proxies for API calls, ad verification, account management, and other non-web-scraping scenarios. These scenarios had different stability requirements -- API calls were more sensitive to failure rates, and account management demanded more consistent IP assignments.

Traffic patterns varied dramatically between users. Some users concentrated their usage during business hours, others ran 24/7 operations, and still others ran batch tasks only during specific periods. These different traffic patterns imposed completely different pressure profiles on the system, requiring us to adjust resource allocation strategies.

Users understood availability differently than we defined it. We defined 99% availability as the system being operational 99% of the time. But users understood it as "99 out of every 100 of my requests succeed." The gap between these two definitions led to complaints from several early users.

The Real Gap Between 95% and 99% Availability

95% and 99% look like a difference of only 4 percentage points, but the real gap is much larger. A simple calculation: assume your business sends 100,000 requests per day.

95% availability means 5,000 failures per day. 99% means 1,000 failures per day. From 95% to 99%, failures drop from 5,000 to 1,000. But the user experience difference goes beyond quantity -- it is qualitative: 5,000 failures means business processes are frequently interrupted and operations staff must constantly intervene. 1,000 failures means most business processes can complete automatically, with retry mechanisms handling the majority. Going from 95% to 99% is effectively a leap from "someone needs to watch it" to "it can run unattended."

From an engineering perspective, this leap required systemic improvements across three areas:

Resource redundancy: from enough to more than enough. At the 95% level, resource planning only needed to meet current demand. But when an upstream resource failed, there was no spare capacity to handle shifted traffic. At the 99% level, every resource node maintained at least 30% spare capacity.

Fault detection: from waiting for user reports to automatic discovery. Earlier monitoring only checked whether the service was online. We later added more dimensions: abnormal connection establishment time increases, sudden failure rate spikes during certain periods, and resource pool response time degradation. Detecting these sub-health states allowed many issues to be resolved before they escalated into incidents.

Auto-failover: from manual handling to self-healing systems. At the 95% level, failover required manual operator intervention, typically taking 5-15 minutes per switch. At the 99% level, automatic failover mechanisms could complete fault identification and traffic migration within 30 seconds.

Standardized Fault Handling Process

In 2019, we established a standardized fault handling process that became the foundation of the entire service system:

Fault detection. Multi-dimensional monitoring automatically detects anomalies. If a node's failure rate exceeds a threshold, response time degrades by more than 50%, or consecutive probes fail, the system generates an alert.

Fault confirmation. After an alert triggers, the on-duty person confirms whether the fault is real. After filtering out false positives, the actual fault handling process begins.

Fault localization. Determine the root cause. Is it the proxy node itself, an upstream resource issue, or a network link problem? Different root causes require completely different handling approaches.

Fault handling. Execute the predetermined plan based on fault type. Node failures trigger automatic switching, upstream failures switch to backup resources, and network failures adjust routing strategies.

Fault review. After each fault, the responsible team writes a fault report covering time of occurrence, impact scope, root cause analysis, handling process, and improvement measures.

The Importance of User Education

In 2019, we also realized a critical issue: users had widely varying understanding of proxy stability. Some thought proxies should be 100% stable like electricity or water, while others thought "occasional disconnection is normal."

We began adding content about "what constitutes a reasonable availability expectation" to the product site and documentation. Not all users need 99.9% availability, but every user needs to know what availability they are buying and whether it is sufficient for their needs.

How to Judge Proxy Stability (Part 3): How Availability Is Measured

In 2019, operational experience revealed another important phenomenon: availability varies significantly across regions. A proxy node in Beijing might achieve 99.5% availability, but the same node in Urumqi might only deliver 95%. The causes include ISP interconnection issues, regional network infrastructure differences, and local network quality. Based on these findings, we optimized node distribution strategies to ensure users in different regions received a relatively consistent experience.

Availability vs. Price

In 2019 operations, we noticed a pattern worth discussing: there is a clear correlation between availability and price. Achieving 99% availability requires significantly more resources than 95%. Resource redundancy means maintaining more nodes, automatic failover requires more complex monitoring and scheduling systems, and fast fault response requires 24/7 staffing. Yet many users only compare prices when choosing proxy services, without comparing availability. A cheap service with 95% availability and a slightly more expensive one with 99% availability can have vastly different impacts on actual business costs -- the losses from business interruptions caused by proxy failures often far exceed the proxy service price difference.

Team Growth

In 2019, as the product launched, the team also grew. Operations personnel brought experience running large-scale systems, product managers brought user-centric requirements analysis, and test engineers brought more systematic quality assurance methods. This professional specialization allowed us to handle improvements across multiple fronts simultaneously: one group focused on core system stability optimization, another on monitoring and alert system improvements, and a third on user support and issue tracking. The results became evident in the second half of 2019 -- faster fault handling and deeper post-incident reviews.

One Piece of Advice

Many providers publish availability numbers measured in lab environments under ideal network conditions. But real-world usage scenarios vary enormously. You need to ask three questions:

• Is the test frequency once per minute or once per hour? Higher frequency means more reliable data.
• Is the test node in a single data center or multiple geographic locations? Single-point testing only reflects local conditions.
• Is the measurement based on a single connection failure or consecutive failures before counting an outage? Different methodologies produce vastly different numbers.

The same 99% can feel completely different depending on how it is measured.

Product Data Over the Years

Availability changes after the 2019 product launch:

Operational Experience After Launch

Our 2019 operations taught us several valuable lessons for running a proxy service:

Do not set monitoring alert thresholds too sensitively. Initially we set very strict rules -- alert on any single connection failure. Alert frequency was so high that the team developed alert fatigue. We later adjusted the rules to alert only after consecutive failures or when the failure rate exceeded a certain proportion within a time window.

Do not switch back immediately after recovery. Early on, we had a habit of switching traffic back as soon as the primary node recovered. But we found that some failures were intermittent -- switching back caused the issue to recur, leading to repeated switching. We later added an observation period before switching back.

Notify users promptly. In the early days, when a fault occurred, we focused entirely on fixing it and forgot to notify users. Some users, noticing service anomalies, assumed we had disappeared. We later established a fault notification mechanism -- even if the fix was not complete, we let users know "we are aware and working on it."

About Request-Level Availability

Beyond traditional time-based availability, in 2019 we began promoting the concept of request-level availability. Request availability is calculated as successful requests divided by total requests multiplied by 100%. The advantage of this metric is that it directly reflects the user's actual experience. If the system runs 24/7 but 5 out of every 100 requests fail, the user's perception is that the proxy is frequently problematic.

Fault Classification and Priority

In 2019, we also established a fault classification system defining three severity levels:

P0 fault: Core service unavailable, affecting a large number of users. For example, all proxy nodes go down simultaneously, or user login and APIs are completely unavailable. These require response within 10 minutes and recovery within 30 minutes.

P1 fault: Partial node failure, affecting some users. For example, a resource pool fails or proxy nodes in a region go offline. These require response within 30 minutes and recovery within 2 hours.

P2 fault: Non-core functionality issues, minimal impact. For example, an anomaly in the control panel or an error in a non-critical API. These are handled on the next business day.

This classification system helped us allocate operational effort appropriately. Previously, every fault was treated as equally urgent, and the team had to handle both P0 and P2 issues simultaneously, diluting focus. With the classification, P0 and P1 were handled first, P2 was scheduled orderly, and overall efficiency improved dramatically.

Internal Metrics vs. User Perception

In 2019, we recognized a critical problem: internal system monitoring of availability and user-perceived availability are two different things. Internal monitoring tracks system operational status -- whether servers are down and whether key processes are running. User perception tracks the success of each individual request. When the system is running normally but individual requests fail, internal monitoring shows 100% availability while the user experience might be only 98%.

An important improvement we made was making user-perceived availability the core metric. Instead of just checking whether the system was down, we tracked whether every user request succeeded. This metric shift revealed many issues that had previously gone unnoticed.

Availability Measurement Methods

In 2019, the third question we taught users to ask was about how availability is measured. Many providers publish availability numbers tested under ideal network conditions. You need to ask three questions: Is the test frequency once per minute or once per hour? Is the test from a single data center or multiple geographic locations? Does the measurement count a single failure as an outage or require consecutive failures? The same 99%, measured differently, can feel completely different in practice.

Operational Data After Launch

Availability changes after the 2019 product launch: pre-launch testing was approximately 95% with an average recovery time of about 30 minutes. Month 1 after launch reached approximately 97% with recovery in about 15 minutes. Month 3 reached approximately 98% with recovery in about 8 minutes. Month 6 reached approximately 99% with recovery in about 3 minutes. Month 12 reached approximately 99.2% with recovery in about 2 minutes. This data shows that availability improvement is not linear -- it accumulates gradually through repeated fault handling and system improvements.

A Year of Progress

Going from 95% to 99% took us an entire year. That year, we experienced numerous faults, many late-night emergency responses, and countless moments of asking "why is this happening again." But after every fault, the system was a little more stable than before.

The biggest takeaway at the end of 2019 was this: availability is not achieved through a single architecture upgrade. It requires sustained investment and long-term accumulation. It is not designed in an office -- it is forged through every fault handled and every system improvement made. There are no shortcuts. Every percentage point gained was earned through late nights, repeated root cause analysis, and step-by-step optimization.

The Core Value of Our Service

Looking back at 2019, the most important thing was not the product code itself, but the service system we built around it and the team's commitment to "making users satisfied." Technology can be bought with money and time, but service awareness must be built through repeated user feedback. During this year, we gradually understood the nature of the proxy industry: it is not a business of IP resources. It is a business of trust. What users pay for is not an IP address -- it is the peace of mind that "someone will handle it when things go wrong." This realization influenced all our subsequent product decisions and service process design.

Looking Ahead to 2020

The experience accumulated in 2019 gave us clearer direction for 2020. With productization complete, the next focus was high-availability architecture. We needed the system to recover automatically from single-point failures rather than relying on manual operator intervention.

At the end of 2019, we conducted a comprehensive service stability assessment and identified the core problems to solve in 2020: eliminating single points of failure, reducing fault recovery time, and improving architectural automation. These goals would be addressed one by one in the coming year.

Regional Differences in Availability

In 2019, we discovered that availability varies significantly across regions. A proxy node in Beijing might achieve 99.5% availability, but the same node in Urumqi might only deliver 95%. The causes include ISP interconnection issues (limited bandwidth between China Telecom, China Unicom, and China Mobile), regional network infrastructure differences, and local network quality. Based on these findings, we optimized node distribution strategies and advised users to ask providers for test data specific to their local region.

Availability vs. Price

There is a clear correlation between availability and price. Achieving 99% availability requires significantly more resources than 95%. Resource redundancy means maintaining more nodes, automatic failover requires more complex monitoring systems, and fast response requires 24/7 staffing. But we found that many users only compare prices without comparing availability. A cheap service with 95% availability and a slightly more expensive one with 99% availability can have vastly different impacts on actual business costs -- the losses from business interruptions caused by proxy failures often far exceed the proxy service price difference.

Team Growth

In 2019, the team expanded as operations personnel, product managers, and test engineers joined, making the team more professional. With clear division of responsibilities, one group focused on core system stability, another on monitoring and alerting, and a third on user support. The results of this specialization became evident in the second half of the year -- faster fault handling and deeper post-incident reviews.

Looking Ahead to 2020

The 2019 product launch marked our transformation from a technology team to a service operations team. The direction for 2020: improve architecture fault tolerance, reduce fault recovery time, and move from 99% toward 99.5%.

One Piece of Advice

Do not just look at the availability number. Ask how it is measured. The same 99% can feel completely different depending on the methodology. If possible, ask the provider for multi-node, high-frequency real-time monitoring data rather than a packaged monthly average. Also, distinguish between system availability and request success rate -- the former is the provider's metric, the latter reflects your actual experience.