Building a Stable HTTP Proxy Service in 2017

What Happened That Year

In 2017, the proxy IP industry was in a peculiar phase. Free proxies were still the first choice for individual developers — thousands of IP addresses and ports were just a search away, seemingly good enough. But anyone who actually ran a business on them knew the truth: the average lifespan of a free proxy was measured in minutes. Many times, by the time you configured it, it was already dead.

That year, we decided to build something different — a truly stable HTTP proxy service for teams and developers who had been burned by free proxies.

Market Background

The proxy market in 2017 was undergoing several key changes:

Free proxy quality was declining rapidly. In the early days, there were many high-quality free proxies available. But as scraping demand grew and protection systems improved, free proxy lifespans shortened dramatically. By 2017, most free proxies had an average lifespan of under 30 minutes from publication to failure.

Paid proxies were emerging with uneven quality. The first wave of commercial proxy providers appeared, but most were operated by individual webmasters running a few rented servers. Service quality varied wildly. Some providers disappeared days after receiving payment.

Enterprise demand was growing. More companies were starting data collection operations. They did not need proxies that worked today — they needed services that were stable over the long term, with support available when things went wrong.

This market context directly shaped our first-year strategy: do not chase scale. Get the fundamentals right first.

What Stability Meant to Us

We spent a significant amount of time in 2017 thinking about one question: what makes a proxy truly stable?

We broke it down into three layers. This framework still guides us today.

Layer 1: Protocol reliability.

An HTTP proxy looks simple on the surface — the client sends a CONNECT request, and the proxy forwards data. In real network environments, the reality is far more complex.

When your browser or application is configured to use an HTTP proxy, all requests go to the proxy server first. For HTTP requests, the proxy forwards them directly. For HTTPS requests, the proxy first establishes a CONNECT tunnel with the target server, then transmits encrypted data through the tunnel.

Three aspects are most prone to problems:

The first is connection reuse. HTTP supports Keep-Alive, but the proxy must manage connections properly. If handled incorrectly, multiple requests sharing the same connection can mix up data. You may have seen this happen: two different web pages return identical content.

The second is timeout handling. Every proxy connection has a timeout. If the timeout is too short, slow-loading pages disconnect frequently. If too long, idle connections exhaust server resources. Finding the right balance requires extensive real-world testing.

The third is protocol compatibility. Different target servers support different HTTP versions — some still run HTTP/1.0, while others have upgraded to HTTP/2. The proxy must handle all of them.

We spent most of 2017 doing precisely this work: building test environments, simulating various network scenarios, validating protocol behavior under high concurrency. None of this was directly visible to users, but it laid the foundation for everything that followed.

Layer 2: Resource redundancy.

What happens when one server fails? If the resource pool has only one source, the entire service breaks when that source becomes unavailable.

Many proxy providers in 2017 made the mistake of single-source dependency. They rented one server or used one upstream provider. When that single dependency failed, all users went down together.

We established a multi-source strategy from the beginning. Even with limited scale, we connected to multiple resource sources simultaneously. If any one source had problems, others could take over immediately. This principle has never changed.

Layer 3: Monitor-driven awareness.

If the user knows about a service problem before you do, your monitoring is not good enough.

In 2017 we implemented something that seems obvious today but was not common at the time: proactive node monitoring. Rather than waiting for users to report issues, we tested each node ourselves — sending test requests every few minutes. If consecutive failures occurred, the node was automatically marked as abnormal and traffic was switched to backup nodes.

How to Judge Proxy Stability

The first metric we taught users to look at was connectivity rate.

The calculation is straightforward:

Connectivity rate = (successful connections / total attempts) × 100%

The testing conditions matter enormously:

Test frequency. Testing once per hour gives only 24 data points per day, potentially missing brief network fluctuations. Testing every minute captures much finer granularity. Two services claiming 99% connectivity can deliver completely different experiences depending on test frequency.

Test target. Connectivity is relative to a specific target. Testing against the provider's own servers always yields 100%. But the user's actual target website may be completely unreachable. Realistic testing mimics actual user access patterns.

Test duration. Five minutes and 24 hours reveal very different stability profiles. Some proxies perform well briefly but fail extensively during peak evening hours.

A 99% connectivity rate means 1 failure per 100 requests. That sounds good — until you make 100,000 requests a day, resulting in 1,000 failures. What truly matters is what happens after a failure: are there retry mechanisms? How fast is node switching?

Market Choices in 2017

Users looking for proxy services in 2017 had roughly three options:

Free proxies. Zero cost but unreliable. Most free IPs had lifespans under one hour. Useful for short tests but disastrous for production workloads.

Self-built proxies. If you had a server, you could set up Squid or similar services. Full control but required server maintenance, IP bans, bandwidth issues, and single points of failure.

Commercial proxies. A handful of providers existed. More expensive but with operational support. The key question became: who was genuinely providing a service versus just selling bandwidth?

2017 Free Proxy Data

We conducted a month-long test of free proxy quality in 2017:

HTTP vs SOCKS5 Decision

An important technical decision in 2017 was whether to build HTTP proxy first or jump directly to SOCKS5.

HTTP proxy advantages: broad compatibility. Almost all programming languages, operating systems, and browsers support HTTP proxy configuration natively. No additional software needed.

SOCKS5 advantages: higher flexibility. Supports TCP and UDP beyond HTTP. Required for non-HTTP services.

Our decision was to build both. HTTP proxy launched first to cover the largest user base, while SOCKS5 development and validation ran in parallel.

What We Built That Year

• Completed HTTP/HTTPS and SOCKS5 protocol adaptation and validation
• Built a preliminary multi-source resource pool architecture
• Established proactive monitoring mechanisms
• Acquired our first batch of enterprise users, whose feedback drove subsequent development

Lessons from Year One

The biggest realization was that stability is not just a technical problem. It is an organizational problem, a process problem, and a communication problem.

Technical stability can be achieved through architecture design. Service-level stability requires team coordination: fast response to user issues, failure detection before users notice, maintenance without service interruption.

Looking Ahead to 2018

The foundation we built in 2017 set the direction for 2018: turn technical capabilities into standardized products. Users should not have to talk to us to get started — they should be able to use the service on their own.

The Proxy Connection Lifecycle

Understanding proxy stability requires understanding the full lifecycle of an HTTP proxy request.

When you configure a proxy, an HTTP request goes through these steps:

1. Your program establishes a TCP connection to the proxy server (three-way handshake)
2. After the TCP connection is established, it sends the HTTP request to the proxy
3. The proxy parses the request and determines the target address
4. The proxy establishes a connection to the target server
5. For HTTPS, the proxy performs a CONNECT handshake with the target
6. The proxy begins forwarding data between the client and target

Each step can fail in different ways. The TCP handshake can time out if the proxy is overloaded. The DNS resolution can fail if the cache is polluted. The proxy-to-target connection is the most fragile segment. Even if the proxy itself is fine, problems in the international link between the proxy and target will make the user feel the connection is unstable.

This taught us a valuable lesson: proxy stability is not about any single node. It is the sum of quality across the entire path from you to the proxy, from the proxy to the target, and back again.

Businesses Most Affected by Proxy Stability

Different users have different sensitivity to stability issues. Based on our conversations with early users, these businesses required the highest stability:

Automated data collection. Hundreds of tasks running in parallel magnify any failure rate. An unstable proxy causes cascading retries that increase latency and may trigger timeouts.

E-commerce monitoring. Competitor pricing and product tracking require continuous proxy connections. Even brief interruptions can create data gaps that affect business decisions.

Ad verification. Simulating users in different regions to verify ad placements requires frequent proxy switching with reliable connections.

Account management. Managing multiple social media accounts requires stable IP assignments. Frequent disconnections not only reduce efficiency but may trigger platform security mechanisms.

Year-End Reflection

At the end of 2017, we conducted our first comprehensive retrospective.

What went well:

• Protocol validation investment paid off. The testing framework we built continued to serve us for years
• Multi-source architecture gave us basic high-availability from the start
• Proactive monitoring built a team culture focused on reliability

What could have been better:

• Productization was slow. We were doing technical validation rather than product development
• Lack of standardized documentation. Each new user required extensive manual communication
• No clear SLA commitments. User expectations varied widely based on word-of-mouth

Looking Ahead

Looking back, 2017 was the year the proxy industry transitioned from individual operators to professional services. Users were shifting from feature needs to service needs. Not just "give me a working IP" but "help me when things go wrong."

This shift meant that competition would no longer be about IP resources alone. It would be about service systems: better monitoring, faster response, clearer documentation, more reliable SLAs.

One Piece of Advice

When choosing a proxy service, do not just look at whether it works right now. Ask about the long-term connectivity rate and how it is measured. A 99% rate tested once per hour against the provider's own server is worth far less than the same number tested every minute against real-world targets.