Fly App performance optimization Blog
Docs Community Status Pricing Sign In Get Started RSS Feed
Reading time • 16 min Share this post on Twitter Share this post on Hacker News Share this post on Reddit
32 Bit Real Estate
Kurt Mackey
@mrkurt
Image by Annie Ruygt
Fly.io runs apps close to users, by transmuting Docker containers into micro-VMs that run on our own hardware around the world. This is a post about one of the major costs of running a service like ours, but if you're more interested in how Fly.io works, the easiest way to learn more is to try it out; you can be up and running in just a couple minutes.
Two obvious costs of running Internet apps for users on your own hardware: hardware and bandwidth. We buy big servers and run them in racks at network providers that charge us to route large volumes of traffic using BGP4 Anycast. You probably have at least a hazy intuition for what those costs look like.
But there's a non-obvious cost to what we do: we need routable IPv4 addresses. To make Anycast work, so users in Singapore are routed to Singapore instances, particularly for non-HTTP applications, like UDP DNS servers or TCP game servers, we assign distinct public IPv4 addresses to each app running on Fly.io.
You don't have to think about any of this stuff when you deploy on Fly.io. We just assign apps addresses and get out of the way. But we have to think a lot about this stuff, because IPv4 addresses cost money.
You Can't, Like, Own a Number, Man!
Let's take a second to reflect on what it means to acquire an IPv4 address. To own an IP address is to control what it routes to. The rock-bottom source of truth for IPv4 routing is BGP4. You announce IP address prefixes over IPv4 to your peers, your peers relay those announcements, and the world learns that traffic for your addresses needs to be sent your way.
Lots of providers will accept a BGP4 peering session with a customer. But no reputable provider will take a random announcement over that session. You'll need to demonstrate authority over the prefixes you plan to announce.
Technically, nobody owns an IPv4 address. They're administered, ostensibly as a public benefit, by the 5 Regional Internet Registries --- ARIN, RIPE, APNIC, AFRINIC, and LACNIC. In the long long ago, it was the job the RIRs to allocate blocks of addresses to providers. Today, their IPv4 hives are void of honey, and their new job is just to keep track of who's claiming which long-allocated blocks as they get shuffled around between their new, er, stewards.
All this is to say, if you want to "own" an IPv4 address, you find its previous owner and get them to arrange a transfer with an RIR. There is paperwork involved, and it's sometimes whispered, a usage justification.
With the transfer completed, your newly "acquired" IPv4 addresses will be associated with your BGP4 ASN (oh, you'll need to shell out a couple hundred bucks for the ASN, too), and you can hope that an upstream provider will accept announcements for them. The addresses are now "yours".
For illustration: here's a block of IPv4 addresses we control in IRR Explorer.
Certain Plans Require Additional Gribbles
So, Anycast. Easy. You have a routable IPv4 address. You have machines in a bunch of places. In each of those places, you peer BGP4 with your upstream and advertise that exact same IPv4 address. Global BGP4 routing ensures that when people try to connect to to that address, they'll be sent to the "closest" machine. That's how Fly.io's Anycast network functions: we have blocks of addresses, machines in a bunch of regions, BGP4 peering in each of them, and a Rust-based proxy that picks up connections for those addresses.
You can think of ways to work around needing routable IPv4 addresses for Anycast applications. None of them work for us.
Here's the first thing that doesn't work: IPv6. Let's get it out of the way first. Yes, there is such a thing as IPv6. Yes, it has enough address space to give distinct addresses to each electron in every atom of every living human. And, yes, we do a lot with IPv6, including allocating public IPv6 prefixes to all of our applications. We're IPv6 fans.
But if I had to place a bet, it'd be that you aren't using IPv6. And you're nerdy enough to be reading a blog post on how much IPv4 addresses cost! The silent majority of non-nerd users are nowhere close to being reliably IPv6-connected.
The next thing that doesn't work is name-based hosting. Applications can share IPv4 addresses using TLS SNI. We could park most apps on a single address and just let browsers tell our proxies which apps people are looking for. So far so good: the people who want their own IPv4 addresses would be flying business class, and we could charge them 2x as much.
A word about market segmentation. As a software business operator, you have two basic goals: you want everyone to use your software, and you want to earn profits. Those goals are in tension. The classic b-school 101 answer is to identify the customers who derive the most value from your software. Now figure out their distinctive use cases, charge more for them, and less for everything else. I can keep explaining this poorly, but Joel Spolsky has already explained it well, and you should go read his post.
There's a standard market-seg playbook for SAAS companies. You've seen it every time you've looked at a product pricing grid. Two of the most infamous segmentation premiums are HTTPS (thankfully, this is going extinct) and the SSO tax (still very much alive).
We're not above market segmentation. If you're a billion dollar company, rest assured, we will find ways to extract money from you! But IPv4 addresses aren't one of those ways, because charging extra for them gets in everyone's way. For what it's worth: when enough people ask, we'll ship SSO. But we won't tax it.
We like money as much as anyone else. But SNI breaks down for apps that don't use TLS, and we want to support all kinds of apps, not just web apps. Now, most Fly.io apps are web apps. And we know which apps are which. So we could use SNI for the majority of our apps and IPv4 addresses for the exceptions. But that feels weird, and worse, means we'd be maintaining multiple provisioning paths for customers. It's not worth the hassle, for us or our customers, so IPv4 addresses are just bundled in for everyone.
IPv4: The Internet's Real Estate Market
There are lot of organizations holding addresses they don't need. Internet routing used to be real, real dumb, and to give a big company enough addresses to number their machines, you often had to give them way more addresses than they really needed. Apple has a /8 – that's 256 /16s, each of which has 256 /24s, each of which is 256 addresses. So does Prudential Insurance(?!), and so does Ford. Lots of other organizations had "smaller" blocks that are still unfathomably large to modern sensibilities. So, like a crappy cryptocurrency, IPv4 is pre-mined.
If you invested in IP addresses 15 years ago, you're doing pretty well. If you invested in IP addresses 15 months ago, you're also doing well.
When we wrote the first draft of this post, the going rate for smaller IPv4 blocks was $45 per IP address. was the rate in September of 2021. We looked just now, and the spot price was $50. We give up quoting them; by the time you get to this sentence, they'll have gotten more expensive. Here's some recent transactions.
Did you know big cloud providers publish their IP ranges? They do this because it's useful for automating firewalls. It's also useful if you want to guess how much IPv4 wealth they're sitting on. You could even write a Ruby script to tell you. Which I did. (Dan Goldin did the same thing 6 months before me).
Here's what I saw in March of 2021 when IPv4s cost $25 each:
Total IPs
AWS 3,008 55,708,181 ~$1,392,704,525
Google Cloud 388 7,582,976 ~$189,574,400
DigitalOcean 1,581 2,368,896 ~$59,222,400
And here's what it looked like in September of 2021:
DigitalOcean 1,601 2,528,640 ~$113,788,800
These estimates are low: Amazon announces more than 100 million IP addresses. That's 2.4% of all possible IPv4 addresses!
IP prices vary by block size. It's helpful to think of a 256-address /24 as the irreducible block size. That /24 is 10-15% less expensive than a /18, which is 64 contiguous /24s. There are reasons for the price difference, and we don't know all of them. One big reason is that if you're doing classical network engineering and you need to number 10,000 hosts, the routing is easier to do with a /18 than with 40 /24s. Another reason is probably vanity.
A fun, useless fact: time was when there were particularly valuable /24s. Backbone routing was done principally with Cisco 7500 routers, which used TCAM memory --- think "hardware hash table" --- to make quick forwarding decisions. TCAM space was limited, and the Tier 1 network providers used BGP filters to enforce a maximum prefix length; for instance, in the late 1990s, Sprint wouldn't forward anything longer than a /19, which meant in effect that if you wanted to be reachable to anyone through Sprint, you needed at least a /19 to play.
But because IPv4 allocation had been so janky in the early 1990s (it literally only worked on octet boundaries, so you either had a /24 or you had a /16, with nothing in between), there were legacy prefixes that had to be grandfathered in through the filters. These were known as "swamp space"; a swamp /24 was announceable across the Internet, even though a commodity /24 wasn't.
None of this matters anymore, and you can announce all the /24s you want today.
Anyways, AWS's average block comprises 18,000 IPv4s. Bonkers. Giant booksellers who sell cloud services as a side hustle don't even bother with IPv6. Here is a complete list of the IPv6 addresses dig aaaa amazon.com returns:
dig aaaa amazon.com +short
Financial Network Engineering
Imagine for a moment that you're us. Take a drink, eat an Italian beef sandwich, pet your dog. Now get apps working for developers in, say, twenty different cities. You can buy a beast of a server for $20k USD, chop it up into 500+ virtual machines, and build an API for turning those on and off very quickly. People are pretty happy to pay for virtual machine time; you're almost there.
VMs are mostly useful when they can talk to the Internet. Your developers need IP addresses. Say it costs $45 for a single IP address. That means it costs $23,040 to handle 512 virtual machines with IPv4 addresses. We've now doubled our costs.
But hold up for a second.
Servers depreciate; that's their job. That $20k server depreciates to $500 relatively quickly. But IP blocks are, at least for now, appreciating. Alarmingly.
This is a hard market to time. Nobody believes the Internet will be IPv6-only within the next few years. There are credible people who believe IPv4 addresses will be scarce and useful indefinitely. We might put money on you getting away with an IPv6-only app 20 years from now. But we'd have said that 20 years ago, too.
So, unfortunately, the smart thing to do if you own $1B worth of IPv4 addresses is to buy $1B more IPv4 addresses. Fortunately, everyone is starting to recognize this, so you have some flexibility in financing.
If you're a farmer and you need a new tractor, you probably don't just go buy it in cash. You go to a bank and get a loan, collateralized by the value of the tractor. If you're a company with $500,000 in receivables, you don't necessarily have to wait for your customers to remit payments; financial institutions will loan you money collateralized by your invoices. And, of course, when you buy a house, you live in it while you pay back a mortgage to the bank.
Your local credit union won't collateralize a loan on IPv4 blocks like it would for a car or tractor. IPv4 addresses are just numbers, man. Logan Paul can convince unsophisticated people to put a dollar value on completely useless numbers. But your bank isn't so forward thinking, and the conversation putting a $50 value to a single 32-bit number isn't going to be productive.
But startups have another option: venture debt. Banks will happily lend startups money based almost entirely on the credibility of a startup's investors, and, transitively, their blog posts. When a company raises a $12MM round, they'll spend the next few weeks raising $3MM of debt from banks friendly with that firm.
Venture debt is surprisingly simple. It has an interest rate, which can be pretty close to prime. It usually includes an option grant – the bank gets a small amount of equity (much less than employees get). And venture debt typically requires that you do all your banking with the bank that gave you the loan. Unlike raising a funding round, once you've got investors, there isn't much pitching involved in raising debt. Banks win or lose venture debt deals based on how much certain founders like their web interface. It's not not the most important factor when choosing a bank.
All this is to say, you can in a sense take out a mortgage on a block of IP addresses.
Fly.io apps get routable IPv4 addresses.
You can run more than just an HTTPS app here; DNS servers, game servers, and media relays work too. If your app has a Docker container, you should be able to get it running here easily.
Deploy your app in minutes. →
How Much Sense Does This Make Long-term?
IPv6 will be viable – most people will be able to run apps with no IPv4 addresses – someday. Could happen in 2 years. Far more likely to happen in 20.
The kinds of companies that benefit the most from a decisive transition to IPv6 tend to look a lot like us: new, small, minimal legacy infrastructure. Not coincidentally, those companies also tend not to have much market power. Strong incentive, weak market power: not exactly a recipe for change.
There are other reasons to think IPv4 addresses will retain value. Path dependence is one of them; .COM names are still pricey despite the universal acceptance of new TLDs. Switching costs are another; lots of very large networks are IPv4-only, some with significant numbers of nodes that can't easily be dual-stack. It's easy to imagine a world in which IPv6-only is viable, but IPv4 addresses remain scarce.
Still, IPv4 will be obsolete some day. Addresses will increase in value and then one night, while we're all sleeping, they'll become worthless.
This could be a knock against using a bunch of debt to pay for IP assignments, or not.
An alternative is leasing: companies of varying levels of sketchiness will happily lease IP blocks. We've had a bunch of offers for two bucks per month per address. Maybe we could negotiate that down to one bucks. A three year lease for 65,000 IP addresses at $1/mo each works out to $2.34 million. Four years is $3.12 million.
We'd almost certainly get more addresses by leasing. And we wouldn't have the inventory risk of holding thousands of IPv4 addresses that could conceivably be worth $0 if IPv6 takes over. But leases have fixed terms, and running apps don't. Meanwhile, we do know empirically that people hate renumbering. So do the IPv4 lessors. We'd be in no position to negotiate a good rate on lease renewal – in fact, we'd be in a bad position even if IPv4 addresses got much cheaper. We want to know that when we assign an address to an app, the people running that app will get to keep using that address indefinitely. Leasing isn't worth the hassle for us.
IPv4 Is Exhausting
At scale, routable IPv4 addresses are surprisingly expensive. But they're appreciating. Like an office building or a house, we have tools for financing them that we don't for machines or salaries. So, part of our business is acquiring addresses outright, and attaching them to apps people run here.
For most ordinary apps, this doesn't matter much; HTTPS web applications would run just as well with SNI and shared (or temporary) addresses as they do with permanent IPv4 allocations. But not all apps are ordinary, and we want the oddball apps to work well on Fly.io without anyone having to think much about them.
Mostly, we just believe that people who run apps on Fly.io now will still be running them in 10 years, and Fly.io is better if we can keep that IP for them the entire time.
Really, though, we're nerds, and we think it's funny that for all the talk of NFTs and ICOs, the Internet itself has been running a high-dollar token auction for the last 20 years.
One last thing: a new 5-digit ASN will cost you about $500 from ARIN, but there are auctions for 4-digit ASNs, and they run into mid-five-figures. If any of you can explain this to us, we'd be grateful.
Share this post on Twitter Share this post on Hacker News Share this post on Reddit
Next post ↑
How Safari Ruined My Tuesday
Previous post ↓
Fly.io Is Hiring Technical Writers