Benchmarking IPFS Gateways

Disclaimer: I worked at Protocol Labs (the company behind IPFS) for 6 months. I no longer work there.

Performance is a big question mark with IPFS. There are many threads (hah!) exploring different avenues of this much broader question.

The avenue I decided to choose: IPFS gateways. What is an IPFS gateway? It's how browser users can access IPFS-hosted content without running a daemon. Said another way, it helps create a seamless bridge from the web today to the DWeb (IPFS in this case).

There's no shortage of community-provided gateways out there. The most common is ipfs.io, since Protocol Labs - the company that created IPFS - runs it. But other strong contenders exist, including Cloudflare.

Looking at the available options, a thought occurred in my head - which one is the best for my day-to-day use? Well, luckily I had some extra time this week so I decided to find out.

Overview of the Test

I wanted to keep this test simple. I'm not going to claim this is the end all be all. I welcome discussion, criticism, and iteration.

With that said, the test consists of calls to three different files:

You can actually access them using the links above. I have them pinned on Pinata's pinning service and plan to keep them up for the foreseeable future.

I call each of those files from the addresses of seven different gateway providers:

I also threw in one more twist - running that test across different geographies:

  • US-East
  • US-West
  • EU-London
  • EU-Frankfurt
  • AP-Tokyo
  • AP-Mumbai
  • AP-Sydney
  • SA-Sao Paolo

To test from these different geo's I used t2.micro instances from AWS.
If you're interested in the code to run these tests, I've uploaded it to GitHub. You can follow the README to get started on your own machine.

NOTE: If you want to skip my analysis of the results, feel free to access the raw data I used here.

Method for Determining Winners

One note before I share my methodology: I'm not looking for the fastest gateway. Or the one with least variance in performance. I'm looking for a balance between speed and consistency.

To that end, I determined a winner across four sub-categories. Then I used a fifth sub-category as a tiebreaker. Those categories are:

  • Lowest Minimum speed across 10 tries
  • Lowest Maximum speed across 10 tries
  • Lowest Average speed across 10 tries
  • Lowest Median speed across 10 tries
  • Tiebreaker: Lowest Standard Deviation across 10 tries

Results

OK, these next few tables get granular. So for folks looking for a generic answer to take home to the bank: Cloudflare wins. For readers who are into the nuance, keep scrolling!

Quick legend on what the letters in the table below mean:

  • PL = Protocol Labs
  • C = Cloudflare
  • I = Infura
  • P = Pinata
  • E = Eternum
  • S = Siderus
  • T = Temporal

If you want to see the raw data & tables in excel, you can check it out here.

Text Results

Region Min Max Avg Median Std Dev* Winner
us-east 0.03 (I) 0.08 (C) 0.04 (I) 0.04 (I) - Infura
us-west 0.02 (PL) 0.14 (PL) 0.05 (PL) 0.02 (PL) - Protocol Labs
eu-london 0.04 (PL) 0.06 (PL) 0.05 (PL) 0.05 (PL) - Protocol Labs
eu-frankfurt 0.03 (S) 0.19 (PL) 0.06 (PL) 0.04 (C) - Protocol Labs
ap-tokyo 0.02 (PL) 0.32 (PL) 0.07 (PL) 0.03 (PL) - Protocol Labs
ap-mumbai 0.02 (C) 0.41 (P) 0.08 (C) 0.03 (C) - Cloudflare
ap-sydney 0.02 (C) 0.28 (C) 0.06 (C) 0.03 (C) - Cloudflare
sa-saopaolo 0.04 (C) 0.41 (C) 0.09 (C) 0.06 (C) - Cloudflare

Image Results

Region Min Max Avg Median Std Dev* Winner
us-east 0.06 (I) 0.10 (C) 0.08 (C) 0.08 (C) - Cloudflare
us-west 0.07 (C) 0.35 (C) 0.12 (C) 0.08 (C) - Cloudflare
eu-london 0.14 (S) 0.28 (P) 0.22 (P) 0.17 (S) 0.03 (P) Pinata
eu-frankfurt 0.07 (C) 0.18 (P) 0.11 (P) 0.07 (C) 0.03 (P) Pinata
ap-tokyo 0.06 (PL) 0.80 (C) 0.19 (C) 0.07 (PL) 0.21 (C) Cloudflare
ap-mumbai 0.05 (S) 0.76 (C) 0.13 (C) 0.06 (C) - Cloudflare
ap-sydney 0.08 (S) 0.80 (C) 0.16 (C) 0.09 (S) 0.23 (C) Cloudflare
sa-saopaolo 0.09 (C) 0.52 (C) 0.18 (C) 0.12 (C) - Cloudflare

Video Results

Region Min Max Avg Median Std Dev* Winner
us-east 1.46 (C) 1.71 (C) 1.52 (C) 1.48 (C) - Cloudflare
us-west 1.42 (S) 10.27 (T) 2.62 (C) 1.51 (S) - Siderus
eu-london 2.66 (C) 3.54 (E) 3.08 (E) 2.99 (E) - Eternum
eu-frankfurt 1.46 (C) 2.62 (P) 1.88 (P) 1.55 (S) - Pinata
ap-tokyo 1.45 (C) 11.55 (C) 2.67 (C) 1.67 (C) - Cloudflare
ap-mumbai 1.43 (S) 13.90 (C) 2.73 (C) 1.48 (S) 3.92 (C) Cloudflare
ap-sydney 0.35 (S) 16.76 (C) 3.47 (C) 1.97 (C) - Cloudflare
sa-saopaolo 1.44 (C) 15.88 (C) 3.04 (C) 1.59 (C) - Cloudflare

*Std Dev is only used as a tie breaker

Discussion of Results

As you can see - in general, Cloudflare's gateway is a safe choice. It provides solid performance across geo's and file types. They're also the best gateway, regardless of file type, if you're in the Mumbai, Sydney, or Sao Paolo regions.

Makes sense, it is their sole business to be good at this.

Pulling up second and third place are Protocol Labs and Pinata. The first is the company behind IPFS. The second is a well reputed startup pinning service in the community. Seems to make sense to me.

One final note about these results. I'm missing a big data point when looking at these performance metrics: the traffic on each gateway. My hypothesis is that Protocol Labs gets more traffic on their gateways. After all, most people default to their gateway address.

More traffic = more problems. And that variable is not captured anywhere in my data. But, I hope that this analysis ends up drawing attention to alternative gateways as a way to load balance.

If we self-regulate (even a little!) how we use the resources available to us as a community, then that's a win.

Final Note - Cold Start Problem

Pulling this data has been pretty eye-opening. One observation in particular has me thinking. Something I've observed before in the world of Serverless.

Cold Start Problem

In the context of Serverless, Mike Roberts describes the cold start problem as:

It takes some time for a FaaS platform to initialize an instance of a function before each event. This startup latency can vary significantly, even for one specific function, depending on a large number of factors, and may range anywhere from a few milliseconds to several seconds...
Initialization of a Lambda function will either be a “warm start”—reusing an instance of a Lambda function and its host container from a previous event—or a “cold start” —creating a new container instance, starting the function host process, etc. Unsurprisingly, when considering startup latency, it’s these cold starts that bring the most concern.
- Mike Roberts, Serverless Architectures

In the context of IPFS gateways, this problem seems to exist (in some capacity at least). For example, check out Cloudflare's performance on the text file (~1.3KB) by geography:

Cold Start Problem in Full Effect

Pretty clear. There's a big performance improvement between attempt 1 and attempts 2 - 10 for every geo. But, I'm not sure this is an IPFS issue. To illustrate - here's the same cut, except with Infura's performance:

Maybe Not?

Much less clear to me that the cold start problem exists. To be frank, I'm not sure where this discrepancy comes from. Is it IPFS itself? Is it the architecture of the gateways? Am I doing something wrong? Unclear to me.

But it's a thing.

Parting Thoughts

Let's be clear - this isn't perfect. It's a simple & generalized test that I put together to look at some numbers. I answered my personal question: which gateway should I be using? But, it's possible I didn't answer yours.

If so, feel free to use my raw data for your own analysis. Or, run the test yourself and share the results. OR if you're feeling extra ambitious - tell me why the test is wrong by pushing a PR to edit the test. I welcome it.

Finally, if you have suggestions on other IPFS-related topics to explore feel free to hit me up on Twitter.

Show Comments

Get the latest posts delivered right to your inbox.