Performance is a big question mark with IPFS. There are many threads (hah!) exploring different avenues of this much broader question.
The avenue I decided to choose: IPFS gateways. What is an IPFS gateway? It's how browser users can access IPFS-hosted content without running a daemon. Said another way, it helps create a seamless bridge from the web today to the DWeb (IPFS in this case).
There's no shortage of community-provided gateways out there. The most common is ipfs.io, since Protocol Labs - the company that created IPFS - runs it. But other strong contenders exist, including Cloudflare.
Looking at the available options, a thought occurred in my head - which one is the best for my day-to-day use? Well, luckily I had some extra time this week so I decided to find out.
Overview of the Test
I wanted to keep this test simple. I'm not going to claim this is the end all be all. I welcome discussion, criticism, and iteration.
With that said, the test consists of calls to three different files:
You can actually access them using the links above. I have them pinned on Pinata's pinning service and plan to keep them up for the foreseeable future.
I call each of those files from the addresses of seven different gateway providers:
- Protocol Labs (https://ipfs.io/ipfs/)
- Cloudflare (https://cloudflare-ipfs.com/ipfs/)
- Infura (https://ipfs.infura.io/ipfs/)
- Pinata (https://gateway.pinata.cloud/ipfs/)
- Eternum (https://ipfs.eternum.io/ipfs/)
- Siderus (https://siderus.io/ipfs/)
- Temporal (https://gateway.temporal.cloud/ipfs/)
I also threw in one more twist - running that test across different geographies:
- SA-Sao Paolo
To test from these different geo's I used t2.micro instances from AWS.
If you're interested in the code to run these tests, I've uploaded it to GitHub. You can follow the README to get started on your own machine.
NOTE: If you want to skip my analysis of the results, feel free to access the raw data I used here.
Method for Determining Winners
One note before I share my methodology: I'm not looking for the fastest gateway. Or the one with least variance in performance. I'm looking for a balance between speed and consistency.
To that end, I determined a winner across four sub-categories. Then I used a fifth sub-category as a tiebreaker. Those categories are:
- Lowest Minimum speed across 10 tries
- Lowest Maximum speed across 10 tries
- Lowest Average speed across 10 tries
- Lowest Median speed across 10 tries
- Tiebreaker: Lowest Standard Deviation across 10 tries
OK, these next few tables get granular. So for folks looking for a generic answer to take home to the bank: Cloudflare wins. For readers who are into the nuance, keep scrolling!
Quick legend on what the letters in the table below mean:
- PL = Protocol Labs
- C = Cloudflare
- I = Infura
- P = Pinata
- E = Eternum
- S = Siderus
- T = Temporal
If you want to see the raw data & tables in excel, you can check it out here.
|us-east||0.03 (I)||0.08 (C)||0.04 (I)||0.04 (I)||-||Infura|
|us-west||0.02 (PL)||0.14 (PL)||0.05 (PL)||0.02 (PL)||-||Protocol Labs|
|eu-london||0.04 (PL)||0.06 (PL)||0.05 (PL)||0.05 (PL)||-||Protocol Labs|
|eu-frankfurt||0.03 (S)||0.19 (PL)||0.06 (PL)||0.04 (C)||-||Protocol Labs|
|ap-tokyo||0.02 (PL)||0.32 (PL)||0.07 (PL)||0.03 (PL)||-||Protocol Labs|
|ap-mumbai||0.02 (C)||0.41 (P)||0.08 (C)||0.03 (C)||-||Cloudflare|
|ap-sydney||0.02 (C)||0.28 (C)||0.06 (C)||0.03 (C)||-||Cloudflare|
|sa-saopaolo||0.04 (C)||0.41 (C)||0.09 (C)||0.06 (C)||-||Cloudflare|
|us-east||0.06 (I)||0.10 (C)||0.08 (C)||0.08 (C)||-||Cloudflare|
|us-west||0.07 (C)||0.35 (C)||0.12 (C)||0.08 (C)||-||Cloudflare|
|eu-london||0.14 (S)||0.28 (P)||0.22 (P)||0.17 (S)||0.03 (P)||Pinata|
|eu-frankfurt||0.07 (C)||0.18 (P)||0.11 (P)||0.07 (C)||0.03 (P)||Pinata|
|ap-tokyo||0.06 (PL)||0.80 (C)||0.19 (C)||0.07 (PL)||0.21 (C)||Cloudflare|
|ap-mumbai||0.05 (S)||0.76 (C)||0.13 (C)||0.06 (C)||-||Cloudflare|
|ap-sydney||0.08 (S)||0.80 (C)||0.16 (C)||0.09 (S)||0.23 (C)||Cloudflare|
|sa-saopaolo||0.09 (C)||0.52 (C)||0.18 (C)||0.12 (C)||-||Cloudflare|
|us-east||1.46 (C)||1.71 (C)||1.52 (C)||1.48 (C)||-||Cloudflare|
|us-west||1.42 (S)||10.27 (T)||2.62 (C)||1.51 (S)||-||Siderus|
|eu-london||2.66 (C)||3.54 (E)||3.08 (E)||2.99 (E)||-||Eternum|
|eu-frankfurt||1.46 (C)||2.62 (P)||1.88 (P)||1.55 (S)||-||Pinata|
|ap-tokyo||1.45 (C)||11.55 (C)||2.67 (C)||1.67 (C)||-||Cloudflare|
|ap-mumbai||1.43 (S)||13.90 (C)||2.73 (C)||1.48 (S)||3.92 (C)||Cloudflare|
|ap-sydney||0.35 (S)||16.76 (C)||3.47 (C)||1.97 (C)||-||Cloudflare|
|sa-saopaolo||1.44 (C)||15.88 (C)||3.04 (C)||1.59 (C)||-||Cloudflare|
*Std Dev is only used as a tie breaker
Discussion of Results
As you can see - in general, Cloudflare's gateway is a safe choice. It provides solid performance across geo's and file types. They're also the best gateway, regardless of file type, if you're in the Mumbai, Sydney, or Sao Paolo regions.
Makes sense, it is their sole business to be good at this.
One final note about these results. I'm missing a big data point when looking at these performance metrics: the traffic on each gateway. My hypothesis is that Protocol Labs gets more traffic on their gateways. After all, most people default to their gateway address.
More traffic = more problems. And that variable is not captured anywhere in my data. But, I hope that this analysis ends up drawing attention to alternative gateways as a way to load balance.
If we self-regulate (even a little!) how we use the resources available to us as a community, then that's a win.
Final Note - Cold Start Problem
Pulling this data has been pretty eye-opening. One observation in particular has me thinking. Something I've observed before in the world of Serverless.
Cold Start Problem
In the context of Serverless, Mike Roberts describes the cold start problem as:
It takes some time for a FaaS platform to initialize an instance of a function before each event. This startup latency can vary significantly, even for one specific function, depending on a large number of factors, and may range anywhere from a few milliseconds to several seconds...
Initialization of a Lambda function will either be a “warm start”—reusing an instance of a Lambda function and its host container from a previous event—or a “cold start” —creating a new container instance, starting the function host process, etc. Unsurprisingly, when considering startup latency, it’s these cold starts that bring the most concern.
- Mike Roberts, Serverless Architectures
In the context of IPFS gateways, this problem seems to exist (in some capacity at least). For example, check out Cloudflare's performance on the text file (~1.3KB) by geography:
Pretty clear. There's a big performance improvement between attempt 1 and attempts 2 - 10 for every geo. But, I'm not sure this is an IPFS issue. To illustrate - here's the same cut, except with Infura's performance:
Much less clear to me that the cold start problem exists. To be frank, I'm not sure where this discrepancy comes from. Is it IPFS itself? Is it the architecture of the gateways? Am I doing something wrong? Unclear to me.
But it's a thing.
Let's be clear - this isn't perfect. It's a simple & generalized test that I put together to look at some numbers. I answered my personal question: which gateway should I be using? But, it's possible I didn't answer yours.
If so, feel free to use my raw data for your own analysis. Or, run the test yourself and share the results. OR if you're feeling extra ambitious - tell me why the test is wrong by pushing a PR to edit the test. I welcome it.
Finally, if you have suggestions on other IPFS-related topics to explore feel free to hit me up on Twitter.