IPFS Gateway performance as an Immutable metadata endpoint

What is a metadata endpoint?

A metadata endpoint serves as the source of NFT metadata on the Immutable platform. When items are minted, our platform queries this endpoint to retrieve specific metadata for each item. Collection owners have the ability to update this metadata and request a refresh for a specific set of tokens.

Why IPFS?

IPFS (InterPlanetary File System) is a cost-effective solution, particularly for startups and small projects. It eliminates the need to build your own caching or API infrastructure by leveraging Public Gateways like Pinata, Web3.storage, and NFT.storage.

What is IPFS? https://docs.ipfs.tech/concepts/what-is-ipfs/
What is a gateway? https://docs.ipfs.tech/concepts/ipfs-gateway/

Understanding Metadata Endpoints

The metadata endpoint functions as a shared platform service responsible for retrieving asset metadata. Its requirements differ slightly from simply fetching the NFT image. One important aspect to note is that when retrieving assets from an endpoint, there is a 5-second timeout. This timeout ensures a stable service for all platform users. Additional platform limitations can be found in the documentation linked here [deep dive on Immutable metadata].

Deep dive on Immutable metadata: https://docs.x.immutable.com/docs/deep-dive-metadata/

Comparison of Popular Gateways:

Here is a comparison of some popular gateways:

Pinata:
- Free Tier: Available
- Features: JS SDK, REST API
- Limitations: Maximum 500 pinned files, unable to upload directories with more than 5k-10 files, global rate limit of 15 requests per minute per CID, IP address rate limit of 200 requests per minute, does not support IPFS DAG feature.
- Useful Links
  - https://www.pinata.cloud/pricing
  - https://docs.pinata.cloud/sdks
Web3 Storage:
- Free Tier: Available
- Features: JS SDK, Go SDK, API
- Limitations: 30 requests per 10 seconds per token, 5GB storage limit on the free tier.
- Useful Links:
  - https://web3.storage/pricing/
  - https://web3.storage/docs/

For the latest features and limits, refer to the services' websites.

Performance Experiment:

In our experiment, we uploaded four batches of metadata directories containing 20k, 80k, 120k, and 200k items to both web3.storage and nft.storage. We then queried the files using Python and GoLang to measure the retrieval time for metadata.

If you are interested in performing similar tests or exploring the upload and performance code, you can find the open-source code here.

Conclusions

Using IPFS as a metadata endpoint for large collections that require regular updates is not optimal. Performance is notably impacted when dealing with a large number of files in a directory. The data from our experiment showed that non-cached response times nearly doubled with each increase in batch size. Caching and distribution performance can vary, depending on the cache status of the queried node.

Fortunately, the 5-second timeout should suffice for most use cases, including larger collections, provided the data has been propagated on IPFS. For large directories, it is advisable to upload them well in advance to ensure sufficient time for full archiving on IPFS. Collections with 100k+ items may take up to 48 hours to fully archive for reasonable performance.

Among the gateways tested, NFT Storage proved to be the easiest to work with when using the NFT Up application. It uploaded the entire collection in approximately 30 minutes, although larger collections took longer to archive. Performance-wise, NFT Storage and Web3 Storage exhibited similar results.

We did not test Pinata due to its inability to support large directory uploads. Additionally, GoLang outperformed Python in terms of performance, and leveraging its retry functionality could yield even better results. However, further testing in the same environment as the crawler is required to validate this observation.

Actual Results

NFT Storage

20k Files:

https://nftstorage.link/ipfs/bafybeihokq3q2tcfdsmcxk4d2sgdwql46jrqtked37xfu3dsskwrtjf2iq/

Python	Go
ID: 60 Pass 1 Response time (ms): 1337.66 Pass 2 Response time (ms): 1344.82 Pass 3 Response time (ms): 1486.84 Pass 4 Response time (ms): 137.22 Pass 5 Response time (ms): 1331.75	ID: 107 Pass 1 response time (ms): 1703.00 Pass 2 response time (ms): 118.37 Pass 3 response time (ms): 117.09 Pass 4 response time (ms): 99.75 Pass 5 response time (ms): 118.65
ID: 9157 Pass 1 Response time (ms): 1508.74 Pass 2 Response time (ms): 160.33 Pass 3 Response time (ms): 1448.84 Pass 4 Response time (ms): 1428.28 Pass 5 Response time (ms): 1535.10	ID: 6630 Pass 1 response time (ms): 1368.38 Pass 2 response time (ms): 94.59 Pass 3 response time (ms): 97.84 Pass 4 response time (ms): 87.49 Pass 5 response time (ms): 106.42
ID: 17073 Pass 1 Response time (ms): 1779.54 Pass 2 Response time (ms): 1582.19 Pass 3 Response time (ms): 2741.36 Pass 4 Response time (ms): 1779.01 Pass 5 Response time (ms): 1775.71	ID: 14270 Pass 1 response time (ms): 1263.56 Pass 2 response time (ms): 95.66 Pass 3 response time (ms): 97.39 Pass 4 response time (ms): 99.89 Pass 5 response time (ms): 102.59
ID: 18626 Pass 1 Response time (ms): 1456.60 Pass 2 Response time (ms): 1344.65 Pass 3 Response time (ms): 1248.71 Pass 4 Response time (ms): 1604.75 Pass 5 Response time (ms): 138.54	ID: 16239 Pass 1 response time (ms): 1728.05 Pass 2 response time (ms): 88.77 Pass 3 response time (ms): 89.34 Pass 4 response time (ms): 141.88 Pass 5 response time (ms): 86.64

80k Files:

https://nftstorage.link/ipfs/bafybeia6urum6qre6qlmexstxsfpcv7hnmakaerizwr2notzcnd7tji6fi/

Python	Go
ID: 192 Pass 1 Response time (ms): 3680.83 Pass 2 Response time (ms): 15498.73 Pass 3 Response time (ms): 3033.19 Pass 4 Response time (ms): 5699.16 Pass 5 Response time (ms): 3483.09	ID: 157 Pass 1 response time (ms): 3220.51 Pass 2 response time (ms): 150.40 Pass 3 response time (ms): 83.83 Pass 4 response time (ms): 100.10 Pass 5 response time (ms): 106.80
ID: 4155 Pass 1 Response time (ms): 3890.32 Pass 2 Response time (ms): 4391.63 Pass 3 Response time (ms): 5306.92 Pass 4 Response time (ms): 156.79 Pass 5 Response time (ms): 3229.33	ID: 4733 Pass 1 response time (ms): 3121.73 Pass 2 response time (ms): 104.88 Pass 3 response time (ms): 114.44 Pass 4 response time (ms): 115.91 Pass 5 response time (ms): 114.24
ID: 36004 Pass 1 Response time (ms): 3427.09 Pass 2 Response time (ms): 2870.44 Pass 3 Response time (ms): 3645.16 Pass 4 Response time (ms): 5419.92 Pass 5 Response time (ms): 2670.80	ID: 31051 Pass 1 response time (ms): 2810.81 Pass 2 response time (ms): 122.47 Pass 3 response time (ms): 71.53 Pass 4 response time (ms): 98.22 Pass 5 response time (ms): 98.37
ID: 76076 Pass 1 Response time (ms): 3872.95 Pass 2 Response time (ms): 2941.47 Pass 3 Response time (ms): 172.92 Pass 4 Response time (ms): 153.25 Pass 5 Response time (ms): 4002.98	ID: 62239 Pass 1 response time (ms): 2655.93 Pass 2 response time (ms): 120.18 Pass 3 response time (ms): 104.54 Pass 4 response time (ms): 91.27 Pass 5 response time (ms): 89.69

120k FIles:

https://nftstorage.link/ipfs/bafybeigftlfvhbargvmm6wpa7qg5omlxba7alfh6cuqn5ohakutpglwv5a/

Python	Go
ID: 45 Pass 1 Response time (ms): 5843.59 Pass 2 Response time (ms): 6762.12 Pass 3 Response time (ms): 147.24 Pass 4 Response time (ms): 7431.45 Pass 5 Response time (ms): 170.85	ID: 212 Pass 1 response time (ms): 6013.06 Pass 2 response time (ms): 130.51 Pass 3 response time (ms): 83.33 Pass 4 response time (ms): 103.69 Pass 5 response time (ms): 96.47
ID: 30456 Pass 1 Response time (ms): 6193.35 Pass 2 Response time (ms): 5197.16 Pass 3 Response time (ms): 151.55 Pass 4 Response time (ms): 7585.94 Pass 5 Response time (ms): 6863.15	ID: 41511 Pass 1 response time (ms): 6018.40 Pass 2 response time (ms): 100.40 Pass 3 response time (ms): 86.73 Pass 4 response time (ms): 109.94 Pass 5 response time (ms): 79.38
ID: 72444 Pass 1 Response time (ms): 5544.59 Pass 2 Response time (ms): 8195.07 Pass 3 Response time (ms): 161.84 Pass 4 Response time (ms): 6988.37 Pass 5 Response time (ms): 8114.08	ID: 52777 Pass 1 response time (ms): 6058.90 Pass 2 response time (ms): 131.21 Pass 3 response time (ms): 85.33 Pass 4 response time (ms): 91.24 Pass 5 response time (ms): 88.06
ID: 113690 Pass 1 Response time (ms): 5540.85 Pass 2 Response time (ms): 6074.34 Pass 3 Response time (ms): 160.31 Pass 4 Response time (ms): 6440.48 Pass 5 Response time (ms): 159.93	ID: 118622 Pass 1 response time (ms): 6788.29 Pass 2 response time (ms): 100.22 Pass 3 response time (ms): 91.80 Pass 4 response time (ms): 89.20 Pass 5 response time (ms): 89.02

200k FIles:

https://nftstorage.link/ipfs/bafybeih5hhftuksngsmvufosw5d6g3gvbwzxnymwzoankyss4ugefv64lu

Python	Go
ID: 629 Pass 1 Response time (ms): 2108.21 Pass 2 Response time (ms): 96.00 Pass 3 Response time (ms): 99.62 Pass 4 Response time (ms): 95.39 Pass 5 Response time (ms): 104.03	ID: 882 Pass 1 response time (ms): 10879.63 Pass 2 response time (ms): 8915.72 Pass 3 response time (ms): 8257.41 Pass 4 response time (ms): 8892.49 Pass 5 response time (ms): 9095.12
ID: 45017 Pass 1 Response time (ms): 2107.65 Pass 2 Response time (ms): 113.00 Pass 3 Response time (ms): 8478.66 Pass 4 Response time (ms): 137.17 Pass 5 Response time (ms): 148.65	ID: 44631 Pass 1 response time (ms): 12274.17 Pass 2 response time (ms): 11340.61 Pass 3 response time (ms): 9630.38 Pass 4 response time (ms): 9873.01 Pass 5 response time (ms): 10372.01
ID: 16149 Pass 1 Response time (ms): 1810.76 Pass 2 Response time (ms): 140.92 Pass 3 Response time (ms): 117.49 Pass 4 Response time (ms): 110.59 Pass 5 Response time (ms): 107.43	ID: 129189 Pass 1 response time (ms): 10859.37 Pass 2 response time (ms): 10084.73 Pass 3 response time (ms): 10673.78 Pass 4 response time (ms): 12278.67 Pass 5 response time (ms): 10617.60
ID: 199342 Pass 1 Response time (ms): 1689.55 Pass 2 Response time (ms): 138.46 Pass 3 Response time (ms): 104.90 Pass 4 Response time (ms): 111.81 Pass 5 Response time (ms): 105.50	ID: 179927 Pass 1 response time (ms): 10858.38 Pass 2 failed: Get "https://bafybeih5hhftuksngsmvufosw5d6g3gvbwzxnymwzoankyss4ugefv64lu.ipfs.dweb.link/179927": dial tcp 209.94.90.1:443: connect: connection refused Pass 3 response time (ms): 10196.51 Pass 4 failed: Get "https://bafybeih5hhftuksngsmvufosw5d6g3gvbwzxnymwzoankyss4ugefv64lu.ipfs.dweb.link/179927": dial tcp 209.94.90.1:443: connect: connection refused Pass 5 response time (ms): 11040.17

Web3 Storage

20k Files:

https://bafybeiez44iz46zg7nrtovw2mkm5wdew4va23uqa54bfkobbb4mkjf7xeq.ipfs.w3s.link

Python	Go
ID: 174 Pass 1 Response time (ms): 1353.62 Pass 2 Response time (ms): 1466.23 Pass 3 Response time (ms): 151.65 Pass 4 Response time (ms): 1396.60 Pass 5 Response time (ms): 1612.14	ID: 12 Pass 1 response time (ms): 1576.20 Pass 2 response time (ms): 56.16 Pass 3 response time (ms): 67.01 Pass 4 response time (ms): 119.19 Pass 5 response time (ms): 71.58
ID: 5451 Pass 1 Response time (ms): 1247.67 Pass 2 Response time (ms): 1840.82 Pass 3 Response time (ms): 1323.08 Pass 4 Response time (ms): 134.59 Pass 5 Response time (ms): 139.68	ID: 6437 Pass 1 response time (ms): 1249.70 Pass 2 response time (ms): 64.89 Pass 3 response time (ms): 83.43 Pass 4 response time (ms): 71.19 Pass 5 response time (ms): 66.39
ID: 10963 Pass 1 Response time (ms): 1288.64 Pass 2 Response time (ms): 1289.29 Pass 3 Response time (ms): 174.84 Pass 4 Response time (ms): 142.64 Pass 5 Response time (ms): 1339.69	ID: 11200 Pass 1 response time (ms): 1662.49 Pass 2 response time (ms): 62.95 Pass 3 response time (ms): 63.12 Pass 4 response time (ms): 61.12 Pass 5 response time (ms): 64.26
ID: 19369 Pass 1 Response time (ms): 1204.49 Pass 2 Response time (ms): 1430.94 Pass 3 Response time (ms): 1158.08 Pass 4 Response time (ms): 1266.66 Pass 5 Response time (ms): 147.81	ID: 17832 Pass 1 response time (ms): 1373.96 Pass 2 response time (ms): 62.04 Pass 3 response time (ms): 54.31 Pass 4 response time (ms): 63.13 Pass 5 response time (ms): 61.54

80k Files:

https://bafybeiectphf7uxuixy37f7pun4tzemlot57yr5fzonf4zbbrsqs6teghu.ipfs.w3s.link

Python	Go
ID: 451 Pass 1 Response time (ms): 2796.29 Pass 2 Response time (ms): 2407.41 Pass 3 Response time (ms): 3120.40 Pass 4 Response time (ms): 2171.77 Pass 5 Response time (ms): 160.17	ID: 45 Pass 1 response time (ms): 2361.79 Pass 2 response time (ms): 69.27 Pass 3 response time (ms): 75.23 Pass 4 response time (ms): 73.04 Pass 5 response time (ms): 60.74
ID: 3355 Pass 1 Response time (ms): 2391.00 Pass 2 Response time (ms): 2879.95 Pass 3 Response time (ms): 4189.82 Pass 4 Response time (ms): 2534.01 Pass 5 Response time (ms): 2127.78	ID: 4439 Pass 1 response time (ms): 2366.38 Pass 2 response time (ms): 88.82 Pass 3 response time (ms): 63.38 Pass 4 response time (ms): 92.42 Pass 5 response time (ms): 73.43
ID: 32685 Pass 1 Response time (ms): 2259.17 Pass 2 Response time (ms): 2535.59 Pass 3 Response time (ms): 148.88 Pass 4 Response time (ms): 2516.60 Pass 5 Response time (ms): 2411.75	ID: 37054 Pass 1 response time (ms): 2302.56 Pass 2 response time (ms): 72.01 Pass 3 response time (ms): 59.67 Pass 4 response time (ms): 62.51 Pass 5 response time (ms): 60.53
ID: 78456 Pass 1 Response time (ms): 3874.22 Pass 2 Response time (ms): 180.87 Pass 3 Response time (ms): 2475.43 Pass 4 Response time (ms): 2503.20 Pass 5 Response time (ms): 2470.15	ID: 78889 Pass 1 response time (ms): 2202.26 Pass 2 response time (ms): 66.06 Pass 3 response time (ms): 56.21 Pass 4 response time (ms): 83.01 Pass 5 response time (ms): 62.28

120k FIles:

https://bafybeigz3zdhx74vysngkifsdk5pxg2wiklghhb6gxj27hwfw2gmptegzi.ipfs.w3s.link

Python	Go
ID: 231 Pass 1 Response time (ms): 3343.42 Pass 2 Response time (ms): 4495.43 Pass 3 Response time (ms): 3576.80 Pass 4 Response time (ms): 3607.71 Pass 5 Response time (ms): 144.23	ID: 212 Pass 1 response time (ms): 6013.06 Pass 2 response time (ms): 130.51 Pass 3 response time (ms): 83.33 Pass 4 response time (ms): 103.69 Pass 5 response time (ms): 96.47
ID: 44787 Pass 1 Response time (ms): 3162.69 Pass 2 Response time (ms): 3512.06 Pass 3 Response time (ms): 6950.55 Pass 4 Response time (ms): 160.61 Pass 5 Response time (ms): 145.77	ID: 41511 Pass 1 response time (ms): 6018.40 Pass 2 response time (ms): 100.40 Pass 3 response time (ms): 86.73 Pass 4 response time (ms): 109.94 Pass 5 response time (ms): 79.38
ID: 59177 Pass 1 Response time (ms): 4183.35 Pass 2 Response time (ms): 3316.03 Pass 3 Response time (ms): 3263.45 Pass 4 Response time (ms): 152.75 Pass 5 Response time (ms): 4634.48	ID: 52777 Pass 1 response time (ms): 6058.90 Pass 2 response time (ms): 131.21 Pass 3 response time (ms): 85.33 Pass 4 response time (ms): 91.24 Pass 5 response time (ms): 88.06
ID: 113778 Pass 1 Response time (ms): 5567.19 Pass 2 Response time (ms): 3405.10 Pass 3 Response time (ms): 166.13 Pass 4 Response time (ms): 3138.76 Pass 5 Response time (ms): 3096.38	ID: 118622 Pass 1 response time (ms): 6788.29 Pass 2 response time (ms): 100.22 Pass 3 response time (ms): 91.80 Pass 4 response time (ms): 89.20 Pass 5 response time (ms): 89.02

200k FIles:

https://bafybeielxzakou2npi4fuo2tlgjqximwwarm32xawdwqmlaa25epj2rxse.ipfs.w3s.link

Python	Go
ID: 149 Pass 1 response time (ms): 3819.71 Pass 2 response time (ms): 72.17 Pass 3 response time (ms): 81.43 Pass 4 response time (ms): 57.45 Pass 5 response time (ms): 55.57	ID: 1933 Pass 1 response time (ms): 4330.20 Pass 2 response time (ms): 101.06 Pass 3 response time (ms): 82.42 Pass 4 response time (ms): 73.32 Pass 5 response time (ms): 61.44
ID: 44588 Pass 1 response time (ms): 3713.36 Pass 2 response time (ms): 73.03 Pass 3 response time (ms): 61.92 Pass 4 response time (ms): 87.81 Pass 5 response time (ms): 65.98	ID: 45611 Pass 1 response time (ms): 3748.37 Pass 2 response time (ms): 58.88 Pass 3 response time (ms): 68.77 Pass 4 response time (ms): 63.25 Pass 5 response time (ms): 55.53
ID: 57000 Pass 1 response time (ms): 3460.99 Pass 2 response time (ms): 63.54 Pass 3 response time (ms): 62.54 Pass 4 response time (ms): 76.98 Pass 5 response time (ms): 66.16	ID: 98229 Pass 1 response time (ms): 3480.62 Pass 2 response time (ms): 65.11 Pass 3 response time (ms): 73.67 Pass 4 response time (ms): 71.52 Pass 5 response time (ms): 60.11
ID: 111681 Pass 1 response time (ms): 6831.21 Pass 2 response time (ms): 73.00 Pass 3 response time (ms): 73.67 Pass 4 response time (ms): 65.52 Pass 5 response time (ms): 92.33	ID: 197907 Pass 1 response time (ms): 3382.29 Pass 2 response time (ms): 68.09 Pass 3 response time (ms): 68.14 Pass 4 response time (ms): 65.33 Pass 5 response time (ms): 65.43