What is a metadata endpoint?
A metadata endpoint serves as the source of NFT metadata on the Immutable platform. When items are minted, our platform queries this endpoint to retrieve specific metadata for each item. Collection owners have the ability to update this metadata and request a refresh for a specific set of tokens.
IPFS (InterPlanetary File System) is a cost-effective solution, particularly for startups and small projects. It eliminates the need to build your own caching or API infrastructure by leveraging Public Gateways like Pinata, Web3.storage, and NFT.storage.
Understanding Metadata Endpoints
The metadata endpoint functions as a shared platform service responsible for retrieving asset metadata. Its requirements differ slightly from simply fetching the NFT image. One important aspect to note is that when retrieving assets from an endpoint, there is a 5-second timeout. This timeout ensures a stable service for all platform users. Additional platform limitations can be found in the documentation linked here [deep dive on Immutable metadata].
Deep dive on Immutable metadata: https://docs.x.immutable.com/docs/deep-dive-metadata/
Comparison of Popular Gateways:
Here is a comparison of some popular gateways:
Free Tier: Available
Features: JS SDK, REST API
Limitations: Maximum 500 pinned files, unable to upload directories with more than 5k-10 files, global rate limit of 15 requests per minute per CID, IP address rate limit of 200 requests per minute, does not support IPFS DAG feature.
For the latest features and limits, refer to the services' websites.
In our experiment, we uploaded four batches of metadata directories containing 20k, 80k, 120k, and 200k items to both web3.storage and nft.storage. We then queried the files using Python and GoLang to measure the retrieval time for metadata.
If you are interested in performing similar tests or exploring the upload and performance code, you can find the open-source code here.
Using IPFS as a metadata endpoint for large collections that require regular updates is not optimal. Performance is notably impacted when dealing with a large number of files in a directory. The data from our experiment showed that non-cached response times nearly doubled with each increase in batch size. Caching and distribution performance can vary, depending on the cache status of the queried node.
Fortunately, the 5-second timeout should suffice for most use cases, including larger collections, provided the data has been propagated on IPFS. For large directories, it is advisable to upload them well in advance to ensure sufficient time for full archiving on IPFS. Collections with 100k+ items may take up to 48 hours to fully archive for reasonable performance.
Among the gateways tested, NFT Storage proved to be the easiest to work with when using the NFT Up application. It uploaded the entire collection in approximately 30 minutes, although larger collections took longer to archive. Performance-wise, NFT Storage and Web3 Storage exhibited similar results.
We did not test Pinata due to its inability to support large directory uploads. Additionally, GoLang outperformed Python in terms of performance, and leveraging its retry functionality could yield even better results. However, further testing in the same environment as the crawler is required to validate this observation.