![](/style/images/good.png)
![](/style/images/bad.png)
State of IPFS in JS
source link: https://blog.ipfs.tech/state-of-ipfs-in-js/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
![neoserver,ios ssh client](/neoserver-download.png)
# š§āāļøĀ Purpose
JavaScript (and its typed variant TypeScript) is the most ubiquitous programming language with the largest community of developers. Our goal is to enable all those developers to reap the benefits of IPFS and do so productively.
This blog entry intends to give an update on what is happening with IPFS development in JS. Itās been a while since weāve done this, so thereās a lot to cover. This is the first update of more to come. It isnāt a full roadmap, but we aim to give clarity to some of the history of IPFS development in JS, decisions by some maintainers on what to do going forward, and ways you can help.
# šĀ Names and Terms
To help with this update, the following names and terms will be used to aid with clarity:
- Kubo (opens new window) ā This project was formerly known as go-ipfs. See here (opens new window) for more info.
- js-ipfs (opens new window) - This is the long-standing IPFS implementation written in JS. As described below, we will be deprecating it after Pomegranate is released. Weāre currently not planning to rename this implementation like we did with Kubo (opens new window) given its limited lifespan.
- Pomegranate (opens new window) - This is a to-be-created IPFS implementation in JS (opens new window) that is discussed below. The final name is TBD (to be determined), and you can track the naming effort here (opens new window). While it will use many of the underlying libraries of js-ipfs (e.g., js-libp2p (opens new window), js-ipfs-bitswap (opens new window)), it is a separate project with a different API.
- IPFS-in-JS - This refers broadly to the development of IPFS using the JavaScript and TypeScript languages. It doesnāt mean the ājs-ipfsā project or āPomegranateā.
- DelegateĀ nodes - These are nodes that expose theĀ
/api/v0/dht/*
Ā endpoints of the Kubo RPC API (opens new window)Ā for delegated routing. BecauseĀ js-ipfsĀ nodes donāt have the DHT enabled by default and wouldnāt make good DHT servers in browsers anyways, they need the help ofĀ delegateĀ nodes to resolve DHT queries. - Preload nodes - TheseĀ are nodes that expose theĀ
/api/v1/refs
endpoint of the Kubo RPC API (opens new window) which can be called so that the remote node will fetch CIDs (but not pin). This is necessary to ensure that blocks that are added in the browser areĀ preloadedĀ onto a long-running IPFS node so that itās made available to the rest of the network. Preload nodes garbage collect those blocks after a period.
# ā³Ā IPFS-in-JS Development History
The js-ipfs project started in 2014 and sought to mirror the functionality and API of Kubo (then called go-ipfs). This manifested in the concept of the JS Core API (opens new window), which was effectively a conversion of the Kubo RPC API (opens new window) into JavaScript. There were multiple implementations of this interface including:
- ipfs-core (opens new window): JavaScript native implementation of IPFS functionality.
- ipfs-http-client (opens new window): an implementation that delegates out with RPC HTTP calls, and thus could be used to control and interact with a Kubo daemon or js-ipfs daemon.
This effectively had Kubo and js-ipfsā development awkwardly interlinked. You would find support for certain functionality in Kubo that wasnāt supported in js-ipfs, and js-ipfs would often be held back in experimenting on a feature because the change wouldnāt get added to the JS Core API without it also being implemented in Kubo.
In addition to interlinking with the API and design choices of Kubo, the top-level js-ipfs project followed a similar project structure to Kubo of being a kitchen sink of interaction options with:
- Daemon to be able to run js-ipfs as a standalone process
- A CLI to the core implementation
- Two RPC client and server implementations:
- RPC-over-HTTP API that matched Kubo
- gRPC-over-websockets API for bidirectional streaming
- HTTP Gateway (opens new window) server (though it lagged behind Kubo and did not support the latest specs for things like subdomain isolation and verifiable block/CAR responses (opens new window))
This ākitchen sinkā aspect was also antithetical to the āJS wayā of being lean and modular. For example, by importing js-ipfs, you always get MFS, IPNS, etc. even if they donāt use them, which leads to bigger bundles and more dependencies with more attack vectors.
āā¦ You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.ā ā Joe Armstrong, creator of Erlang progamming language
This is unlike the more flexible js-libp2p model where you can extend the capabilities of the node by configuring additional modules.
# šĀ Historical IPFS JS Usage Patterns
Given the flexible nature and multiple runtime support of JavaScript, several usage patterns emerged over the years. With the development history above, the IPFS network and protocols were accessed using JavaScript primarily with the following approaches.
# Remote Control of Kubo
With this approach, a long-running Kubo node be controlled by ipfs-http-client (opens new window) via Kubo's RPC API.
Based on npm download metrics, this is by far the most popular approach with ~10x more npm downloads (opens new window).
# š Pros
- Gives access to a full-featured IPFS implementation in Kubo with active maintenance and feature development.
# š Cons
- Requires bringing a long-running Kubo daemon into oneās architecture and potentially a reverse proxy like nginx for TLS termination. The Kubo daemon potentially becomes a black box to debug and tune for a JS development shop.
- Requires proper security to protect which Kubo RPC API endpoints are exposed to the public internet (opens new window).
# js-ipfs in a Node.js Context
With this approach, you either embed ipfs-core (opens new window) directly in your application by importing and instantiating the IPFS object or accessing a shared js-ipfs daemon. With the former approach, the Node.js process of the application would also open IPFS peer-to-peer connections.
# š Pros
- Having the full application stack in JavaScript makes for a simpler architecture and easier debugging.
# š Cons
- Historically lacking in some functionality that was only more recently added like a working DHT client and server.
- The API is large and hard to grasp. It contains many methods translated directly from the Kubo CLI, designed to be quick to type but often redundant.
# js-ipfs in a Browser Context
Because browsers impose constraints on the number of network connections opened and storage space used by each browser tab, they donāt make for reliable IPFS nodes. For this reason, many of the responsibilities of an IPFS node which include: routing, retrieval, and providing have all been handled in a delegated fashion in the browser.
Unfortunately, the default delegation setup in js-ipfs today is overly complicated and not well-engineered. Specifically, this happens via preload and delegate nodes hosted by Protocol Labs that support secure WebSockets (WSS) that browsers can safely and reliably connect to.
For delegated routing/retrieval, js-ipfs asks the delegate nodes to perform DHT operations on its behalf using the Kubo RPC API (/v0/dht/*
). The delegate nodes query the DHT for peers and content providers, and then they fetch the content, which they provide for the browser nodes that are connected to them.
For delegated providing from the browser, js-ipfs contacts the preload node using Kubo RPC API (/v0/refs
) to preload the entire dag from the browser. The preload node fetches the data from the browser over Bitswap and then advertises it on the DHT.
Note: Preloading could be understood as ephemeral pinning. The CIDs' data is fetched by a preload node for a short period during which it is available to the IPFS network, and after which it is garbage collected.
js-ipfs nodes running in the browser can connect to other js-ipfs nodes via the WebRTC Star transport, but this requires a centralized signaling server (opens new window) to do the initial handshake and cannot be used to connect to Kubo nodes as support for this transport has never materialized.
# š Pros
- Itās useful and works for multi-user apps with small amounts of ephemeral data transmitted in real-time.
- Even though js-ipfs with WebRTC requires a centralized signaling server and is not the ideal long-term solution (opens new window), it enables direct browser-to-browser communication (after the initial SDP manifest (opens new window) is exchanged over signaling server).
# š Cons
- Availability and performance are heavily dependent on non-trivial centralized infrastructure:
- If PL shuts down the hardcoded HTTP endpoints at
preload.ipfs.io
(opens new window), js-ipfs delegation is dead, unless someone also runs their own Kubo instance somewhere in the cloud with a TLS cert for HTTPS and sets it up as a preload in js-ipfs config. This is a non-trivial ask and a steep barrier to adoption. - This was designed as a temporary "hack" anticipating a proper libp2p protocol. Unfortunately, years later weāre still doing the original Kubo RPC
/v0/refs
hack.
- If PL shuts down the hardcoded HTTP endpoints at
- Inefficient design:
- It doesnāt scale well because certain preload nodes get disproportionate traffic due to hardcoding in old js-ipfs versions.
- Preload via
/v0/refs
is very wasteful (opens new window).
- Not specified or adopted by other implementations: the WebRTC Star transport is only implemented in js-libp2p and neither WebRTC Star nor the delegate/preload APIs are specified.
# š§āš»Ā IPFS-in-JS Development the Last 18 Months
The last 18 months on the JS front have been spent on:
- General maintenance of the project including mostly maintaining parity between the JS Core API with Kubo.
- Modernizing js-ipfs and js-libp2p to TypeScript and ESM only. š
- Improving security and performance of js-libp2p given its criticality for other projects outside of IPFS such as Ethereum Lodestar. For example, a set of DoS and eclipse attack mitigations (opens new window) were added.
- Adding a working DHT client/server implementation to js-libp2p. (Yes, in a NodeJS context you can read from and write to the public IPFS DHT.)
- Expanding connectivity options of IPFS-in-JS implementations. Notably, we introduced a WebTransport transport (opens new window) and are adding a new WebRTC transport (opens new window) that does not require a centralized signaling server to enable both browser-based IPFS nodes to dial Kubo nodes directly and browser-to-browser connectivity. (This is discussed more below.)
At this point, we have a solid peer-to-peer library in js-libp2p that can be built upon, and many lessons learned for how to better progress with Javascript/Typescript in a world of multiple IPFS implementations (opens new window).
# š” The Future of IPFS-in-JS in 2022 and 2023
With all the background of IPFS-in-JS over the past covered, this section will transition into our practical plans for 2022 and 2023.
# Go and JS Development are Decoupling
We never had full compatibility between Kubo and js-ipfs, we donāt think we can, and we donāt think itās worth investing more down this path. At least for the implementations maintained by Protocol Labsā EngRes (Engineering & Research) group (opens new window), the Go and JS implementations will diverge and develop the APIs that are best for their respective user bases.
In practical terms, this translates to:
- ipfs-http-client will remain the RPC-over-HTTP API for controlling js-ipfs (you can also use ipfs-grpc-client over WebSockets). The current js-ipfs RPC APIs will be maintained until js-ipfs support ceases (discussed below).
- With investment in Pomegranate, a new RPC API for Pomegranate will emerge.
- We wonāt test that ipfs-http-client has compatibility with recent versions of Kubo.
- If you want to control a Kubo node via JS, use the Kubo-specific library js-kubo-rpc-client (opens new window) (api).
# Seize and Leverage New Browser-Friendly P2P Transports
We are transitioning to a world where browsers can connect to other libp2p nodes (including other browsers) without Central Authority TLS certs thanks to new transports like WebTransport and WebRTC. See connectivity.libp2p.io (opens new window) for more details.
This means browser nodes have more optionality to which long-running IPFS nodes they delegate routing, retrieval, and providing. For example, making DHT or Bitswap requests to other nodes on the network is now viable.
Note: browser nodes will still want to delegate providing content to a node with more longevity since the new transports wonāt stop browser nodes from disappearing from the network when the user closes a tab or puts their laptop to sleep.
Weāll lean into realizing these breakthroughs and remove the more convoluted mechanisms from the past that relied on the Kubo RPC API and preload nodes discussed in js-ipfs in a Browser context.
# Support Fully Speced Delegated Routing Protocols and Endpoints
While it will be possible from a connectivity perspective to make DHT queries from a browser, we expect various applications will want to still delegate out routing. Reframe (opens new window) is a protocol for delegated routing that other IPFS implementations like Kubo have implemented. While it currently uses HTTP as a transport, it is speced and not tied to the Kubo RPC API. If/when there is a speced protocol for ambient discovery of āLimited Delegated Routersā provided by libp2p, we will support that as well.
# PL Delegate and Preload Nodes Will Be Shutting Down
Given the new browser-friendly p2p transports discussed above, weāll shut down the complicated āsong-and-danceā with the legacy delegate/preload nodes and the Kubo RPC API described in js-ipfs in a Browser context. This yields a simpler setup for oneās application and removes centralized infrastructure.
For delegated routing, one can configure Reframe (opens new window) endpoints. When it comes to providing content from a browser node, it will be up to developers to account for user behavior like closing tabs or laptop lids. The general recommendation is to either run your own preload node or upload content explicitly to a pinning service for providing.
# Release Pomegranate in 2023
Pomegranate is the to-be-developed IPFS implementation with all that weāve learned over the last 8 years while leveraging what is available to us in JS runtime.
Some defining attributes include:
Web-first isomorphic API - run in browsers, electron, node, deno, bun, etc - no node.js APIs, only standard JavaScript (e.g. web streams, not node streams, Uint8Arrays not Buffers). Node APIs will only be considered for special cases like mDNS.
Leaner API not tied to the legacy ācore APIā concept - Pomegranate will not have API compatibility with js-ipfs. It will expose a more ergonomic JS-developer-first API than what we have with the js-ipfs ācore APIā that was heavily influenced by Kubo. (One can also create an adapter from the ācore APIā to Pomegranateās API if they want to drop Pomegranate to their existing application using js-ipfsā ipfs-core.)
ESM and TypeScript only - Thereās no more debate on the utility of these for JS development. Weāll adopt them from day one.
Leverage existing interplanetary libraries - While weāre moving away from the interface and composition in js-ipfs, weāre not abandoning the underlying layers like js-libp2p, js-bitswap, etc. Those libraries have received a lot of maintenance attention in the last 18 months (including TypeScript and ESM updates) and are battle-tested in production. We will depend on them in Pomegranate as well.
Unified file API - high-level commands that act like a filesystem and return CIDs. For example:
const dirCid = await ipfs.mkdir('/foo') const dirCid2 = await ipfs.touch(dirCid, 'file.txt') const dirCid3 = await ipfs.pipe([ 'file content', ipfs.open(dirCid2, 'file.txt'), ]) // or something const content = await ipfs.cat(dirCid3, 'file.txt') // could be a network request
Expose a block API for low-level IPLD operations.
Focus on the browser use case - We wonāt do anything that precludes operating in NodeJS or a cloud service worker, but by default, we will prioritize delivery paths that deliver browser functionality sooner. This is because the browser runtime is the unique runtime the Pomegranate IPFS implementation can enable that other implementations canāt. As a result, it means there arenāt plans to invest in things like a JS implementation of the HTTP Gateway spec. Weāll let other implementations like Kubo, Iroh, etc. pursue that use case.
Enable configurable levels of delegation. With routing, retrieval, and providing there will be varying levels of delegating from none (all handled by the local Pomegranate node) to full (all handled by HTTP Gateways (opens new window) and Pinning Services (opens new window)).
# Pause js-ipfs Maintenance Once Pomegrate Is Released
Shortly after you can add and cat files across the network with Pomegranate, PL EngRes (opens new window) will cease maintenance on js-ipfs. In the absence of an established group with a credible track record to take js-ipfs over, the community is welcome to fork js-ipfs and maintain the fork. (We want to avoid issues that can occur with casually giving away publishing rights.)
As discussed before, we are not ceasing support and development of many of the libraries that js-ipfs depends on like js-libp2p and js-bitswap. These projects will be actively maintained as core dependencies to Pomegranate and other projects.
# A New Name Is Coming
As outlined here (opens new window), Protocol Labs wants to make space for additional IPFS implementations to be made, including in JS. We want to make it clear that js-ipfs is not IPFS and that js-ipfs is not the IPFS implementation in JS. go-ipfs successfully made this transition earlier in 2022 with its minimal rename to Kubo (opens new window). We will certainly not make the same name-squatting mistake with a new implementation like Pomegranate. Details and plans will be shared here (opens new window) and in the IPFS forums (opens new window).
# Doc Updates Galore
From dedicated websites (opens new window), examples (opens new window), to official docs (opens new window), and courses (opens new window), many places will need updating in light of new names and implementations. This is going to be a sizable undertaking that hasnāt been scoped out yet. This will be tracked here (opens new window).
# šŗĀ Timeline
The timeline for enacting all of the above is still actively being figured out. Weāll be updating the proposed roadmap (opens new window).
# š¤Ā Ways You Can Help
- š³Ā Propose a name for the new āPomegranateā JS library (opens new window) and cast your votes.
- š£Ā Give feedback on the Pomegranate roadmap (opens new window). Let us know how youāre using js-ipfs now so we can see if/how your use case would be supported with Pomegranate in the future.
- š«Ā Join the team - weāre hiring and need more JavaScript and TypeScript developers who are eager to make the vision above a reality. Itās ideal if you have experience working at the protocol/bytes/streams level. Please learn more here (opens new window).
- āĀ Contribute - Open source contributors welcome. Have a great idea and need some funding? Consider a grant request (opens new window).
Thank you for reading and being on this journey to make IPFS exceptional in JS runtimes!
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK