IPFS and Merkle Forest

What is IPFS?

IPFS is short for Inter Planetary File System. It is a peer-to-peer, distributed file system to make the web faster, safer, and more open. To shift from present version of web to a distributed version of web, we need IPFS. Essentially, the aim is to replace HTTP.

But hey, why replace HTTP?

1. Crazy bandwidth cost: The present Web uses HTTP based on a single client-server model. One always has to approach the central server to download any kind of file. Imagine, if you could get bits of the same file from nodes near you? You can get the file downloaded faster and by using less bandwidth. With video delivery, a P2P approach could save 60% in bandwidth costs.

IPFS makes it possible to distribute high volumes of data with high efficiency. And zero duplication means savings in storage.

2. 404 is so freaking common! : The average lifespan of a web page is 100 days. After that, one can expect to see a 404 message. The present Web is so fragile. Links break all the time. It’s as good as burning books.

IPFS provides historic versioning (like git) and makes it simple to set up resilient networks for mirroring of data.

3. Centralised infrastructure, ugh: All the power of our data rests with the main server. If that breaks down, we are done. If Twitter breaks down, we can’t tweet anymore. If Facebook breaks down, well, it is already broken haha.

IPFS remains true to the original vision of the open and flat web, but delivers the technology which makes that vision a reality.

4. Offline is the new online: In developing countries, during natural disasters, interim bad network, what do we do? Just sit? The networks we’re using are so 20th Century. We can do better.

IPFS powers the creation of diversely resilient networks which enable persistent availability with or without Internet backbone connectivity.

How does IPFS work? (In simple terms)

So if you want to retrieve a data structure or a file is saved on the web using IPFS, you won’t hit the central server at all. You’d ask your peers in the network to give you a path to that file. Your peers will give you something known as a ‘cryptographic hash’ of that file. It is a unique fingerprint of that file.

Suppose you want to get /foo/bar/baz.png and its cryptographic hash is WmXGTaGWTx1uUtfSb2sBAvArMEVLK4rQEc4g5bv7wwdz1U. (This can be generated using SHA1, SHA2, or any other algorithm). You hit the web with this link.

Wikipedia has already started using IPFS:

Ana Sayfa

The URL is in the format: https://ipfs.io/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html

A bit about Merkle Trees

The research paper on Merkle link can be found here. Ralph Merkle is the brain behind Merkle data structure.

A beautiful illustration of what is a Merkle Tree can be found here.

Basic implementation of Merkle Tree in C++

#include <stdio.h>#include <stdlib.h>#include <iterator>#include <vector>using namespace std;// Hashing functions.                                                                                                                                                                                                                                                                                                                                                int multiplyThem(int a, int b) {  return a*b;}int addThem(int a, int b) {  return a+b;}class Merkle {private:  vector<int> values;  int (*hasher)(int, int);public:  Merkle(int (*f)(int,int)) {    this->hasher = f;  }  int size() {    return values.size();  }  void add(int value) {    values.push_back(value);  }  int root() {    vector<int> current;    current = getHashedParents(this->values);    while (current.size() != 1) {      current = getHashedParents(current);    }    return current[0];  }private:  vector<int> getHashedParents(const vector<int> &children) {    vector<int> result;    for (int i=0; i < children.size(); ) {      int a = children[i], b = children[i];      if (++i < children.size()) {        b = children[i++];      }      int hash = this->hasher(a,b);      printf("hash(%d, %d)=>%d ", a, b, hash);      result.push_back(hash);    }    printf("\n");    return result;  }};int main(int argc, char** argv) {    Merkle merkle(multiplyThem);    merkle.add(1);    merkle.add(2);    merkle.add(3);    merkle.add(4);    merkle.add(5);    printf("Merkle Root = %d\n\n", merkle.root());    merkle = Merkle(addThem);    merkle.add(1);    merkle.add(2);    merkle.add(3);    merkle.add(4);    merkle.add(5);    printf("Merkle Root = %d\n\n", merkle.root());    return 0;}

The heart of IPFS is IPLD.

IPLD is short for Inter Planetary Linked Data. The files/data structures are linked to each other using Merkle links.

(What is Merkle DAG? It is a Merkle directed acyclic graph. It is similar to Merkle tree. However, a Merkle DAG need not be balanced and its non-leaf nodes are allowed to contain data.)

In IPFS, links between two nodes are in form of cryptographic hashes. This is possible because of Merkle DAG data structure. Merkle DAGs provide IPFS many useful properties, including:

1. Content Addressing: All content is uniquely identified by its cryptographic hash, including links.
2. Tamper proof: All content is verified with its checksum. If data is tampered with or corrupted, IPFS detects it because the hash will change.
3. No duplication: All objects that hold the exact same content are equal (i.e. their hash value is equal), and only stored once.

Just by giving away the merkle root to someone, you can handover huge volume of data to that person. Because, a merkle root essentially holds signature of all blocks underneath it.

Interoperability of systems can also persist in a merkle forest, where each tree represents a separate merkle tree. In a forest, one tree can be bitcoin, one can be ethereum, one can be a regular SQL database. So to interchange information between these trees, these content based cryptographic hash functions are efficient. Rather than sending the entire file over, only hash is sent out. Imagine using Ethereum for some transaction and adding in a Git page within the transaction.

Presently this kind of a system is used by:

1. Bitcoin
2. Ethereum
3. Git
4. Bit Torrent

There are more, but these are the major ones.

Get started with IPFS right here right now!

Getting Started | IPFS Docs

It’s, again, a ground breaking technology which is pushing the web from Web2.0 to Web3.0

IPFS 👽 and Merkle Forest🌳 was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.