The Ethereum ecosystem

So far we have mainly discussed what a node needs to do – processing and validating blocks and transactions and maintaining state. This by itself is not particularly useful – we need components to access nodes, manage accounts and balances and to develop, test and deploy smart contracts. There is an abundance of different tools and solutions around the Ethereum blockchain that do exactly this, and we will spend some time exploring this ecosystem today.

Clients

From a technical point of view, the Ethereum network is a peer-to-peer network, formed by individual nodes each of which runs a client software. This software will typically

• discover peers, i.e. other nodes in the network
• notify peers of new blocks that appear in the network, typically because a miner has generated them
• maintain the state of the blockchain
• process blocks and make updates to the state

In addition, clients will most likely offer an API which other applications can use to get information on the blockchain, to query balances, inspect blocks, send transactions and so forth. There are a couple of clients that do all this, here is a short list which is probably far from being complete.

• The Go-Ethereum (geth) client is the client maintained by the Ethereum foundation, and, as the name suggests, written in Go. This is the client that we will use for the later posts in this series, when we discuss how to start and run a client in a private development network or one of the official test networks. According to the Etherscan node tracker, more than 60% of the nodes in the Ethereum production network are running geth.
• Ganache is a client mostly intended for development and testing purposes. It is part of the Truffle framework for Ethereum development and written in JavaScript / Node.js. Ganache is also the development server running as part of the Brownie framework that we will discuss and use in a future post
• OpenEthereum, formerly known as Parity, is an Ethereum client written in Rust. It was started by Parity Technologies and later transitioned to the newly founded OpenEthereum project. It is the second-most popular client in the production network, accounting for roughly every fourth node
• Hyperledger Besu is an Ethereum client written in Java, which is developed by the Hyperledger project hosted by the Linux foundation and mainly targeting permissioned corporate networks

dApps and gateways

To build something useful, we need to allow applications to interact with the blockchain. As nodes offer an API, we can do this – our application can try to identify a node, connect to this node and use the API to talk to the blockchain.

In practice, this poses two problems. First, to connect to a node, you need to know its URL. Thus your application would somehow need access to a list of available nodes. If, however, you do not control the nodes, this list will be volatile, and it can be a considerable effort to keep this list up to date. If you do not want to run your own node, you can use one of the organisations that offer nodes as a service. I think of these services as gateways into the Ethereum network – they basically provide access to a node that your application can use and make sure that the node is stable and up and running. Examples for node providers are

• Infura which we will use in a later post in this series to access the test network
• Alchemy is another provider that also offers a free tier for developers
• Cloudflare also offers an Ethereum gateway

Be careful, not all providers offers all API requests, if in doubt consult the documentation on the respective web page.

The second problem that we have is to actually prepare and submit the API requests. Of course this can be done with custom built code that directly assembles the JSON RPC requests, but over time, libraries that encapsulate access to the API have been developed. One very popular libraries is web3.js which is a JavaScript library to access an Ethereum node. In addition to methods that correspond to methods of the JSON RPC API, it also contains a few helper functions for account and wallet management or the handling of smart contracts. We will use web3.js for our sample frontend in a later post in this series.

Being a huge fan of Python, I was of course looking for a Python library doing approximately the same thing, and in fact there is one, which, not really surprisingly, is called web3.py. We will use this as well throughout this series to demonstrate how we can interact with a smart contract from within a Python application or even deploy a smart contract.

We are now ready to discuss a vision called Web3 by its proponents (thus explaining the name of the libraries). The idea of Web3 is that instead of having centralized applications, running on servers controlled by single organisations, applications are fully distributed. Application logic is residing mostly in browsers (i.e. JavaScript applications) and smart contracts, and the frontend applications interact with the blockchain and smart contracts via nodes. In this model, code is executed either in your frontend or within the EVM, i.e. fully decentrally, and data is stored mainly in the blockchain (and in file storage services that we will discuss below). Therefore these applications are called dApps (decentralized applications). I will not discuss the pros and cons of this architecture, but simply wanted to make sure that you have heard the term. What we will develop throughout this series is, in fact, a dApp, consisting of a smart contract and a React frontend running in your browser.

Development environments

The diagram above displays a more or less complete runtime architecture. To develop frontends and smart contracts efficiently, however, you will typically want to leverage additional tools that allow you to write your contracts in a high-level language, compile them to bytecode, quickly spin up a test environment which a defined state and run tests of your smart contracts and your frontend. Over time, a large variety of development environments and frameworks have appeared to support this.

First, you will of course need to decide for a high-level language for your smart contracts. Essentially, there are two choices. The most common one is Solidity, which is a mixture of languages like Java, JavaScript or C++ with some object-oriented features like overloading, inheritance and interfaces. We will use Solidity throughout this series. A second choice is Vyper which is a bit more “Pythonic” and replaces the older Serpent, but even though I am a huge fan of Python, I have not really looked at it yet. Finally, there is LLL which uses a Lisp-like syntax and was one of the first languages for smart contract development, but does not seem to be used very often in practice any more.

Once you have decided which language and compiler to use, you will have to pick the actual development environment. Especially for first steps, the Remix IDE is an excellent choice. Remix is an full-fledged development environment running entirely in your browser. With Remix, you can edit source code, compile, deploy into a private development blockchain running in a JavaScript sandbox in your browser (or to a network of your choice) and test and debug your contracts. We will use Remix for our first smart contract to get things going quickly.

If you prefer to work locally and feel at home in the JavaScript ecosystem, you might want to take a look at the Truffle suite. Truffle allows you to compile and deploy your smart contracts using JS as a a scripting language or using a JS based interactive console. Truffle spins up its own integrated local development node and allows you to easily run unit tests. As I prefer Python over JavaScript for this sort of tasks, I will instead use Brownie which (apparently being heavily inspired by Truffle) offers a similar functionality for Python. Using Brownie, you can compile your contracts, start a console which will also bring up a local Ganache node automatically (or connect to an existing node), deploy your contracts and interact with them. Brownie can also be imported and used as a library which allows you to script things, and integrates nicely with pytest, so that you can use pytest and Brownie to easily write unit tests for your smart contracts and even create coverage reports.

Storage, data and analytics

Let us now look at a few services that are available in the Ethereum universe that more advanced applications might need. The first type of service that you will sooner or later need is a storage service. Storing data in the blockchain is expensive, and most data related to the blockchain, like a piece of digital art minted as a token, is not actually stored in the blockchain itself. Instead, the blockchain holds a reference to the image (and maybe a cryptographic hash) and the image itself is stored somewhere else.

Of course you could use any type of storage solution, maybe even your favorite cloud service provider. This, however, would mean to again rely on a centralized infrastructure. If you do not want this, you will have to go for a decentralized file storage platform. A popular choice is IPFS, which is a peer-to-peer network technology for storing files. If you store a file on IPFS, it is essentially cut into chunks, and each peer will store one or more of these chunks, a bit similar to BitTorrent. In addition, the location of the file in IPFS is referenced by the hash of the files content (the so-called CID). This implies that when the content changes, the hash changes, so that once stored, the file is effectively immutable, which is one of the reasons why this file storage solution is often used as part of blockchain-based applications. Note, however, that a node is free to drop your file at any time, so you might want to find a node which is ready to pin your content. Providers like filecoin.io offer similar decentralized file storage solutions.

In addition to storing data, the Ethereum network produces a huge amount of data every second – blocks, transaction, state updates, log and so forth. Several services exist that allow you to easily scan and explore that data. One of the best known services of this type is Etherscan. Using Etherscan, you can retrieve and analyze individual blocks and transactions, list all transactions related to a specific account, track token and retrieve various types of statistics. Etherscan also offers an API and other development services, like a registry for smart contracts. OpenSea, which is a market place for NFTs, also offers services to discover and display token. Here is how the NFT that we will actually build and deploy in the course of this series looks like in OpenSea.

Wallets

When you want to interact with the Ethereum blockchain, you will need an account and some piece of software that assembles and submits transactions. Wallets combine these two functionalities. They manage your accounts, allow you to inquire your balance and to submit transactions and can often also manage token.

Some wallets run in your browser, others are mobile apps or desktop applications. A very popular wallet that is available as a browser plugin is MetaMask. Be aware, however, that if you use MetaMask, your private keys will be stored in the browser storage and are therefore theoretically accessible by malicious JavaScript code. In addition, if you reset your profile, the data is lost, so be careful when using such a solution, think about security and have a solid backup strategy. MyEtherWallet is another popular choice.

Frameworks like web3.js also have some limited wallet functionality built into them, and the React frontend that we will build does actually use this library to offer some rudimentary functionalities that a wallet typically has.

This completes our short journey through the Ethereum ecosystem. Of course, this list is far from complete and constantly evolving, we have, for instance not talked about oracles like ChainLink, framework providers like OpenZeppelin or market places. The tools and services that we have covered are, however, sufficient to get us started, and in the next post, we will actually get our hands dirty for the first time and build our first smart contract in Solidity, switching from the slightly more theoretical discussion so far to a more hands-on style. See you!