Fun with Solidity and Brownie

For me, choosing the featured image for a post is often the hardest part of writing it, but today, the choice was clear and I could not resist. But back to business – today we will learn how Brownie can be used to compile smart contracts, deploy them to a test chain, interact with the contract and test it.

Installing Brownie

As Brownie comes as a Python3 package (eth-brownie), installing it is rather straightforward. The only dependency that you have to observe which is not handled by the packet manager is that to Ganache, which Brownie uses as built-in node. On Ubuntu 20.04, for instance, you would run

sudo apt-get update
sudo apt-get install python3-pip python3-dev npm
pip3 install eth-brownie
sudo npm install -g ganache-cli@6.12.1

Note that by default, Brownie will install itself in .local/bin in your home directory, so you will probably want to add this to your path.

export PATH=$PATH:$HOME/.local/bin

Setting up a Brownie project

To work correctly, Brownie expects to be executed at the root node of a directory tree that has a certain standard layout. To start from scratch, you can use the command brownie init to create such a tree (do not do this yet but read on) . Brownie will create the following directories.

  • contracts – this is where Brownie expects you to store all your smart contracts as Solidity source files
  • build – Brownie uses this directory to store the results of a compile
  • interfaces – this is similar to contracts, place any interface files that you want to use here (it will become clearer a bit later what an interface is)
  • reports – this directory is used by Brownie to store reports, for instance code coverage reports
  • scripts – used to store Python scripts, for instance for deployments
  • tests – this is where all the unit tests should go

As some of the items that Brownie maintains should not end up in my GitHub repository, I typically create a subdirectory in the repository that I add to the gitignore file, set up a project inside this subdirectory and then create symlinks to the contracts and tests that I actually want to use. If you want to follow this strategy, use the following commands to clone the repository for this series and set up the Brownie project.

git clone https://github.com/christianb93/nft-bootcamp
cd nft-bootcamp
mkdir tmp
cd tmp
brownie init
cd contracts
ln -s ../../contracts/* .
cd ../tests
ln -s ../../tests/* .
cd ..

Note that all further commands should be executed from the tmp directory, which is now the project root directory from Brownies point of view.

Compiling and deploying a contract

As our first step, let us try to compile our counter. Brownie will happily compile all contracts that are stored in the project when you run

brownie compile

The first time when you execute this command, you will find that Brownie actually downloads a copy of the Solidity compiler that it then invokes behind the scenes. By default, Brownie will not recompile contracts that have not changed, but you can force a recompile via the --all flag.

Once the compile has been done, let us enter the Brownie console. This is actually the tool which you mostly use to interact with Brownie. Essentially, the console is an interactive Python console with the additional functionality of Brownie built into it.

brownie console

The first thing that Brownie will do when you start the console is to look for a running Ethereum client on your machine. Brownie expects this client to sit at port 8545 on localhost (we will learn later how this can be changed). If no such client is found, it will automatically start Ganache and, once the client is up and running, you will see a prompt. Let us now deploy our contract. At the prompt, simply enter

counter = Counter.deploy({"from": accounts[0]});

There is a couple of comments that are in order. First, to make a deployment, we need to provide an account by which the deployment transaction that Brownie will create will be signed. As we will see in the next section, Ganache provides a set of standard accounts that we can uses for that purpose. Brownie stores those in the accounts array, and we simply select the first one.

Second, the Counter object that we reference here is an object that Brownie creates on the fly. In fact, Brownie will create such an object for every contract that it finds in the project directory, using the same name as that of the contract. This is a container and does not yet reference a deployed contract, but if offers a method to deploy a contract, and this method returns another object which is now instantiated and points to the newly deployed contract. Brownie will also add methods to a contract that correspond to the methods of the smart contract that it represents, so that we can simply invoke these methods to talk to the contract. In our case, running

dir(counter)

will show you that the newly created object has methods read and increment, corresponding to those of our contract. So to get the counter value, increment it by one and get the new value, we could simply do something like

# This should return zero
counter.read()
txn = counter.increment()
# This should now return one
counter.read()

Note that by default, Brownie uses the account that deployed the contract as the “from ” account of the resulting transaction. This – and other attributes of the transaction – can be modified by including a dictionary with the transaction attributes to be used as last parameter to the method, i.e. increment in our case.

It is also instructive to look at the transaction object that the second statement has created. To read the logs, for instance, you can use

txn.logs

This will again show you the typical fields of a log entry – the address, the data and the topics. To read the interpreted version of the logs, i.e. the events, use

txn.events

The transaction (which is actually the transaction receipt) has many more interesting fields, like the gas limit, the gas used, the gas price, the block number and even a full trace of the execution on the level of single instructions (which seems to be what Geth calls a basic trace). To get a nicely formatted and comprehensive overview over some of these fields, run

txn.info()

Accounts in Brownie

Accounts can be very confusing when working with Ethereum clients, and this is a good point in time to shed some light on this. Obviously, when you want to submit a transaction, you will have to get access to the private key of the sender at some point to be able to sign it. There are essentially three different approaches how this can be done. How exactly these approaches are called is not standardized, here is the terminoloy that I prefer to use.

First, an account can be node-managed. This simply means that the node (i.e. the Ethereum client running on the node) maintains a secret store somewhere, typically on disk, and stores the private keys in this secret store. Obviously, clients will usually encrypt the private key and use a password or passphrase for that purpose. How exactly this is done is not formally standardized, but both Geth and Ganache implement an additional API with the personal namespace (see here for the documentation), and also OpenEthereum offers such an API, albeit with slightly different methods. Using this API, a user can

  • create a new account which will then be added to the key store by the client
  • get a list of managed accounts
  • sign a transaction with a given account
  • import an account, for instance by specifying its private key
  • lock an account, which stops the client from using it
  • unlock an account, which allows the client to use it again for a certain period of time

When you submit a transaction to a client using the eth_sendTransaction API method, the client will scan the key store to see whether is has the private key for the requested sender on file. If yes, it is able to sign the transaction and submit it (see for instance the source code here).

Be very careful when using this mechanism. Even though this is great for development and testing environments, it implies that while the account is unlocked, everybody with access to the API can submit transactions on your behalf! In fact, there are systematic scans going on (see for instance this article) to detect unlocked accounts and exploit them, so do not say I have not warned you….

In Brownie, we can inspect the list of node-managed accounts (i.e. accounts managed by Ganache in our case) using either the accounts array or the corresponding API call.

web3.eth.get_accounts()
accounts

You will see the same list of ten accounts using both methods, with the difference that the accounts array contains objects that Brownie has built for you, while the first method only returns an array of strings.

Let us now turn to the second method – application-managed accounts. Here, the application that you use to access the blockchain (a program, a frontend or a tool like Brownie) is in charge of managing the accounts. It can do so by storing the accounts locally, protected again by some password, or in memory. When an application wants to send a transaction, it now has to sign the transaction using the private key, and would then use the eth_sendRawTransaction method to submit the signed transaction to the network.

Brownie supports this method as well. To illustrate this, enter the following sequence of commands in the Brownie console to create two new accounts, transfer 1000 Ether from one of the test accounts to the first of the newly created accounts and then prepare and sign a transaction that is transferring some Ether to the second account.

# Will spit out a mnemonic
me = accounts.add()
alice = accounts.add()
# Give me some Ether
accounts[0].transfer(to=me.address, amount=1000);
txn = {
  "from": me.address,
  "to": alice.address,
  "value": 10,
  "gas": 21000,
  "gasPrice": 0,
  "nonce": me.nonce
}
txn_signed = web3.eth.account.signTransaction(txn, me.private_key)
web3.eth.send_raw_transaction(txn_signed.rawTransaction)
alice.balance()

When you now run the accounts command again, you will find two new entries representing the two accounts that we have added. However, these entries are now of type LocalAccount, and web3.eth.get_accounts() will show that they have not been added to the node, but are managed by Brownie.

Note that Brownie will not store the accounts on disk if not being told to do so, but you can do this. By default, Brownie keeps each local account in a separate file. To save your account, enter

me.save("myAccount")

which will prompt you for a password and then store the account in a file called myAccount. When you now exit Brownie and start it again, you can load the account by running

me = accounts.load("myAccount")

You will then be prompted once more for the password, and assuming that you supply the correct password, the account will again be available.

More or less the same code would apply if you had chosen to go for the third method – user-managed accounts. In this approach, the private key is never stored by the application. The user is responsible for managing accounts, and only if a transaction is to be made, the private key is presented to the application, using a parameter or an input field. The application will never store the account or the private key (of course, it will have to reside in memory for some time), and the user has to enter the private key for every single transaction. We will see an example for this when we deploy a contract using Python and web3 in a later post.

Writing and running unit tests

Before closing this post, let us take a look at another nice feature of Brownie – unit testing. Brownie expects test to be located in the corresponding subdirectory and to start with the prefix “test_”. Within the file, Brownie will then look for functions prefixed with “test_” as well and run them as unit tests, using pytest.

Let us look at an example. In my repository, you will find a file test_Counter.py (which should already be symlinked into the tests directory of the Brownie project tree if you have followed the instructions above to initialize the directory tree). If you have ever used pytest before, this file contains a few things that will look familiar to you – there are test methods, and there some fixtures. Let us focus on those parts which are specific to the usage in combination with Brownie. First, there is the line

from brownie import Counter, accounts, exceptions;

This imports a few objects and makes them available, similar to the Brownie console. The most important one is the Counter object itself, which will allow us to deploy instances of the counter and test them. Next, we need access to a deployed version of the contract. This is handled by a fixture which uses the Counter object that we have just imported.

@pytest.fixture
def counter():
    return accounts[0].deploy(Counter);

Here, we use the deploy method of an Account in Brownie to deploy, which is an alternative way to specify the account from which the deployment will be done. We can now use this counter object as if we would be working in the console, i.e. we can invoke its methods to communicate with the underlying smart contract and check whether they behave as expected. We also have access to other features of Brownie, we can, for instance, inspect the transaction receipt that is returned by a method invocation that results in a transaction, and use it to get and verify events and logs.

Once you have written your unit tests, you want to run them. Again, this is easy – leave the console using Ctrl-D, make sure you are still in the root directory of the project tree (i.e. the tmp directory) and run

brownie test tests/test_Counter.py

As you will see, this brings up a local Ganache server and executes your tests using pytest, with the usual pytest output. But you can generate much more information – run

brownie test tests/test_Counter.py --coverage --gas

to receive the output below.

In addition to the test results, you see a gas report, detailing the gas usage of every invoked method, and a coverage report. To get more details on the code, you can use the Brownie GUI as described here. Note, however, that this requires the tkinter package to be installed (on Ubuntu, you will have to use sudo apt-get install python3-tk).

You can also simply run brownie test to execute all tests, but the repository does already contain tests for future blog entries, so this might be a bit confusing (but should hopefully work).

This completes our short introduction. You should now be able to deploy smart contracts and to interact with them. In the next post, we will do the same thing using plain Python and the web3 framework, which will also prepare us for the usage of the web3.js framework that we will need when building our frontend.

Writing a smart contract in Solidity using the Remix IDE

Today, we will actually write our first smart contract, compile and deploy it and discuss some of the features of the Solidity programming language. To be able to start without any installation, we will use the Remix IDE, which is a fully browser-based development environment for Ethereum smart contracts.

Getting started with the Remix IDE

Without further ado, let us go ahead and work on our first contract. Open your browser and point it to http://remix.ethereum.org. The initial load might take a few seconds, but once the load is complete, you should see a screen with the layout of a typical IDE. On the left hand side of the screen, there is navigation bar, with the items (from top to the bottom)

  • Home
  • Explorer
  • Compile
  • Deploy
  • Debug

Between that screen and the main screen, there is – initially, as we start in the explorer view – a directory tree. Remix will pre-populate this tree with some sample contracts and save them in your browser storage, so if you add, change or remove a file, close the browser and reopen it, your changes will still be visible. It is also possible to synchronize with a GitHub account or with the local machine. Finally, the bottom area of the screen is the terminal that provides some additional output and also allows you to interact directly with Remix using the JS API.

Initially, the contracts directory already contains three smart contracts, called 1_Storage.sol, 2_Owner.sol and 3_Ballot.sol. Delete all those files, and create a new file Counter.sol with the same content as this contract in my GitHub repository. Open the contract in the explorer.

Next, click on the “Compile” icon on the navigation bar (you will have to select the contract in the explorer first). Hit the button “Compile Counter.sol”. After some seconds, the compile should complete. Next, select the “Deploy” icon in the navigation bar. Stick to the standard selection of the environment (this is the built-in virtual machine that is running in your browser) and click “Deploy”. When the deployment succeeds, you should see an entry in the section “Deployed contracts”.

Expanding this entry should reveal two buttons called “increment” and “read”. These buttons allow you to invoke the methods of the contract that we have defined. First, click on “read” once to get the current value of the contract, which should be zero. Then, hit “increment” once and click “read” again.

If everything worked, reading after incrementiing should show that the value has increased by one. In the screenshot above, I have hit “increment” three times, so the return value of the read method is three (this is a bit confusing – the zero in front of the variable type is the index of the return value, in our case this is the first return value). Congratulations, we have just tested our first smart contract!

Structure of a smart contract

This is fun, but let us now go back and try to understand what we have actually done. For that purpose, let us go once through the source code. This is not meant to be a systematic introduction to Solidity, and you might want to consult the actually quite readable documentation in parallel, but should give you a first idea of how a contract looks like (we will in fact need and study a few more advanced features of the language as we progress). The first two lines are actually not executed, but more or less metadata.

// SPDX-License-Identifier: GPL-3.0

pragma solidity >=0.8.0 <0.9.0;

The first line is a comment. By convention, a Solidity contract contains a license identifier, I have chosen the GPL license for this example. The second line is a pragma. Like in C or C++, a pragma is an additional instruction for the compiler. In this case, we restrict the Solidity version to somewhere between 0.8 and 0.9 (there is a reason why I have chosen 0.8 as minimum version – this is the version of Solidity which introduced overflow checks, and our counter can of course overflow – at least theoretically, as you will probably not find the time to hit the increment button 2256 times…), and the compile will fail if its own version does not match.

The next block declares our actual contract. In Solidity, a contract is a bit similar to a class in other object-oriented languages. We give our contract a name (Counter) and, inside the curly braces, define its events, methods and attributes.

contract Counter {

    event Increment (
        address indexed sender,
        uint256 oldValue,
        uint256 newValue
    );

    uint256 private counter;

    function increment() public {
        emit Increment(msg.sender,counter,counter + 1);
        counter++;
    }

    function read() public view returns (uint256) {
        return counter;
    }
  
}

Let us ignore the event that we define for a second, we will discuss this in the next section. After the event declaration, we first declare an attribute counter. This will be a 256-bit integer, which is the native datatype of Solidity, i.e. a 32 byte number. We also declare the attribute as private, i.e. not visible for other contracts, but of course it is not really private – after all, everybody with access to a blockchain explorer can inspect the storage and look at its value.

Speaking of storage, Solidity distinguishes different types of storage. For some data types, like integers, there is a default storage type, for some other data types, you will have to add the storage type explicitly as an additional qualifier called the data location. The three storage types are

  • Memory – this is linear storage that is cleaned up and reinitialized with every new transaction. Thus you can use memory inside an invocation of your contract, but as soon as the invocation finishes, the data is lost
  • Storage – this instructs Solidity to place the variable in the storage part of the blockchain state of the contract address. Thus, variables which are located in storage are actually persisted in the blockchain. This is expensive, but the only way to store information across invocations
  • Calldata – this refers to read-only memory which is used to store parameters for invocations of your contract from an EOA or other contracts and can also be used for function parameters and return values

In our case, we do not have to explicitly define a data location, as our counter is a state variable (i.e. defined on the contract level) and, being an integer, it will automatically be stored in the contract storage. Next, we define two functions, i.e. methods of the contract. We first define the increment function, which does not accept any argument and does not return anything. Note the additional public keyword which declares that our method can be called from outside the contract. Inside the function, we first emit an event (more on this later), and then simply increment the counter by one.

The next function is the function to read out the counter value. This function does not have an argument either, but a return value of type uint256. The interesting part of this function is the additional keyword view. A function can be declared as a view if it does not alter the state (which, in our case, means that the function does not alter the counter value).

What is the point of this? Recall that a change of state can only happen as part of a transaction. Thus a Solidity function can usually only be called as part of a transaction that has an associated gas cost and is executed by all nodes while validating the corresponding transaction. For many cases, this is an enormous overhead. Suppose, for instance, that you build a contract that manages a coin or a token. You would probably not want to burn gas every time you simply want to read out a balance. Here, a view function comes to the rescue. As such a function does not alter the state, but essentially only reads and processes state, it can be executed locally, outside of a transaction, and the execution will not cost you any gas at all.

Invoking a smart contract

To better understand how a function marked as view differs from an ordinary function, it is helpful to understand how a smart contract is actually invoked by an application. There are actually two different ways how this can be done.

First, you can send a transaction to the contract address, using the method eth_sendTransaction or eth_sendRawTransaction of the JSON RPC API. To pass parameters to the contract, include them in the transaction data field. This field is also used to select the method of the smart contract you want to invoke. By convention, the first four bytes of the data field are filled with the hash value of the signature of the method that you want to invoke. Recall that the EVM bytecode does not know anything about methods or arguments, contract execution will always start at address zero. Therefore, the compiler will generate code that reads the data field to first determine the method that needs to invoked, then gets the parameters from the data field, copies them to memory and finally jumps to the code representing the correct method (we will dive deeper into this in a later post).

The transaction is then executed, thereby running the bytecode, incluced in a block and persisted. This is an asynchronous process, and therefore, a contract invocation done via a transaction cannot return a value to the caller. In addition, it consumes gas. These are two good reasons why an alternative is helpful.

This alternative is the eth_call method of the API. This method also executes the bytecode at the target address, but differs in an important point – it is only executed in the local node (i.e. the node to which you direct the API call). No transaction is generated, which implies that no state update can be done, but also that no gas is consumed. The execution is synchronous and can return a value.

Theoretically, you can also make a call to a method that modifies the state. This will again run locally and can be thought of as a simulation, without making any permanent changes to the state. Similary, you could use a transaction to read the counter value, but this would cost you gas and therefore Ether and not even allow you to consume the returned value, as a transaction produces a receipt which does not contain something like a return value. IDEs like Remix or tools like Brownie or Truffle use a description of the methods of a contract which is emitted by a compiler and known as the ABI to determine whether it makes sense to invoke something via a call or a transaction (we will see how this works a bit later when we use Python to compile and deploy a smart contract).

It is instructive to run the increment method in Remix and inspect the resulting transaction. You might already have noted that every time you hit the “increment” button, a new transaction is indicated in the terminal (the tab at the bottom of the screen). At the right hand side of a transaction, you should see a blue “Debug button” (this is also fun, but we resist the temptation for a moment) and a little arrow. Click on the arrow to expand the transaction.

What Remix shows you is actually a mix of data contained in the transaction and the transaction receipt. You see a few fields that we have already discussed in one of the previous posts like the “to” field, the gas limit and the value. You also see a field called “input” which is actually the transaction data and should be

0xd09de08a

This is a four-byte value, and using an online hash calculator (just google for “keccak online”), you can easily verify that these are the first four bytes of the keccak hash of the string “increment()”, i.e. the signature of the method that you invoke.

Logs, transaction receipts and events

It should now be rather obvious what our contract is doing, but we have skipped a point that is a bit more complex – events and logs. If you look at the transaction that we have generated using the increment method, you will see that it also contains a field called logs. The idea of logs is to provide a facility that allows a smart contract to log some data without having to write it into the expensive storage. For that purpose, the EVM offers a set of instructions called LOG0 to LOG4.

The logs of a transaction are part of the transaction receipt that is created by a client when a transaction is processed. As the transaction receipts can be derived from the history of all transactions, they are not actually stored on the blockchain itself (i.e. they are not part of a block), but are maintained by each node individually. A block header does, however, contain a hash of the transaction receipts, so that transaction receipts and therefore logs can be validated.

To structure logs, a log can be assigned to one or more topics. More precisely, to produce a log entry with no topic, you would use the LOG0 instruction, to produce a log entry with one topic, you would use LOG1 and so forth. Consequently, there can only be up to four topics for each log entry. In addition to the topics, a log entry contains a data field of arbitrary length, and the address of the contract that emitted the log. So a full entry consists of

  • the sender, i.e. the contract creating the log
  • between zero and four topics
  • a data field

In Solidity, logs show up as events. Events are defined as part of a contract, as it is the case in our example contract above. By convention, Solidity will always use the first topic to hold the signature of the event. The parameters of the event can be indexed (then Solidity stores them in the remaining topics), or non-indexed (then Solidity uses the data field to store them). Thus, in our example, the events would correspond to log entries where the sender is the contract, the first topic is the signature of the event, i.e. the keccak hash of the string “Increment(address,uint256,uint256)” (note that all white spaces and all parameter names have been removed), which is 0x64f50d594c2a739c7088f9fc6785e1934030e17b52f1a894baec61b98633a59f, the second topic is the sender and the data will contain the old and new value of the counter. You can inspect the transaction receipt and with it the log entries of the transaction that we have created by typing

web3.eth.getTransactionReceipt("0x35325d4aac18a862380ee6dc4a6b7ed1e8eb9c4035dc7d24b35ad847b19deadf")

into the console, where you have to replace the argument with the hash value of the generated transaction. The logs will appear as an array, and each entry consists of the address of the creating contract, the data, and the list of topics.

Why do we need topics? The idea of logs and events is that an application can register for certain events and take action if an event is observed. To make this work, we need an efficient way to scan the chain for specific events. Now going through all transactions and looking at all log entries is of course not an efficient approach, and we need a way to quickly determine whether a certain block has produced log entries which are of interest for us. For that purpose, Ethereum employs a structure known as Bloom filter, which is a data structure designed to quickly figure out whether a data set holds a specific entry. Each block contains the Bloom filter of the addresses and topics of the log entries produced by the transactions in the block, and thus a client can quickly scan all blocks for log entries that are associated with specific topics. The JSON RPC API allows you to define filters for new log entries, specifying the contract address, the block range and the topics, so that you can retrieve only those logs which are of interest for your application.

Today, we have actually written, compiled and executed our first smart contract, and we have managed to understood a bit how this works and what happens during contract execution. In the next post, I will show you how to do the same thing with an editor of your choice and the Brownie development environment.