GoSig BFT Consensus Protocol for Block chain Simplified

What is a BFT consensus protocol?

Any blockchain must organize the transactions in a linear order of the time of their execution, which may be different from their time of submission. Instead of trying to achieve consensus on individual transactions, it is more efficient to group them in blocks which are a finite ordered list of transactions. If a block is accepted as valid and a consensus has been achieved, that means the same thing as achieving a consensus on all of the included transactions in the mentioned order. Blocks are ordered in a sequence which is aptly named as a block-chain or simply blockchain. The chain has a beginning called the genesis block. Everyone starts off with the belief that the genesis block is valid.

 

The distance of any block from the genesis block in term of the number of blocks in-between is called the height of the block. The height of the genesis block can be set as zero. The height of the first block after that is one, and so on. The job of the blockchain protocol is to make sure that every honest party agrees on the block that is acceptable given any particular height.

 

In a BFT protocol, each participating party sends messages to other parties in a prescribed manner until they come to a conclusion about a certain candidate block. There is the requirement that once an honest party is able to decide a certain transaction to be valid and agreed upon, no other honest party would deny that. Each honest party confirms a consensus by making sure that a large majority of other parties (honest or dishonest) agree to the same block for that particular height. The consensus algorithm is run fresh for each height, one after another.

 

Each party needs to know the public key of the other parties to validate their signatures. Otherwise, one single dishonest party can send the P-messages on behalf of everyone to create a consensus of his choice.

Why use a BFT protocol?

Blockchain BFT protocols provide a better way of achieving block consensus than the proof of work used in Bitcoin.

 

A theorem from Fisher, Lynch and Paterson rules out any possibility of achieving a consensus with even one faulty process in a completely asynchronous setting. So, in order to have a consensus, we assume some form of weak synchrony. Weak synchrony implies that the processes have almost synchronized time and the network is connected for some time during which the messages are guaranteed to be delivered within a specified amount of time.

 

BFT protocols have the following advantages over proof of work algorithm used in Bitcoin –

  1. There are two properties that we need for consensus, safety and liveness.  Bitcoin guarantees liveness under any condition, but safety is guaranteed only when the network is connected. On the other hand, BFT provides safety under any condition and liveness when the network is connected.
  2. Proof of Work consumes a huge amount of computation resources. In contrast, BFT does not require such expensive computation.

The only drawback is that the chain must know all of the miner’s public keys. This can be resolved by doing a Proof of Stake algorithm along with the consensus. Currently out of the scope of this blog.

 

GoSig BFT protocol

GoSig is a BFT consensus protocol that provides security even in the public network.

In this blog, I will try to develop the protocol in steps to explain the concepts behind it.

 

As discussed in the previous section, the GoSig protocol must satisfy the 2 properties of consensus i.e Safety and Liveness.

  1. Safety: Whenever any honest user commits a block for a given height, no other honest user can commit a different block at the same height even when the attacker has full control of the network. This ensures that the commits are final and unchangeable. So once a transaction is included in a block committed by any honest user, he knows for sure that that transaction cannot be undone. Note that only the honest party himself can know that since any other party cannot know whether he is honest or not and a dishonest party can do anything including committing something for no good reason. The protocol needs to ensure this property even when the attacker has full control of the network and can create an arbitrary number of network partitions.
  2. Liveness: It should always be possible to commit a new block by all honest parties at the next height as long the network connection works, i.e. the attacker loses control of the network. Basically, this condition is about not getting into a deadlock. This is important for the chain to proceed. Of course, the parties cannot commit if the network does not forward their P messages, so this has to only work when the network is connected.

 

Step 1 – A first cut BFT algorithm:

We assume that 2 third users are honest. Every user is able to make a choice somehow or the other. There is a possibility of not all choices are visible to every honest user when the network is fragmented or due to delay in the network.

 

Based on the above assumptions, every honest user should automatically agree on a choice as long as the network is connected.

 

Let’s look at what happens when the network is broken. For every height, any honest party does the following.

 

  1. The honest party selects one from the possible choices for the next block and chooses one of them by some deterministic means.
  2. The honest party then broadcasts a signed P message for his chosen block and waits for the others to broadcast theirs.

When the honest user receives 2f+1 votes from other parties for any particular choice for the block, he commits this block as the chosen one.

Let us see if the above algorithm satisfies the ‘safety’ property. If any block receives 2f+1 P messages, at least f+1 of them must be from honest parties, because the maximum number of dishonest parties are the only f. These honest parties will never send P message for a different block. So the maximum number of P messages that can be produced for any other block must be (3f+1) – (f+1) = 2f which is not sufficient for a commit. So, if some block is committed for a certain height by an honest party, no other block can be committed by any honest party. So the first condition is satisfied. This is great!

However, it might happen that the honest parties accidentally chose more than one different candidates for the next block. For example f honest users may P message candidate_1 and f+1 users may end up P messaging candidate_2.  The attacker simply does not do anything and enjoys what’s going on the system. In this case, no choice of the block has sufficient P messages for commit, so the system is stuck. The second property of liveness has been violated even when the network is connected.

 

Step 2 – Network Deadlock

We need some way to get out of the above deadlock situation. The obvious idea is of course to restart the whole protocol over whenever there is such kind of a deadlock. The problem with this is its impossible to know when a deadlock has happened from just incoming P-messages.

 

Suppose we conclude that a deadlock has happened after receiving some s P-messages from different parties and the party still fails to commit.  Now, let’s say in some case f honest users may P message candidate_1 and f+1 users may end up P messaging candidate_2. This is obviously a deadlock and hence these 2f+1 messages must be able to detect it and hence s ≤ 2f+1.

 

Suppose, now in a different situation 2f (group A) honest users have P-messaged candidate_1 and 1 (group B) honest user did not P-message anything yet (because he is yet to receive the possible candidates). Now, the attacker can send a P-message candidate_2 to group A which would convince the group A to abort (it has 2f+1 P-messages and still cannot commit, and s ≤ 2f+1). After group A aborts this and restarts the protocol, suppose they all decide candidate_2 (maybe because they have come to know of candidate_2 now). So group A now has 2f P-messages for candidate_2. The attacker now provides 1 P-message for candidate_2 so that group A can now commit candidate_2.

 

Notice that all this time, group B has not been able to either abort or commit any block. So, now the attacker can replay the recording of the first round for group A (showing evidence of 2f P-messages from group A for candidate_1 from his saved messages collected from the round) and adds his own 1 P-message for candidate_1. This provides 2f+1 messages for candidate_1 to group B and hence group B commits candidate_1. In effect, group A committed candiate_2 and group B committed candidate_1 for the same height. This breaks our condition 1.

 

The easiest solution to this problem is to timeout around after some reasonable time if a commit cannot be achieved before that. As long as the clocks of each processor are more or less same, this should make sure everyone is in the same retry round. This assumption is a synchrony assumption called partial synchrony. We will go into more details about it later. For the time being, let us just assume that all nodes maintain the exact same time and message propagation time is zero for any message.

The next proposed solution is very simple.

  1. The honest party selects one from the possible choices for the next block and chooses one of them by some deterministic means.
  2. The honest party then broadcasts a signed P message for his chosen block and waits for the others to broadcast theirs.
  3. When the honest party receives 2f+1 P-votes from other parties for any particular choice for the block, he commits this block as the chosen one.
  4. When an honest party has passed a fixed amount of round time (say R), he forgets everything and restarts the process from step 1.

It solves our liveness problem, but the safety property is broken now. Suppose 2f+1 parties send P-messages for candidate_1. The messages reach only 1 honest party who commits candiate_1. The rest of the user waits until R time has passed and restarts the process where they can easily commit candidate_2 with one P-message from the attacker.

 

Step 3 – Adding tentative commit of TC messages:

 

To avoid both of the problems, we need to put one extra step of messages that we would call the tentative commit or TC messages. The idea is to separate P-messaging for a different block in the new round and actually committing a block in the previous round. Sometimes a user needs to observe the new round while he still did not decide anything for his current choice for the candidate block. Every honest user does the following in every round.

  1. The honest party selects one from the possible choices for the next block and chooses one of them by some deterministic means. If he has TC-ed a block in any previous round, he must P the same block again if that is one of the candidates, otherwise, he does not P anything.
  2. The honest party then broadcasts a signed P message for his chosen block and waits for the others to broadcast theirs.
  3. When the honest party receives 2f+1 P-votes from other parties for any particular choice for the block, he sends a TC message that contains a proof of all the P-messages. Basically, he copies all the signatures contained in each of the P-messages for the same candidate block. This is to ensure that a dishonest user cannot create a TC message without first receiving 2f+1 P messages for the same candidate. After this, he would send no more P messages for the higher round for a different block (but will do it if he sees the same block suggested by the leader).
  4. Whenever a user sees any TC message in a round, he immediately TC-messages the same block. This is easy, he simply copies the proof of P-messages and signs it.
  5. Whenever a user finds 2f+1 TC messages for the same block for the current round and the current height, he commits it.
  6. When an honest party has passed a fixed amount of round time (say R), he forgets everything and restarts the process from step 1. If the user has TC-ed a block in this round, he must keep proposing and P-voting the same block in later rounds until he TC-ed for a different block in a later round.

This was much more complicated than the previous one. This is mostly because it has a new step and lots of nuances. Let’s see how this satisfies the safety and the liveness properties.

Safety property: We need to prove that if any honest party commits a block, there would be no more commits at the same height. This ensures that there is only one possible commit for any height.

 

When an honest party commits a block, he must have observed 2f+1 TC messages for the same block. Since the attacker can only send f of them, f+1 TC messages must have come from honest users. These honest users will never P-vote for the same height for a different block. So, any further rounds can only have a maximum of 2f P-votes for any other block which would not even be sufficient for a TC message. Without any TC message for any other block, no honest user will commit any other block at any further round, proving our case. Two different commits cannot, of course, happen at the same block because of the fact that it is not possible to make TC message for two different candidates at the same height. If basically proves that even though honest parties can commit a block for the same height in different rounds, they would always commit the same block.

Liveness property: We need to make sure that no matter what happens in the network and message order and whatever the messages sent by the attacker, when the network recovers, all honest users will commit some block. We break it into the following cases.

  1. If no TC-messages are sent in any round before it is over, there is no change in the system. The whole process just restarts. This won’t go on forever as in every round there is a probability of this not happening.
  2. If there are some TC-messages created, 2f+1 P-messages must be present in this round supporting the same candidate, because the proof of 2f+1 P-messages is contained in the TC message. f+1 of them must be sent by honest parties. Hence the maximum number of P-votes for a different candidate in the same round is 2f which is not enough to create a TC-message. So, if a round has one TC message, it does not have any other supporting a different candidate. This means the TC messages in the same round always support the same candidate. This makes it okay to forward a TC-message whenever an honest party sees it.
  3. If the TC-ed users are honest, he would eventually be promoted to the leader when the network is connected, at which point he would P-message the same block and everyone will agree.
  4. Since an honest user always signs a TC-message that he sees in around, when the network is connected, every honest user will see such message and will send TC-message supporting the same block. Since there are at least 2f+1 honest parties and there is only one candidate that can get TC-messages, this candidate will get 2f+1 TC-messages when the network is connected.

Now that we have a way to achieve consensus, I would delve into how block proposals are created by any party.

 

Creating the block proposals:

 

Given that blocks can be proposed by anyone, we need to handle the following situations

  1. If the attacker is allowed to create lots of block at any point in time, the attacker will keep creating too many blocks to simply burden the system with too much processing. Such an attack is called a denial of service attack.
  2. If the attacker can create too many blocks in a round to consider, it would become improbable for all honest users to agree on which one to P-message, thus making the system go on forever without committing anything.

 

To tackle the first problem, we allow one party to suggest only one candidate per round, and that too only after a commit happened for a previous height. So, to suggest a candidate, any user has to first prove that there was a commit that happened in the previous round. Since a candidate will eventually but definitely get committed after 2f+1 TC-messages for the same has been broadcasted, we require that those TC-messages be presented while suggesting a new block for the next height. So, any valid new candidate suggestion for the height h must contain 2f+1 TC-messages.

 

To tackle the second problem, a small set of leaders are selected at random. This ensures that less number of blocks are proposed, thereby increasing the probability of commit. The random selection must be publicly verifiable to be random, otherwise, the attacker will simply choose random numbers that favour him and simply claim that he got those numbers randomly.

 

One way to get a publicly verifiable random number for any party would be to simply concatenate current height, current round and the party’s public key and take the hash of it. Since cryptographic hashes have uniform random distribution, this makes a good source of the random number. We say a party has been selected as a leader if his random number for the round is less than a certain value which is chosen based on the fraction of parties to be chosen as leaders per round on average. If the maximum value of a hash is N and the fraction of parties chosen as a leader be x, then the condition for selection would be H<x/N, where H is the random number for the given party (H=Hash(height||round||publickey)).

 

Although the above idea works, Jing Chen et. al., the creators of the Algorand algorithm had a better idea. They used a technique called a verifiable random number that has all the properties of what we had suggested, and it cannot be known to any other party before the owner of the private key discloses it. Instead of taking the hash of the concatenation of the height, round and public key, they start with the hash of the concatenation of height and round. They then use a unique signature algorithm to sign this hash using the party’s private key. A unique signature has no random element and there is only one valid signature for a single message. RSA is one such signature algorithm. Another is quite a new one called the BLS signature. They then sign the hash using the party’s private key. The random number H=Hash(S) where S=Signpry(Hash(height||round)) . The signature S is included with the random number as a proof so that it is publicly verifiable that the hash is correctly generated and the generator has no way to control it. Since the signature can only be generated by the owner of the private key, other parties cannot know about it until he discloses it.

 

The leader must include the signature as part of his candidate suggestion. This makes sure that the leader cannot be identified before he already made the suggestion. That way, he cannot be corrupted by the attacker beforehand.

 

To increase the likelihood of every honest user P-messaging the same candidate, we need to find a deterministic way to choose between multiple candidates. A simple rule can be that we choose the one with the minimum value of the verifiable random number in the first suggestion.

 

Now that we have a working BFT algorithm, I would like to dive into the details of network delay and clock delay.

Handling delays and timing tolerance:

 

Let T be the maximum difference between any two parties, S be the maximum network delay for receiving a message when the network is connected, R is the time for each round, M be the maximum processing time for each round. We consider the following under full network connectivity.

  1. When a round starts, the honest leaders wait for T time so that everyone is in the same round.
  2. After this, they broadcast their candidates by P-messaging them.
  3. It takes S time for everyone to get this message.
  4. Everyone then P-messages the correct choice.
  5. It takes S time for everyone to receive all P-messages.
  6. Everyone then TC-messages the block.
  7. It takes S time for everyone to receive the TC-messages and commit it.
  8. Let the maximum amount of time taken for processing be M

So in total, R must be greater than T+3S+M. This will always ensure a commit under full network connectivity while allowing the system to work fast.

 

Summary:

 

The correct protocol I have described here is the Gosig protocol proposed in the paper https://arxiv.org/pdf/1802.01315.pdf by Li, Wang and Chen (I changed a few little details, but they don’t matter). We went step by step into developing a working BFT protocol that can be used in a blockchain. There are quite a few others with somewhat different approaches, we do not cover them here.

Why BFT protocols cannot have 1/3 dishonest parties

Introduction:

Any blockchain must organize the transactions in a linear order of the time of their execution, which may be different from their time of submission. Instead of trying to achieve consensus on individual transactions, it is more efficient to group them in blocks which are a finite ordered list of transactions. If a block is accepted as valid and a consensus has been achieved, that means the same thing as achieving a consensus on all of the included transactions in the mentioned order. Blocks are ordered in a sequence which is aptly named as a block-chain or simply blockchain. The chain has a beginning called the genesis block. Everyone starts off with the belief that the genesis block is valid.

The distance of any block from the genesis block in term of the number of blocks in-between is called the height of the block. The height of the genesis block can be set as zero. The height of the first block after that is one, and so on. The job of the blockchain protocol is to make sure that every honest party agrees on the block that is acceptable given any particular height.

At the heart of any blockchain, is a system that needs to agree on the system state. When multiple actors with equal privilege need to maintain a set of records, they must agree on whether some record is part of such database. This is very straightforward in a perfectly connected system with only honest parties, but it gets complicated with unreliable network and dishonest parties trying to gain from the insecurities of the system. More specifically, since honest parties cannot guarantee communication with the other honest parties and neither can they tell between honest and dishonest parties, a trivial protocol will let a dishonest party convince two different honest parties two different states of the system that cannot be true together. For example, an attacker in Bitcoin may try to make an honest party believe that he paid 1 BTC to A and make another honest party believe that he paid 1 BTC to B, where he only had 1 BTC to begin with. Such a system would allow the dishonest party to receive services from both A and B each worth 1 BTC by spending only 1 BTC altogether. In blockchain terms, such a transaction is called a double spend and a blockchain would always intend to stop such attacks. Getting multiple honest parties to agree on the correct state of the system while some dishonest parties are present is called a consensus algorithm. We will look closely into a particular kind of consensus protocol.

Bitcoin and Ethereum achieve consensus by something called a proof of work. Basically, any party can declare himself to be a leader deciding which one of the several possible correct sets of transactions is valid by solving a cryptographic puzzle ensuring some processing has been done by such party. Since there is a large reward for such a party successfully being selected as the leader, it creates a competitive environment where several parties compete to be chosen as a leader. However, since a leader is not allowed to pack conflicting/invalid transactions in the block (a block is an ordered list of valid transactions proposed by the leader), this creates only one valid chain of transactions. However, this allows some temporary disagreement between different honest parties (also known as a temporary fork), which get resolved as time goes on. It is not difficult to come by resources explaining PoW based blockchain, the reader can simply read such articles to know more unless he is already familiar with such concepts. In this blog, we dive deep into a different kind of consensus algorithm – byzantine fault tolerance or BFT.

BFT Consensus

In a BFT or a Byzantine Fault Tolerant consensus protocol, each participating party sends messages to other parties in a prescribed manner until they come to a conclusion about a certain candidate block. There is the requirement that once an honest party is able to decide a certain transaction to be valid and agreed upon, no other honest party would deny that. Each honest party confirms a consensus by making sure that a large majority of other parties (honest or dishonest) agrees to the same block for that particular height. The consensus algorithm is run fresh for each height, one after another.

The basic idea is as follows –

For every height –

  1. Some parties create a candidate block by compiling an ordered list of transactions. How they choose such transactions and which parties are allowed to make suggestions would be discussed later.
  2. After an honest party receives a set of candidate blocks, he chooses one of them – (by some mechanism that would that would not be discussed in this blog). He then lets others know about his choices by sending a signed message. We will call such a message as a P-message.
  3. When an honest party receives a certain number of P-messages from other parties for a particular block (among all the choices), he goes on to say that he has achieved consensus on that particular choice of a block for the current height by what is called as committing the block.
  4. The protocol is then repeated for the next height.

Note that each party needs to know the public key of the other parties to validate their signatures. Otherwise, one single dishonest party can send the P-messages on behalf of everyone to create a consensus of his choice.

The system cannot of course work with an arbitrary number of dishonest parties, there must be a limit for the maximum number of dishonest parties the system can be protected against. In the following section, we will discuss this limit.

Maximum number of dishonest users for achieving consensus

Before we get to limits, we need to state the conditions the protocol needs to satisfy:

  1. Safety: Whenever any honest user commits a block for a given height, no other honest user can commit a different block at the same height even when the attacker has full control of the network. This ensures that the commits are final and unchangeable. So once a transaction is included in a block committed by any honest user, he knows for sure that that transaction cannot be undone. Note that only the honest party himself can know that since any other party cannot know whether he is honest or not and a dishonest party can do anything including committing something for no good reason. The protocol needs to ensure this property even when the attacker has full control of the network and can create an arbitrary number of network partitions.
  2. Liveness: It should always be possible to commit a new block by all honest parties at the next height as long the network connection works, i.e. the attacker loses control of the network. Basically, this condition is about not getting into a deadlock. This is important for the chain to proceed. Of course, the parties cannot commit if the network does not forward their P messages, so this has to only work when the network is connected.

That being said, we try to figure out the maximum number of dishonest user any system can support to achieve consensus. Let us assume that any honest user needs to see P messages for the same block for the given height from a t fraction of the total number of parties. That is if there are a total of N parties, the honest users would look for more than t/N P messages supporting the same block before they commit the block. Let us also assume that z is the maximum fraction of dishonest parties that the protocol can support.

We first see what is required to achieve condition 1. We already assumed that the attacker has full control of the network. We also assume that the dishonest parties make a coalition among themselves to form a combined attacker. The aim of the attacker is to make two different honest users commit two different blocks. The attacker partitions the network into three groups. One containing x/N honest users (we will call this partition A), one containing the rest of the honest users (partition B) and the last partition containing all the dishonest parties (partition C). This means the fraction of parties in partition B is (1-x-z). Note that the attacker can control the network and all delivery of messages, but cannot create a P message on behalf of an honest user. Now the attacker tries to make both of those partitions commit different blocks for the same height.

Since the attacker can have z fraction of P messages for both blocks, the total fraction of P messages the attacker can achieve for a block in partition A is x+z. Similarly, the total fraction of P messages the attacker can create in partition B is z+(1-x-z) = 1-x. Since any BFT protocol needs to make sure that conflicting blocks cannot be committed, at least one of them must be less than or equal to t.

Since the attacker can choose any value for x, it must be so that for any value of x either  x+z ≤ t  or 1-x ≤ t.

The reader can now note that the logical condition (A or B) to be true means either A has to be true or B has to be true. This means if A is false, then B must be true. This is the same thing as (not A) ⟹ B.

So, for all x, (x+z ≤ t  or 1-x ≤ t) is the same thing as saying for all x, x+z > t ⟹ 1-x ≤ t.

This can be simplified as  x+z > t ⟹ x+z ≥ 1-t+z  (1) .

When stated this way we can see that this condition would be met if t ≥ 1-t+z or 2t -1≥ z or z ≤ 2t-1 (2).

That would get what we want. But we have to prove that this condition is really necessary.

Suppose it is not true that z ≤ 2t-1; so, it must be true that z> 2t-1.

We will create a condition where x+z > t but x+z < 1-t+z  thus violating (1). The minimum possible value for x+z is of course z and maximum is 1. The attacker can choose any value for x such that x+z is in that range. Suppose,  m = max(z, t)  and the attacker chooses x = (m + 1-t+z)/2 -z so that x + z = (m + 1-t+z)/2 .

We have assumed that z > 2t-1 , so 1-2t + z > 0 or 1-t+z > t > 0 (3) . Hence, since m   =  max(z, t), x + z = (m + 1-t+z)/2 > z. This means such an x is admissible.

Also, 1-t+z > t (according to 3)  and since t<1  (as it a fraction), 1-t+z > z (4). (3) and (4)  together implies  1-t+z > max(z, t) =m or  m < 1-t+z  (5). Hence, x + z = (m + 1-t+z)/2 < ((1-t+z) + 1-t+z)/2 = 1-t+z.

Basically, since x + z  is the mean of  1-t+z  and m,  and since 1-t+z > m, it must be that m < x+z < 1-t+z . Observe that m=max(z,t) we have t  ≤  m < x+z < 1-t+z  which achieves the contradiction. Hence, we have proved that it has to be the case that z ≤ 2t-1 according to  (2).

Now, let us consider condition 2. It simply says that when the network is connected, the commit must be possible. So, the fraction of honest parties must be able to cause a commit. This means (x+(1-z-x)) = 1-z > t  or z < 1-t . The following diagram shows the allowable values for z.

The fraction of dishonest user is marked in green. The maximum possible value for z  is of course just below the intersection of the lines z = 1-t and z = 2t-1. We say ‘just below’ and not on the intersection because z < 1-t and so z cannot be on this line. If the intersection is at t=t0, we have 1-t0 = 2t0 -1 or 3t0 = 2 or t0 = ⅔ . Hence, z < 1-t0  = 1-⅔ = ⅓ . Hence the maximum fraction of dishonest users have to be less than ⅓ and that would be achieved when t= ⅔. So if there are f dishonest parties in the system, there must be at least 2f+1 honest parties, so that the total number of parties is 3f+1 which makes the number of dishonest parties strictly less

than the ⅓ fraction of the total number of parties. This also means the threshold here is floor(⅔(3f+1)) which is 2f+1 We will count the users in terms of  f from now on.

Summary: The red line in the above graph is the safety line. The z  below that can maintain the safety of the chain. The blue line is the liveness line. The z  below that can maintain liveness of the chain. Since both must be maintained, only the green area is allowed for the value of z. The maximum value of z  is, of course, the apex which is reached when z = ⅓. In fact, z must be less than for reasons described earlier.

EOS Demux using PostgreSQL

I have been looking for EOS demux implementation using PostgreSQL but found none. So I thought of writing this short tutorial explaining how we can implement EOS demux using PostgreSQL. Those who don’t know about EOS Demux please have a look at the article explaining the same in a very intuitive way.  https://github.com/EOSIO/demux-js.

 

A short explanation of EOS Demux: Demux architecture allows application developers to use traditional Mongo or Postgres SQL databases in a way that means the data stored in them is still verifiable by the blockchain. This enables the best of both worlds: the flexibility and speed of traditional databases, coupled with the trust and immutable properties of a blockchain. The overall flow of an application looks as follows:

 

To use PostgreSQL as a data store we need to use massive.js. So go ahead and install massive.js using the following command

> npm i massive --save
To demonstrate the working of PostgreSQL based demux implementation we are going to use a very basic contract. The contract is about a blog, using which user can do CRUD operation on the blog. As blog data is big we are going to store the blog data into PostgreSQL and metadata related to the blog like user, email etc. will be a part of the blockchain. Blockchain maintains all the information which
are required to make blog entry verifiable and secure. Following is the contract.
#include <eosiolib/eosio.hpp>

#include <eosiolib/multi_index.hpp>

#include <eosiolib/types.hpp>

#include <eosiolib/print.hpp>

#include <eosiolib/system.h>

//using namespace eosio;


class blog_contract : public eosio::contract{

    public:


    blog_contract(account_name self) : eosio::contract(self),

          bloges(self, self){}


        /// @abi action

        void create(account_name author, uint64_t id, std::string email, const std::string& data){

            require_auth(author);

            blogs.emplace(author, [&](auto& new_blog) {

                new_blog.id = id;

                new_blog.author = author;

            });

        }

        //.. destroy, show and update actions .. 


    private :

        /// @abi table blog i64

        struct blog{

            uint64_t id;

            uint64_t author;

            uint64_t primary_key() const {

                return id;

            }

            EOSLIB_SERIALIZE(blog,(id)(author));

        };

        typedef eosio::multi_index<N(blog), blog> blog_table;

        blog_table blogs;

};


EOSIO_ABI( blog_contract, (create)(destroy)(show)(update) )
If you look closely at ‘create’ function of the contract, you would notice that the function’s signature contains blog data as one of the parameters but we are not storing the blog data as part of EOS’ database. The purpose of having blog data as part of the function signature is that it gets available to be stored in PostgreSQL. Whatever we mentioned as function parameters, those parameter’s data will be available to the action handler. Looking at the diagram ‘Action watcher’ keeps looking for any change in blockchain via ‘Action Reader’. Whenever there is any action happened in the Blockchain, Action watcher delegate that action to ‘Action Handler’. Then ‘Action Handler’ further calls ‘updaters’ and ‘effects’ which actually execute necessary action code. Based upon the discussion the configuration looks as follows:
//First get the postgreSQL instance via massive as follows:
const massive = require('massive');

let db;

massive({
 host: host,
 port: port,
 database: database, 
 user: user,
 password: password 
}).then(instance => {
 db = instance;
 return Promise.resolve(db);
}).catch(e => {
 console.log(e)
 console.log('error while getting massive instance')
 });
Let’s define Updaters and Effects
//Updaters
function createBlog(db, payload, blockInfo, context){
 db.blog_data.insert({
 id : payload.data.id,
 author : payload.data.author,
 data : payload.data.data, //Storing blog data
 email : payload.data.email
 }).then(new_blog => {
 console.log('blog created')
 }).catch(err => {
 console.log('error while inserting data')
 })
}

//Register blockchain Actions to which this update will be called.

const updaters = [{
 actionType: "blogger.p::create",
 updater: createProfile}
]

module.exports = updaters
//Effects
function logUpdate(state, payload, blockInfo, context) {
 console.info(“blog created\n")
}
const effects = [
{
 actionType: "blogger.p::create",
 effect: logUpdate,
}
]

module.exports = effects
Now create MassiveActionHandler, Action Reader, and Action Watcher as follows:
const {
 readers: {
 eos: { NodeosActionReader }
 },
 handlers :{
 postgres : { MassiveActionHandler }
 },
 watchers: { BaseActionWatcher }
 } = require('demux-js')

const actionHandler = new MassiveActionHandler(
 updaters,
 effects,
 db
)

const actionReader = new NodeosActionReader(
 httpEndpoint,
 0, // Start at most recent blocks
)


const actionWatcher = new BaseActionWatcher(
 actionReader,
 actionHandler,
500,
)
actionWatcher.watch()
You also need to make sure the following table exists in PostgreSQL, which keeps track of block number up to which action has already been taken.
CREATE TABLE _index_state (
 id serial PRIMARY KEY,
 block_number integer NOT NULL,
 block_hash text NOT NULL,
 is_replay boolean NOT NULL
);

Hacking ethereum to inject our own consensus alogrithm Part-2

This is the second blog post for hacking Ethereum. If you have not gone through the first blog post please click here and have your local environment set up. The main entry point for Ethereum is /go-ethereum/cmd/geth/main.go

Open up main.go and look at ‘main’ method, which is the entry point whenever we execute ./geth binary.  This function basically executes the application with whatever command parameter we have supplied to it. As we know in ‘go’ there is specific function ‘init’ which gets executed before calling the module, let’s see what’s inside in initialisation code. The important line to notice is:

 

app.Action = geth

 

So default action if we don’t supply any via command line parameter is geth. ‘geth’ is a function inside main.go. This function is actually starting up all background services for a node.

 

func geth(ctx *cli.Context) error {

node := makeFullNode(ctx)

startNode(ctx, node)

node.Wait()

return nil

}

 

Let further analyse the function ‘makeFullNode(ctx)’. Go to its implementation, we see at 2nd line  “utils.RegisterEthService(stack, &cfg.Eth)”

It is registering “eth” service using utils. As we are running full node following is the code which actually gets executed:

 

func RegisterEthService(stack *node.Node, cfg *eth.Config) {

.....

err = stack.Register(func(ctx *node.ServiceContext) (node.Service, error) {

fullNode, err := eth.New(ctx, cfg)

if fullNode != nil && cfg.LightServ > 0 {

ls, _ := les.NewLesServer(fullNode, cfg)

fullNode.AddLesServer(ls)

}

return fullNode, err

})

.....

 

This function is registering new service to the stack which is basically a node. It means whenever we start a node this service will be executed. Let’s analyse the line fullNode, err := eth.New(ctx, cfg)

 

Going to the implementation of the ‘eth.New’ we go to ‘backend.go’ file. Looking at the constructor code, this service is nothing but Ethereum service. This constructor function configures the whole Ethereum. As our goal for this tutorial is to change the consensus algorithm following is the code of interest under ‘backend.go -> func New’

 

eth := &Ethereum{

config: config,

chainDb: chainDb,

chainConfig: chainConfig,

eventMux: ctx.EventMux,

accountManager: ctx.AccountManager,

engine: CreateConsensusEngine(ctx, &config.Ethash, chainConfig, chainDb),

shutdownChan: make(chan bool),

networkID: config.NetworkId,

gasPrice: config.GasPrice,

etherbase: config.Etherbase,

bloomRequests: make(chan chan *bloombits.Retrieval),

bloomIndexer: NewBloomIndexer(chainDb, params.BloomBitsBlocks),

}
So the code where we need to make changes is CreateConsensusEngine. Looking at the code it is clear that if we want to have our own consensus algorithm we need to implement ‘engine’ interface. Go to location ‘go-ethereum/consensus/‘ and created a new directory ‘myalgo’. Let’s make a very simple class (consensus algorithm) which will just print the message at various stages of consensus workflow. After that, we will extend it into the more advanced algorithm. For this we need to create two files ‘myalgo.go’ and ‘api.go’ under location ‘go-ethereum/consensus/myalgo’. You can look at those files from GitHub, the content which is important for us to discuss further is listed below:
myalgo.go
func (MyAlgo *MyAlgo) VerifyHeader(chain consensus.ChainReader, header *types.Header, seal bool) error {

log.Info("will verfiyHeader")

return nil

}
func (MyAlgo *MyAlgo) VerifyHeaders(chain consensus.ChainReader, headers []*types.Header, seals []bool) (chan<- struct{}, <-chan error){

log.Info("will verfiyHeaders")

abort := make(chan struct{})

results := make(chan error, len(headers))
go func() {

for _, header := range headers {

err := MyAlgo.VerifyHeader(chain, header, false)

select {

case <-abort:

return

case results <- err:

}

}

}()

return abort, results

}
func (MyAlgo *MyAlgo) VerifyUncles(chain consensus.ChainReader, block *types.Block) error {

log.Info("will verfiy uncles")

return nil

}
func (MyAlgo *MyAlgo) VerifySeal(chain consensus.ChainReader, header *types.Header) error{

log.Info("will verfiy VerifySeal")

return nil

}
func (MyAlgo *MyAlgo) Prepare(chain consensus.ChainReader, header *types.Header) error{

log.Info("will prepare the block")

parent := chain.GetHeader(header.ParentHash, header.Number.Uint64()-1)

if parent == nil {

return consensus.ErrUnknownAncestor

}
header.Difficulty = MyAlgo.CalcDifficulty(chain, header.Time.Uint64(), parent)

return nil

}
func (MyAlgo *MyAlgo) CalcDifficulty(chain consensus.ChainReader, time uint64, parent *types.Header) *big.Int {

return calcDifficultyHomestead(time, parent)

}
func (MyAlgo *MyAlgo) Finalize(chain consensus.ChainReader, header *types.Header, state *state.StateDB, txs []*types.Transaction,

uncles []*types.Header, receipts []*types.Receipt) (*types.Block, error){

log.Info("will Finalize the block")

header.Root = state.IntermediateRoot(chain.Config().IsEIP158(header.Number))

b := types.NewBlock(header, txs, uncles, receipts)

return b, nil

}
func (MyAlgo *MyAlgo) Seal(chain consensus.ChainReader, block *types.Block, stop <-chan struct{}) (*types.Block, error){

log.Info("will Seal the block")

//time.Sleep(15 * time.Second)

header := block.Header()

header.Nonce, header.MixDigest = getRequiredHeader()

return block.WithSeal(header), nil

}

As you can see most of them are just dummy function, always returning success result. In this algorithm, we are not actually finding any nonce, so after plugging this consensus algorithm it is expected that block is generated in an infinite loop without any delay. Let’s plug this algorithm and see what we are getting in logs. To plug this algorithm in Ethereum insert following code to “CreateConsensusEngine” function in ‘go-ethereum/eth/backend.go’ file.

 

if chainConfig.MyAlgo != nil{

fmt.Println(“myalgo is configured as consensus engine")

return myalgo.New(chainConfig.MyAlgo, db)

}

 

So this method basically checks if we have configured our configuration to use ‘MyAlgo’ or not. To enable ‘MyAlgo’ we need to make changes to our ‘privategensis.json’, Replace the content of the file with the following:

 

{

"config": {

"chainId": 15,

"homesteadBlock": 0,

"eip155Block": 0,

"eip158Block": 0,

"myalgo" : {}

},

"difficulty": "2000000",

"gasLimit": "21000000",

"alloc": {

}

}

 

Note the additional change “myalgo:{}” this line will enable our algorithm instead of default one. Delete previously create a directory from tutorial-1 and initialise Ethereum with new configuration:

 

> rm -rf ~/.ethereum/myprivatenet
> ./geth --datadir ~/.ethereum/myprivatenet init privategensis.json

 

Start the node as we did previously in the last tutorial. You will notice some additional logs and module after starting the node:

 

INFO [06-21|16:49:41.400938] Starting P2P networking

 

myalgo is configured as consensus engine

INFO [06-21|16:49:43.658541] UDP listener up self=enode://f08e5aa4fbde25b09ee977001af4a642abfd150c47e571ec5b69deed8c8c644c0a785d47a890adb6b2b55155cd32d7c035f4ec3ce7f8af54cd07219c8ad06ee7@[::]:30301

INFO [06-21|16:49:43.658717] RLPx listener up self=enode://f08e5aa4fbde25b09ee977001af4a642abfd150c47e571ec5b69deed8c8c644c0a785d47a890adb6b2b55155cd32d7c035f4ec3ce7f8af54cd07219c8ad06ee7@[::]:30301

INFO [06-21|16:49:43.661563] IPC endpoint opened url=/Users/hemants/.ethereum/myprivatenet/geth.ipc

INFO [06-21|16:49:43.662937] HTTP endpoint opened url=http://localhost:8101 cors=http://localhost:8000 vhosts=localhost

Welcome to the Geth JavaScript console!
instance: Geth/v1.8.12-unstable/darwin-amd64/go1.9.2
modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 myalgo:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

 

Log message: myalgo is configured as consensus engine, confirm that our configuration has been picked up and now Ethereum is configured with our own consensus algorithm. Also, a new module gets added to the console, it means we can interact with our consensus algorithm from the console. Great !!. This ‘myalgo’ got added due to the following two changes:

 

In consensus/myalgo/api.go:
type API struct {

chain consensus.ChainReader

myAlgo *MyAlgo

}

func (api *API) EchoNumber(ctx context.Context, number uint64) (uint64, error) {

fmt.Println("called echo number")

return number, nil

}
In internal/web3ext/web3ext.go:
var Modules = map[string]string{

"admin": Admin_JS,

...

"txpool": TxPool_JS,

"myalgo": MyAlgo_JS,

}

const MyAlgo_JS = `

web3._extend({

property: 'myalgo',

methods: [

new web3._extend.Method({

name: 'echoNumber',

call: 'myalgo_echoNumber',

params: 1,

inputFormatter: [null]

}),

]

})

 

This ‘echoNumber’ function is just echoing whatever we pass as an input parameter. This is to just show how we can add our own custom console modules.

 

> myalgo

{

echoNumber: function()

}

> myalgo.echoNumber(5)

called echo number

5

 

Now let’s start miner and see which function is being called from our consensus algorithm. As you start the miner you will notice that instantly it start mining blocks, here is the log:

 

> miner.start(1)

null

> INFO [06-21|17:01:52.43016] Transaction pool price threshold updated price=18000000000

INFO [06-21|17:01:52.43104] Etherbase automatically configured address=0x33954C10f5789A5914a09F0B1ef64CD26E0FA66E

INFO [06-21|17:01:52.43109] Starting mining operation

INFO [06-21|17:01:52.43113] will prepare the block

INFO [06-21|17:01:52.431516] will Finalize the block

INFO [06-21|17:01:52.431532] Commit new mining work number=1 txs=0 uncles=0 elapsed=406.763µs

INFO [06-21|17:01:52.431749] will Seal the block

INFO [06-21|17:01:52.431797] Successfully sealed new block number=1 hash=71810a…2882f3

INFO [06-21|17:01:52.436641] 🔨 mined potential block number=1 hash=71810a…2882f3

INFO [06-21|17:01:52.436681] will prepare the block

INFO [06-21|17:01:52.43676] will Finalize the block

INFO [06-21|17:01:52.436768] Commit new mining work number=2 txs=0 uncles=0 elapsed=90.323µs

INFO [06-21|17:01:52.436991] will Seal the block

INFO [06-21|17:01:52.437038] Successfully sealed new block number=2 hash=b6bef3…f2ed9b

INFO [06-21|17:01:52.437498] 🔨 mined potential block number=2 hash=b6bef3…f2ed9b

INFO [06-21|17:01:52.437641] Mining too far in the future wait=2s

INFO [06-21|17:01:54.441536] will prepare the block

INFO [06-21|17:01:54.441837] will Finalize the block

INFO [06-21|17:01:54.441891] Commit new mining work number=3 txs=0 uncles=0 elapsed=2.004s

INFO [06-21|17:01:54.442025] will Seal the block

INFO [06-21|17:01:54.442176] Successfully sealed new block number=3 hash=1d3448…e8ec87

INFO [06-21|17:01:54.442574] 🔨 mined potential block number=3 hash=1d3448…e8ec87

INFO [06-21|17:01:54.442616] will prepare the block

INFO [06-21|17:01:54.442739] will Finalize the block

INFO [06-21|17:01:54.442757] Commit new mining work number=4 txs=0 uncles=0 elapsed=144.138µs

INFO [06-21|17:01:54.442807] will Seal the block

 

Let’s check the height of blockchain :

 

> web3.eth.getBlockNumber(function(e,r){ console.log(r)})
17

 

Wow in a matter of seconds 17 blocks got generated. Now we have our own private Ethereum running locally with our own consensus algorithm, That’s amazing. Now we can implement logic into consensus algorithm.

 

Before we started implementing our own algorithm I would like to highlight one more insight of the code. Ethereum service (backend.go) contains a miner which orchestrate two very important module “worker” and “agent”.

The role of the worker is to generate blocks or work. It collects all pending transaction and makes them part of the block. Please note that here ‘generate block’ does not mean that it mined the block, it simply means we are initialising the block structure and doing some preliminary validations or simply it generate the work. Worker first calls engine’s Prepare function, in case of ethereum’s standard algorithm (ethash), block difficulty is calculated. Then it calls the engine’s Finalize method.

Role of an agent (cpuAgent) is to do work generated by the worker. So the agent actually mines the block, by finding correct nonce in case of POW. The communication between the worker and agent is done via event. When the worker generates a block it generates an event which agent is listening to. Once agent mine the block, it generate the event which worker is listening to and worker starts creating new work for an agent to work on. This loop continues.

For this tutorial, I am planning to implement a very basic consensus algorithm. The details of which are as follows :

 

At Miner side

 


– There is a file called problems.json, which contains 10 problems to be solved by the miner. These problems are a simple arithmetic equation like “3 + 14 * 49”.

– Miner, while sealing the block, will first select the problem from problems.json based upon index which will the first character of current block’s parent hash. So if current block’s parent hash is

“0x29c9f33844df1df9d808412104f08dd318c016f7f06faad241fb0f8c79911c75” then the index to pick problem will be ‘2’ ( ignore 0x which represents hexadecimal),

“0xc4fbd0b2c6bc524794fbff7af77eb0d98ad42aa4c6a34776bc66210502e908e2” then the index to pick problem will be ‘2′ ( c in decimal is 12, so 12%10 = 2)

– Once the miner finds the problem, he will try to solve the problem and once the solution is found he put the solution in ‘nounce’ field of the block.

 

At Validator side

 


– Whenever the validator node got a block, it first selects the problem from “problems.json” using block’s parent hash as described above.

– Validator computes the solution and compares its solution with the Nonce.

– If both are equal, validator put the block in his chain, else he rejects the block.

 

Run Ethereum Local cluster

 


To run multiple nodes we just need to specify different ‘data-dir’ and different ‘rpc-port’. So to initialise node 1

 

./geth --datadir ~/.ethereum/myprivatenet-1 init privategensis.json

To initialise node 2

 

./geth --datadir ~/.ethereum/myprivatenet-2 init privategensis.json

Similarly, you can initialise as many nodes as you want.

To start node, you need to mention previously created ‘data-dir’ and port. So to start node 1

 

./geth -rpc -rpcapi 'web3,eth,debug,personal'  -rpcport 8545 --rpccorsdomain '*' --datadir ~/.ethereum/myprivatenet-1 --networkid 15

To start node 2

 

./geth -rpc -rpcapi 'web3,eth,debug,personal'  -rpcport 8546 --rpccorsdomain '*' --datadir ~/.ethereum/myprivatenet-2 --networkid 15

As of now, these two nodes are running independently without knowing each other. So to connect node1 to node1, go to geth console and find node1 enode address by issuing following command:

 

node1 > admin.nodeInfo.enode

This will return enode address of the node1 like

 

node1 > "enode://c7a9acd7a6f381bb33a51767e07989939448fae4db2ec95cc736f88bebd39c6fce1e75b333225b435a0398ff7377805b08fb8792e6a0886d855ff315e54d5cf6@[::]:30303”

Go to node2 geth console and add node1 using its enode address as follows

 

node2 > admin.addPeer("enode://c7a9acd7a6f381bb33a51767e07989939448fae4db2ec95cc736f88bebd39c6fce1e75b333225b435a0398ff7377805b08fb8792e6a0886d855ff315e54d5cf6@[::]:30303”)

You can check if the connection is successful or not by issue ‘net’ command. This command will return the count of connected nodes.

 

> net

{

listening: true,

peerCount: 1,

version: "1",

getListening: function(callback),

getPeerCount: function(callback),

getVersion: function(callback)

}

 

The complete code is available here: Hope I am able to give you a taste of Ethereum internals and you must be equipped enough to hack it further.

Hacking ethereum to inject our own consensus algorithum Part-1

Recently, I have been going through the source of Ethereum to see if I can change it’s proof of work consensus algorithm with our own algorithm. By doing that I developed an insight of Ethereum source code which I would like to share. The goal of this blog post is to first setup Ethereum development environment and then build very simple consensus algorithm. The consensus algorithm will be very simple as main goal of this activity is to understand the internals of Ethereum. To follow along reader is expected to be equipped with following skills:

  • Understanding of ‘go’ language : As Ethereum (geth implementation) is written in go language it is expected that you should have good grasp of language. If you are not familiar with go language it is good motivation to learn now.
  • Understanding of how blockchain works : Understanding how blockchain works in general is required. There are quite amazing blogs which explain the working of blockchain.
  • Understanding of Ethereum : You cannot hack the application if you don’t know how it works at high level. So understanding of Ethereum at high level is required.

Good development set up makes life easier for making changes to the source code. I have been using ‘goLang’ from jetBrains for a year now. It is very good IDE for go development and have very good debug support. I also use Atom IDE for small projects but for dealing big projects goLang is amazing. Following are the step you need to perform to setup development environment:

  • Make sure go is setup properly. To check if it installed on your machine or not type following command in terminal:

> go version

go version go1.9.2 darwin/amd64

If command responds with proper version, you have go installed. If it is not, install command will throw an error “ -bash: go: command not found”.

To install ‘go’, follow instruction specific to your OS platform listed here : https://golang.org/doc/install

  • Make sure GOPATH environment variable is properly set up
 > echo $GOPATH
 /Users/hemants/Projects/mist/go-workspace

/Users/hemants/Projects/mist/go-workspace

  • Clone Ethereum GitHub repository
> cd $GOPATH/src/github.com/ 
> mkdir ethereum
> git clone git@github.com:ethereum/go-ethereum.git
  • Import Ethereum code into GoLand IDE by clicking on ‘open project’ button and locating the directory where you have checked out Ethereum code.
  • Once imported successfully run to file go-ethereum/cmd/geth/main.go, right click and run. This will report following errors:
cmd/geth/main.go:125:3: undefined: configFileFlag
cmd/geth/main.go:156:3: undefined: initCommand
cmd/geth/main.go:157:3: undefined: importCommand
cmd/geth/main.go:158:3: undefined: exportCommand
cmd/geth/main.go:159:3: undefined: importPreimagesCommand
cmd/geth/main.go:160:3: undefined: exportPreimagesCommand
cmd/geth/main.go:161:3: undefined: copydbCommand
cmd/geth/main.go:162:3: undefined: removedbCommand
cmd/geth/main.go:163:3: undefined: dumpCommand
cmd/geth/main.go:165:3: undefined: monitorCommand
cmd/geth/main.go:165:3: too many errors
  • To resolve above mentioned problem go to Run > Edit Configurations > Go Applications > select the run configuration you want to edit > Run kind and change it to File from Package. Then type the name of the package, github.com/ethereum/go-ethereum/cmd/geth and save the settings.

Our development environment is ready. Congrats !! :).

Set up local private Ethereum blockchain

To set up private Ethereum blockchain running locally, let’s first install ‘geth’ tool into $GOPATH/bin directory. To do that, open terminal and go to location $GOPATH/src/github.com/ethereum/go-ethereum, then execute following command from terminal

> go install -v ./cmd/geth

Go to location $GOPATH/bin/ to test if geth is installed properly or not. Issue following command, you should see following result.

> ./geth version
Geth

Version: 1.8.12-unstable

Architecture: amd64

Protocol Versions: [63 62]

Network Id: 1

Go Version: go1.9.2

Operating System: darwin

GOPATH=/Users/hemants/Projects/mist/go-workspace

GOROOT=/usr/local/Cellar/go/1.9.2/libexec

Next, create file name ‘privategensis.json’ with following content :

{

"config": {

"chainId": 15,

"homesteadBlock": 0,

"eip155Block": 0,

"eip158Block": 0

},

"difficulty": "2000000",

"gasLimit": "21000000",

"alloc": {

}

}

Go $GOPATH/bin and initialise local Ethereum blockchain by issue following command:

> ./geth --datadir ~/.ethereum/myprivatenet init privategensis.json
INFO [06-21|13:43:05.226227] Maximum peer count ETH=25 LES=0 total=25

INFO [06-21|13:43:05.240084] Allocated cache and file handles database=/Users/hemants/.ethereum/myprivatenet/geth/chaindata cache=16 handles=16

INFO [06-21|13:43:05.244943] Writing custom genesis block

INFO [06-21|13:43:05.245018] Persisted trie from memory database nodes=0 size=0.00B time=10.217µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B

INFO [06-21|13:43:05.24539] Successfully wrote genesis state database=chaindata hash=07185f…82bcc4

INFO [06-21|13:43:05.245414] Allocated cache and file handles database=/Users/hemants/.ethereum/myprivatenet/geth/lightchaindata cache=16 handles=16

INFO [06-21|13:43:05.24758] Writing custom genesis block

INFO [06-21|13:43:05.247618] Persisted trie from memory database nodes=0 size=0.00B time=2.699µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B

INFO [06-21|13:43:05.247787] Successfully wrote genesis state database=lightchaindata hash=07185f…82bcc4

This will generate the genesis block for our private blockchain at location : “/.ethereum/myprivatenet”. Lets run our blockchain by issuing following command :

./geth -rpc -rpcapi 'web3,eth,debug,personal' -rpcport 8545 --rpccorsdomain '*' --datadir ~/.ethereum/myprivatenet --networkid 15
INFO [06-21|13:49:27.007881] Maximum peer count ETH=25 LES=0 total=25

INFO [06-21|13:49:27.01546] Starting peer-to-peer node instance=Geth/v1.8.12-unstable/darwin-amd64/go1.9.2

INFO [06-21|13:49:27.015503] Allocated cache and file handles database=/Users/hemants/.ethereum/myprivatenet/geth/chaindata cache=768 handles=128

INFO [06-21|13:49:27.028411] Initialised chain configuration config="{ChainID: 15 Homestead: 0 DAO:  DAOSupport: false EIP150:  EIP155: 0 EIP158: 0 Byzantium:  Constantinople:  Engine: unknown}"

INFO [06-21|13:49:27.028465] Disk storage enabled for ethash caches dir=/Users/hemants/.ethereum/myprivatenet/geth/ethash count=3

INFO [06-21|13:49:27.028479] Disk storage enabled for ethash DAGs dir=/Users/hemants/.ethash count=2

INFO [06-21|13:49:27.028513] Initialising Ethereum protocol versions="[63 62]" network=16

INFO [06-21|13:49:27.029799] Loaded most recent local header number=0 hash=07185f…82bcc4 td=2000000

INFO [06-21|13:49:27.029836] Loaded most recent local full block number=0 hash=07185f…82bcc4 td=2000000

INFO [06-21|13:49:27.029845] Loaded most recent local fast block number=0 hash=07185f…82bcc4 td=2000000

INFO [06-21|13:49:27.03009] Loaded local transaction journal transactions=0 dropped=0

INFO [06-21|13:49:27.030331] Regenerated local transaction journal transactions=0 accounts=0

INFO [06-21|13:49:27.030806] Starting P2P networking

INFO [06-21|13:49:29.146101] UDP listener up self=enode://1dd1494242ee403a69fbb58a57505056f5fea5c9f4b207050ca0a1aecc36d74b4687ec1779320c6282d044aae0078437e5aa7688c29f04e1281cc3f5a0f954e2@[::]:30303

INFO [06-21|13:49:29.146426] RLPx listener up self=enode://1dd1494242ee403a69fbb58a57505056f5fea5c9f4b207050ca0a1aecc36d74b4687ec1779320c6282d044aae0078437e5aa7688c29f04e1281cc3f5a0f954e2@[::]:30303

INFO [06-21|13:49:29.150067] IPC endpoint opened url=/Users/hemants/.ethereum/myprivatenet/geth.ipc

INFO [06-21|13:49:29.150472] HTTP endpoint opened url=http://127.0.0.1:8545cors=* vhosts=localhost

 

In logs, look for line which says “IPC endpoint opened: url=/Users/hemants/.ethereum/myprivatenet/geth.ipc”.

This url allow us to interact with blockchain via IPC channel. Copy the path and fire new terminal and go to $GOPATH/bin and execute following command to attach to IPC channel :

./geth attach /Users/hemants/.ethereum/myprivatenet/geth.ipc

Welcome to the Geth JavaScript console!

modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

>

With this console we can execute various commands to interact with blockchain. If you look carefully, after connecting to IPC channel, console prints out all available modules with which you can interact. Namely modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0 are the modules. To see what functionality is available to use from the console just enter the module name, It will print out all the function names which we can call from the console. Like type ‘personal’ in the console it will response with following function names and properties:

> personal

{

listAccounts: [],

listWallets: [],

deriveAccount: function(),

ecRecover: function(),

getListAccounts: function(callback),

getListWallets: function(callback),

importRawKey: function(),

lockAccount: function(),

newAccount: function github.com/ethereum/go-ethereum/console.(*bridge).NewAccount-fm(),

openWallet: function github.com/ethereum/go-ethereum/console.(*bridge).OpenWallet-fm(),

sendTransaction: function(),

sign: function github.com/ethereum/go-ethereum/console.(*bridge).Sign-fm(),

signTransaction: function(),

unlockAccount: function github.com/ethereum/go-ethereum/console.(*bridge).UnlockAccount-fm()

}

Let’s call listAccounts property to see if we have any account.

> personal.listAccounts

[ ]

As expected we don’t have any account yet. So let’s create one.

personal.newAccount(“password”)

“0x66c4c909098df782ef0a52464749de9ef294762c”

Now call personal.listAccounts

personal.listAccounts

[“0x66c4c909098df782ef0a52464749de9ef294762c”]

Let’s check the balance in this account, It should be zero.

> eth.getBalance(personal.listAccounts[0]).toNumber()

0

To generated some ether we need to mine some blocks. Before mining the block we need to tell our node to run as miner node. To do that issue following command

> miner.start(1)

After some time new blocks get generated. As you are the only miner in this network you will be rewarded with ether. Lets check the balance again

> eth.getBalance(personal.listAccounts[0]).toNumber()

10000000000000000000

Great we now have ethers. We can stop miner by specifying ‘miner.stop()’, to stop unnecessary mining. We can check the height of blockchain by running following command :

> web3.eth.getBlockNumber(function(e,r){ console.log(r)})

2

Which is telling us that current height of blockchain is 2. We can view any block by specifying it number by issuing following command :

> web3.eth.getBlock(1)

{

difficulty: 1903376,

extraData: "0xd88301080c846765746887676f312e392e328664617277696e",

gasLimit: 20979494,

gasUsed: 0,

hash: "0x6d719d3447497302f59aac284cbb7df585c4cc591c4d351d994d4c234c667ac7",

logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",

miner: "0x66c4c909098df782ef0a52464749de9ef294762c",

mixHash: "0x094731b8b22a8efba3327b61fe602d4e76e8857ff12db14c8d53afbd651260ff",

nonce: "0x082d1975e5cbe2c8",

number: 1,

parentHash: "0x07185fe5cd8ac52781a3893b7d08b1240c1ded9f84eb97ee68fe2901ed82bcc4",

receiptsRoot: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",

sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",

size: 537,

stateRoot: "0xa3c4d51416278ab6c77c407027870d9a2c5e214d40e7d32f0275d692ca5ca69f",

timestamp: 1529571309,

totalDifficulty: 3903376,

transactions: [],

transactionsRoot: "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",

uncles: []

}

Amazing, now we have properly running Ethereum private network with development environment set up. We are ready to move on to next tutorial.