Zilliqa

• Skip myip while sending to random lookup (#1834)
• Fixing migrated state quoting (#1835)

• Remove extra quotes for map value during migration (#1823)
• Add an IPC unit test for empty string map key (#1824)
• Fix recover community testnet lookup and it crash (#1825)
• Start listening when lookup finish sync (#1827)

• Add data_migrate to bin folder (#1809)
• Seednodes to skip itself adding to lookupnodes (#1812)
• Fixes for Smart Contract execution on empty map key & RevertCommitTemp (#1818)
• Fix unnecessary mutex (#1819)

New Features & Improvements

• New Features
  • Implement the DS Reputation System (#1587)
  • Contract map storage & Scilla IPC (#1743)
  • Add GetTotalSupply API (#1779)
  • Integrate fetching a substate of a smart contract (#1803)
• Functional improvements
  • Sync/rejoin lagging-behind lookup nodes after recovery (#1726)
  • Control txns backup to S3 (#1780)
  • Change default value for small scale test (#1783)
  • Enable fetching missing mbs/txns from lookup for seed nodes (#1791)
  • Use exists function in CheckTxBody (#1799)
  • Return code & data if they are present (#1804)
• Infrastructure improvements
  • Use GetEpochFin in auto_back_up.py (#1774)
  • Generate txBodies ldb and optionally json from raw bytes from S3 files (#1777)
  • Integrate proto3 to cmake (#1778)
  • Avoid deserialization in CheckIntegrity (#1792)
  • Check the correctness of hash inside the block (#1794)

Bug Fixes

• Functional bug fixes
  • Fix Blocklink index and Set to NO_SYNC in PoW state (#1782)
  • Fix access null pointer (#1784)
  • Force serialization of primitive proto fields for block and tx data (#1793)
  • Fix genesis DS block hash not correct problem (#1798)
  • Fix possible race cond (#1800)
• Infrastructure bug fixes
  • Missing include mbstxns flag (#1775)
  • Fix misleading comments (#1801)

New Features & Improvements

• Functional improvements
  • Add states during contract creation (#1688)
  • Add more lookup checks when creating txn (#1730)
  • Adjust ds microblock gas limit after view change (#1731)
  • Store txns in DB (#1735)
  • Generate pow header hash with peer port (#1736)
  • Optimize DS leader verify fail log (#1737)
  • GETSTARTPOW improvement (#1746)
  • Save txns to S3 (#1756)
• Infrastructure improvements
  • Support customized log path (#1651)
  • Move scripts from testnet to core repo (#1693)
  • Add param for bucket name in release.sh (#1716)
  • Automatically remove ipMapping.xml after using it (#1718)
  • Comply with new release structure (#1732)
  • Improve validateDB logging and add continueOnError option (#1733)
  • Pick from temp (#1734)
  • Add checksum check (#1738)
  • Add redundant backup (#1761)
  • Fetch missing statedelta from seed (#1762)
  • Added multipart etag support (#1765)

Bug Fixes

• Functional bug fixes
  • Fix potential concurrency issue for AccountStore::m_state access (#1717)
  • Remove check for microblock epoch num to match the latest tx block num (#1722)
  • Fix handling of unavailable microblocks (#1723)
  • Remove redundant mutex (#1728)
  • Don't fetch info from blacklisted seed nodes (#1739)
  • Daemon should re-launch process with previous synctype (#1755)
• Infrastructure bug fixes
  • Avoid validateDB crashing when checking normal node (#1741)
  • Fix start uploadIncrDB.py with error (#1766)
  • Fix backup (#1770)

This release integrates these infrastructure improvements:

• Added multipart etag support (#1765)
• Fix start uploadIncrDB.py with error (#1766)

New Features & Improvements

• Functional improvements
  • Save txns to S3 (#1756)
• Infrastructure improvements
  • Add redundant backup (#1761)
  • Fetch missing statedelta from seed (#1762)

Bug Fixes

• Functional bug fixes
  • Daemon should re-launch process with previous synctype (#1755)
• Infrastructure bug fixes
  • None

This release is a hotfix of v4.7.1 with no impact on users.

Infrastructure Improvement

• Improve daemon handling of sync type (#1752)

New Features & Improvements

• Functional improvements
  • Add states during contract creation (#1688)
  • Add more lookup checks when creating txn (#1730)
  • Adjust ds microblock gas limit after view change (#1731)
  • Store txns in DB (#1735)
  • Generate pow header hash with peer port (#1736)
  • Optimize DS leader verify fail log (#1737)
  • GETSTARTPOW improvement (#1746)
• Infrastructure improvements
  • Support customized log path (#1651)
  • Move scripts from testnet to core repo (#1693)
  • Add param for bucket name in release.sh (#1716)
  • Automatically remove ipMapping.xml after using it (#1718)
  • Comply with new release structure (#1732)
  • Improve validateDB logging and add continueOnError option (#1733)
  • Pick from temp (#1734)
  • Add checksum check (#1738)

Bug Fixes

• Functional bug fixes
  • Fix potential concurrency issue for AccountStore::m_state access (#1717)
  • Remove check for microblock epoch num to match the latest tx block num (#1722)
  • Fix handling of unavailable microblocks (#1723)
  • Remove redundant mutex (#1728)
  • Don't fetch info from blacklisted seed nodes (#1739)
• Infrastructure bug fixes
  • Avoid validateDB crashing when checking normal node (#1741)

• MICROBLOCK_GAS_LIMIT changed to 15000
• MAX_GOSSIP_MSG_SIZE_IN_BYTES changed to 8000000

New Features & Improvements

• Development/build-related changes
  • Use Travis native Ubuntu 16.04 (#1647)
  • Fix issues reported by cppcheck (#1652)
  • Standardize Include Guards (#1653)
  • Add more Clang Tidy checks (#1654)
  • Protect destructors of some template classes (#1655)
  • Standardize Include Guard Check (#1657)
  • Add pedantic flag to compiler options (#1659)
  • Fix some uninitialized params & Remove explicit copy constructor for std::move (#1660)
  • Add SECURITY.MD (#1670)
  • Add few essential MD files (#1671)
• Functional improvements
  • Increase max peer connection (#1662)
  • Remove redundant pow test case (#1676)
  • Add retry mechanism for send pow to proxy (#1683)
  • Add log for txn hash (#1692)
  • Add epoch num to GetNodeType (#1695)
  • log popen failure (#1712)
  • Node only do sync if failed to do pow (#1707)

Bug Fixes

• Functional bug fixes
  • Handle peer connection counter correctly (#1663)
  • Stop lookup api server when it is synchronizing (#1677)
  • clear peer counter (#1689)
  • Remove IP check on VC (#1690)
  • Fix log messages for address (#1710)
  • Check whether account exists when create contract (#1714)
  • Fix epochfin for lookup when no shard formed (#1709)
  • Continue check result from proxy until timeout (#1706)
• Infrastructure bug fixes
  • Fetch constant file from dsguard instead of normal node (#1679)
  • Fix of kubectl cp command (#1713)

New Features & Improvements

• None

Bug Fixes

• [release-4.6] Skip adding statedelta (#1700)
• [release-4.6] Check RECOVERY_TRIM_INCOMPLETED_BLOCK before CleanExtraTxBodies (#1704)

New Features & Improvements

• None

Bug Fixes

• [release-4.6] epochfin (#1696)
• [release-4.6] Buffering MB & Txns during seed syncup (#1697)

New Features & Improvements

• None

Bug Fixes

• [release-4.6] Check Server.cpp for Default block (#1685)

New Features & Improvements

• [release-4.6] Add blockhash to tx block JSON (#1668)
• [release-4.6] Custom storage path (#1669)

Bug Fixes

• None

This is the first release used for mainnet (v4.0.1). It is bundled with novel features with sharding at the core. Below we discuss some of the core features of the Zilliqa mainnet:

Sharding

The network supports transaction sharding for both regular payment transactions and those that invoke smart contracts. Processing smart contract transactions on a sharded architecture come with its own set of challenges. To learn more about Zilliqa’s approach to the problem, check out this blog post.

PBFT-Style Blockchain

Zilliqa will be one of the very few PBFT-style (for Practical Byzantine Fault Tolerance) blockchains available today. A PBFT-style consensus mechanism is efficient and gives finality to transactions so that confirmations are not required.

Safe Smart Contracts

The protocol comes shipped with a new smart contract language named Scilla. The language has been designed to eliminate many known vulnerabilities in existing smart contracts and make them amenable to formal verification.

Dual and Eco-Friendly Mining

It is possible to dual mine an Ethash-based PoW blockchain such as Ethereum and Zilliqa. This comes from the fact that Zilliqa uses a combination of PoW and PBFT, where, PoW is only used for Sybil resistance, while, PBFT is used for consensus. Since the PoW period on Zilliqa will last for roughly 1 min every 2–3 hours, we believe that the energy footprint of mining on Zilliqa will be much smaller compared to the blockchains that use PoW to reach consensus on every block.

Low Variance in Block Rewards

The protocol employs a novel incentive mechanism to reward miners by measuring their contributions in the consensus protocol. As a result, thousands of (or more) miners may get rewarded for a single block resulting in low variance.

Since the launch of Testnet v3, we have continued to make improvements to the codebase. We have since launched a new version of the Testnet v3.1.0 and released a testnet status page in our forum (https://forum.zilliqa.com/t/maoshanwang-testnet-status/151).

Here are some of the notable improvements we have made in the past few weeks.

Generalized DataSender Class

We observed from test runs that our assignment algorithm for forwarding messages between the DS committee, shards, and Lookup nodes was rather rudimentary. Additionally, the assignment code was spread out across several functions among the different classes. We have improved on this aspect by creating the DataSender class, which acts as a wrapper for the message forwarding. This new implementation selects nodes to act as message forwarders based on the co-signature of the most recently completed consensus. The fact that these nodes participated in the consensus serves as an indication of their continued liveness and increases the likelihood that the message will be forwarded successfully.

Changes to Transaction Processing, Verification, and Transmission

After observing our testnet perform under stress conditions, we concluded that our practice of buffering transactions with unexpected (i.e., larger) nonce values unnecessarily increased the complexity of handling transactions in a consistent manner across the nodes. This was especially the case when different recovery mechanisms such as view change would be triggered, or when transaction packets from Lookups would be received in an untimely manner. So, instead of maintaining the buffer, we chose to re-populate the current pool of transactions with the ones that would have been buffered previously, thus simplifying the overall workflow of transaction processing.

The verification algorithm was also modified to allow for an adjustable tolerance with respect to the ordering of the transactions proposed by the shard leader. Furthermore, to reduce the likelihood of the leader proposing a block with unexpected transaction ordering, we have modified the transaction packet transmission in such a way that the Lookup nodes send the transactions to the shard leader first. Should the shard backup nodes be missing some of these transactions, we already have code in place whereby the backups can fetch these missing transactions from the leader.

More Restrictive Timestamp Check

Another improvement we have recently made addresses potential security vulnerabilities in the way we verify timestamps across blocks. Previously, any new block proposed by the leader during consensus would be accepted if the timestamp was larger than the previous one in the chain. This time, we have changed the acceptance criterion to be based on the difference between the backup node’s local time and the received timestamp from the leader.

Bounded Message Queues

Our use of boost::queue::push to insert messages to our incoming and outgoing queues allowed the addition of entries beyond the defined size of the queue. This meant resource over-utilization was possible, including the uncontrolled spawning of hundreds of threads. Now we have moved to use boost::queue:bounded_push, which prevents adding new messages beyond the size limit. We have also removed the code that retries message insertion when the limit is reached. In effect, new messages are dropped when the node reaches capacity. We will be testing out this new behavior in the coming weeks, particularly to analyze the resource consumption of nodes in our testnet.

Network recovery mechanism

To prepare for unforeseeable events such as network failure, we have implemented a recovery mechanism for us to re-bootstrap and restore the whole network. To facilitate this, lookup nodes will routinely perform a backup procedure as a pre-emptive measure for network failure. When the network fails, a new network will be launched from the backup databases.

Update to lookup and introduction of seed nodes

We have introduced a new network layer called the seed node network. New seed nodes will be able to register with a lookup multiplier, a special node that mirrors lookup traffic to seed nodes, to be part of the seed network. The role of the seed node is to receive transactions from services such as a wallet and forward the transactions to the lookup. The lookups will then batch the transactions and assign them to the corresponding shard for processing.

Shard node to trigger the rejoining process if it misses final block(s)

A shard node can miss a final block due to various issues such as intermittent network failure. We have introduced a mechanism for the shard node to securely check whether it is missing final block(s). In the event of missing final block(s), the shard node will then re-sync itself and rejoin the shard.

Addition of tolerance for validating IP addresses in the sharding structure

In some instances, the Directory Service leader and backups may receive PoW submissions from the same node but with different IP addresses (or ports), resulting in the backup failing to validate the sharding structure proposed by the leader. Such a situation can occur when, for example, a node has been restarted by its user with a different IP address or port, or perhaps when the change in IP address is due to IP address lease expiration. To accommodate these possibilities, we have added a tolerance value when validating the sharding structure, allowing the DS backups to accept the sharding structure from the DS leader if the number of nodes with differing IP addresses is within said limit.

Changes in Protobuf field definitions to allow backward compatibility in persistent storage

The Protobuf field definitions for serializing and deserializing messages to and from persistent storage are mostly set to “required”. This means that these fields must be set in order for the message object to be initialized. However, there is a possibility of these fields being unused or deprecated in future updates. Deserializing from persistent storage can then become an issue. As part of our effort to support backward compatibility, we have now set these fields to “optional”. The core C++ source code will now implement the checks for those fields that are considered required. This essentially moves the enforcement of required fields from the message content (i.e., the Protobuf definitions) to the source code, which is easier to change between software updates than the format of data already stored.

This is the first release used in Mao Shan Wang Testnet v3.0

If you have been following our bi-weekly project updates, you may find many of the new features of Mao Shan Wang familiar. These are the features we have been working on for the past few months, and we are excited to now make these publicly available for miners and developers to test.

Core Protocol:

1. Support for smart contracts on a sharded architecture
2. GPU mining
3. Coinbase rewards
4. Gas rewards
5. Gas pricer
6. Difficulty adjustment of PoW
7. Upgrading protocol
8. Gossip protocol
9. Multiple-In, Multiple-Out (MIMO) DS committee
10. Event/Log transaction receipt
11. Merged PoWs: Combining the two rounds of PoW into one round
12. Node recovery
13. State delta forwarding
14. Migration to Kubernetes
15. Archival nodes

GPU Selection for PoW Mining

Mao Shan Wang supports GPU mining. Miners may use one or multiple GPU units for one node. For example, a miner with 6 available GPUs can choose to configure one Zilliqa node to employ any combination of the 6 GPUs, or launch multiple Zilliqa nodes with each node assigned to one specific GPU.
Coinbase Rewards
In this implementation of our testnet, we will reward all the nodes (or miners) – including DS and shard nodes – based on their involvement in the consensus protocol. Specifically, the more actively a node participates in the consensus protocol through signing blocks in a transaction epoch, the more it will be rewarded with tokens. For instance, for a microblock/finalblock, a node can contribute up to two signatures, and can thus be rewarded twice.

Gas Rewards

Another feature essential to a public blockchain is the distribution of rewards to miners from gas fees. In the version, gas fees generated by processing transactions are accumulated into the total coinbase rewards. The rewards are then issued  just before the next round of PoW submissions begins.

Gas Pricer

While our transactions are already processed with gas consumption and limits are taken into consideration, the mechanism for determining actual gas pricing had yet to be implemented until recently. Our newly-coded gas pricer works by first, having miners affix to their proof-of-work (PoW) submissions a minimum gas price that they are willing to accept. The Directory Service (DS) committee then reaches consensus over the acceptable global minimum gas price for the network in the coming DS epoch. The network will thereafter only accept transactions that have a gas price larger than or equal to the agreed-upon global minimum.

Support for smart contracts on a sharded architecture

A few months ago, we released a blog post that details our design to support smart contract sharding. In the design, we leverage the DS committee to process certain types of smart contract transactions. As a result, the DS committee now runs an additional consensus round to validate those transactions. This happens after the DS committee has received data on transactions validated by shards.

Upgrading Protocol

It is crucial that a system can be upgrade or updated with new features or patches to ensure its security. We’ve implemented the first version of our upgrading protocol. A source (say https://latest-release.zilliqa.com) hosts the latest version of our source code and binary with SHA-256 value, signed by multi parties, e.g., Zilliqa Research. The software version will be stored in a separated file named VERSION, with the information of version, expected DS epoch and SHA-256 value.

Node Recovery

Node recovery is one of the more significant features that we have completed in the past few weeks. If a node was terminated for any reason (such as to complete an upgrade) and then relaunched, it will read the persistent data stored in the machine’s database (data such as the DS committee members, the sharding structure, DS and Final Blocks, etc.) in a bid to recover its last known state and to begin resynchronizing with the rest of the network.

This is the first release used in D24 Testnet v2.0

We have included in this release of the testnet major improvements to the infrastructure, including stability, mining, dev-ops and smart contract support.

• For stability, we have implemented persistent storage using levelDB, recovery of normal/DS/lookup nodes and the view-change mechanism to replace malicious nodes.
• For mining, we’ve finished the implementation for nodes (or miners) to join the testnet as normal nodes or DS nodes during the period of pow1/pow2. We reward all the nodes including DS and shard nodes through their involvement in the consensus protocol.
• For dev-ops, we containerized our main project Zilliqa. As a result, we can easily launch a 20- to 1000-node Kubernetes cluster on our cloud provider and are running an internal large-scale Zilliqa testnet on it.
• On smart contract support, we have refined the implementation of transaction / account to support smart contracts. We’ve finished implementing the first version of an interpreter to run Scilla contracts.

This is a stability release. New features/fixes includes

• Genesis block to generate valid testnet tokens
• Validation of transactions from wallet
• Out of tree build
• Revamp logic for new node joining
• Shard node rejoining
• DS backup rejoining
• State root check failure fix