fabric2.0 establish a 3 raft nodes cluster that need not docker

code version: fabric 2.0.1

foreword

I am study orderer of fabric2.0 recently, which need at least 3 node to constitute a cluster and TLS is also mandatory.

operation location

mkdir multiNodeConfig
cd multiNodeConfig
export FABRIC_CFG_PATH=$PWD

All our command below will exec under multiNodeConfig path.

config

Since sampleconfig is not enough to establish a 3 raft nodes cluster, I need to generate our msp.

step1: generate msp

create crypto-config.yaml:

OrdererOrgs:
  - Name: SampleOrg
    Domain: example.com
    Specs:
      - Hostname: orderer0
      - Hostname: orderer1
      - Hostname: orderer2
      - Hostname: peer0
    Users:
      Count: 1

then generate msp

cryptogen generate --config=./crypto-config.yaml

step2: modify configtx.yaml

copy sampleconfig except msp

cp fabric_src_path/sampleconfig/* ./ -rp
rm -rf msp

modify the underlying config item

MSPDir:

MSPDir: crypto-config/ordererOrganizations/example.com/msp

Organizations:
We will run 3 orderer node in a machine, So we should ensure they use different ports.

        OrdererEndpoints:
            - orderer0.example.com:7050
            - orderer1.example.com:8050
            - orderer2.example.com:9050

        AnchorPeers:
            - Host: peer0.example.com
              Port: 7051

Orderer: &OrdererDefaults

 # EtcdRaft defines configuration which must be set when the "etcdraft"
    # orderertype is chosen.
    EtcdRaft:
        # The set of Raft replicas for this network. For the etcd/raft-based
        # implementation, we expect every replica to also be an OSN. Therefore,
        # a subset of the host:port items enumerated in this list should be
        # replicated under the Orderer.Addresses key above.
        Consenters:
            - Host: orderer0.example.com
              ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.crt
              ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.crt
            - Host: orderer1.example.com
              Port: 8050
              ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer1.example.com/tls/server.crt
              ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer1.example.com/tls/server.crt
            - Host: orderer2.example.com
              Port: 9050
              ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/server.crt
              ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/server.crt

generate genesis block and channel tx

env

export SYS_CHANNEL_PROFILE=SampleDevModeEtcdRaft
export SYS_CHANNEL_NAME=sys-chan
export CHANNEL_PROFILE=SampleSingleMSPChannel
export CHANNEL_NAME=mychan1
export CC_NAME=mycc1

genesis block

configtxgen -profile \${SYS_CHANNEL_PROFILE} -channelID \$SYS_CHANNEL_NAME -outputBlock ./channel-artifacts/genesis.block

channel tx

configtxgen -profile \$CHANNEL_PROFILE -channelID \$CHANNEL_NAME -outputCreateChannelTx ./channel-artifacts/channel.tx

step3: modify orderer.yaml

My plan is to debug orderer, So I run orderer directly Since docker is not so convenient to debug and also windows.

First, rename orderer.yaml to orderer0.yaml

mv orderer.yaml orderer0.yaml

then modify the underlying config item.

TLS:

    # TLS: TLS settings for the GRPC server.
    TLS:
        Enabled: true
        # PrivateKey governs the file location of the private key of the TLS certificate.
        PrivateKey: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.key
        # Certificate governs the file location of the server TLS certificate.
        Certificate: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.crt
        RootCAs:
          - crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt
        ClientAuthRequired: false
        ClientRootCAs:

cluster:

It is important to setup ClientCertificate and ClientPrivateKey which is used to establish mutual TLS connections with other ordering service nodes.

We use the same certificate and private key pair when acting as a TLS server and client.

    Cluster:
        # SendBufferSize is the maximum number of messages in the egress buffer.
        # Consensus messages are dropped if the buffer is full, and transaction
        # messages are waiting for space to be freed.
        SendBufferSize: 10
        # ClientCertificate governs the file location of the client TLS certificate
        # used to establish mutual TLS connections with other ordering service nodes.
        ClientCertificate: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.crt
        # ClientPrivateKey governs the file location of the private key of the client TLS certificate.
        ClientPrivateKey: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.key

BootstrapFile & LocalMSPDir:

    BootstrapFile: channel-artifacts/genesis.block

    # LocalMSPDir is where to find the private crypto material needed by the
    # orderer. It is set relative here as a default for dev environments but
    # should be changed to the real location in production.
    LocalMSPDir: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/msp

FileLedger

FileLedger:

    # Location: The directory to store the blocks in.
    # NOTE: If this is unset, a new temporary location will be chosen every time
    # the orderer is restarted, using the prefix specified by Prefix.
    Location: production/orderer0

    # The prefix to use when generating a ledger directory in temporary space.
    # Otherwise, this value is ignored.
    Prefix: hyperledger-fabric-ordererledger0

Operations:

Operations:
    # host and port for the operations server
    ListenAddress: 127.0.0.1:8443

Consensus:

Consensus:
    # The allowed key-value pairs here depend on consensus plugin. For etcd/raft,
    # we use following options:

    # WALDir specifies the location at which Write Ahead Logs for etcd/raft are
    # stored. Each channel will have its own subdir named after channel ID.
    WALDir: production/orderer0/etcdraft/wal

    # SnapDir specifies the location at which snapshots for etcd/raft are
    # stored. Each channel will have its own subdir named after channel ID.
    SnapDir: production/orderer0/etcdraft/snapshot

create orderer1.yaml and orderer2.yaml

cp orderer0.yaml orderer1.yaml
cp orderer0.yaml orderer2.yaml

What we need to do is just to modify the number suffix(0->1 or 0->2)

step4: modify core.yaml

id and listenAddress:

    # The peer id provides a name for this peer instance and is used when
    # naming docker resources.
    id: peer0.example.com

    # The networkId allows for logical separation of networks and is used when
    # naming docker resources.
    networkId: dev

    # The Address at local network interface this Peer will listen on.
    # By default, it will listen on all network interfaces
    listenAddress: peer0.example.com:7051

address:

    # When used as peer config, this represents the endpoint to other peers
    # in the same organization. For peers in other organization, see
    # gossip.externalEndpoint for more info.
    # When used as CLI config, this means the peer's endpoint to interact with
    address: peer0.example.com:7051

gossip:
gossip is not need since we have single org and peer.

    gossip:
        # Bootstrap set to initialize gossip with.
        # This is a list of other peers that this peer reaches out to at startup.
        # Important: The endpoints here have to be endpoints of peers in the same
        # organization, because the peer would refuse connecting to these endpoints
        # unless they are in the same organization as the peer.
        bootstrap: peer0.example.com:7051

TLS:

    # TLS Settings
    tls:
        # Require server-side TLS
        enabled:  true
        # Require client certificates / mutual TLS.
        # Note that clients that are not configured to use a certificate will
        # fail to connect to the peer.
        clientAuthRequired: false
        # X.509 certificate used for TLS server
        cert:
            file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/server.crt
        # Private key used for TLS server (and client if clientAuthEnabled
        # is set to true
        key:
            file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/server.key
        # Trusted root certificate chain for tls.cert
        rootcert:
            file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/ca.crt
        # Set of root certificate authorities used to verify client certificates
        clientRootCAs:
            files:
              - crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/ca.crt
        # Private key used for TLS when making client connections.  If
        # not set, peer.tls.key.file will be used instead
        clientKey:
            file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/server.key
        # X.509 certificate used for TLS when making client connections.
        # If not set, peer.tls.cert.file will be used instead
        clientCert:
            file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/server.crt

fileSystemPath:

fileSystemPath: production/peer

mspConfigPath:

    # Path on the file system where peer will find MSP local configurations
    mspConfigPath: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/msp

vm:

endpoint:

externalBuilders:
If you run fabric2.0 in windows, you can see my another blob 《让fabric 2.0 external builder在windows上跑起来》

    externalBuilders:
      - path: E:\share_vir\go\src\github.com\hyperledger\fabric\multiNodeConfig\externalbuilders\golang
        name: external-golang
        environmentWhitelist:
          - GOPROXY
          - GOCACHE
          - GOPATH

host

As smart as you are, Why we setup endpoints by xx.example.com instead of 127.0.0.

From my observation, fabric use grpc to communicate with remote node. and IP of remote node is parsed by dns. So we need to add our host entry in hosts file.

127.0.0.1 orderer0.example.com
127.0.0.1 orderer1.example.com
127.0.0.1 orderer2.example.com
127.0.0.1 peer0.example.com

start

rebuild orderer

Since orderer.yaml is the config of orderer which is hard in code, we should modify source code.
for example, replace from “orderer” to “orderer0” below and compile it, and then replaced by “orderer1” and “orderer2” step by step.

// Load parses the orderer YAML file and environment, producing
// a struct suitable for config use, returning error on failure.
func Load() (*TopLevel, error) {
	config := viper.New()
	coreconfig.InitViper(config, "orderer")
	config.SetEnvPrefix(Prefix)
	config.AutomaticEnv()
	replacer := strings.NewReplacer(".", "_")
	config.SetEnvKeyReplacer(replacer)

	if err := config.ReadInConfig(); err != nil {
		return nil, fmt.Errorf("Error reading configuration: %s", err)
	}

	var uconf TopLevel
	if err := viperutil.EnhancedExactUnmarshal(config, &uconf); err != nil {
		return nil, fmt.Errorf("Error unmarshaling config into struct: %s", err)
	}

	uconf.completeInitialization(filepath.Dir(config.ConfigFileUsed()))
	return &uconf, nil
}

At last, we have orderer0, orderer1 and orderer2, and their config file is orderer0.yaml, orderer1.yaml and orderer2.yaml respectively.

exec

orderer start

exec they in 3 terminals
terminal0:

orderer0 start

terminal1:

orderer1 start

terminal2:

orderer2 start

You will see the log which indicates that cluster of raft node is working all the time.

peer start

cp fabric_src_path/integration/externalbuilders ./ -rp
GOCACHE=(go env GOCACHE) peer node start

env

export CORE_PEER_MSPCONFIGPATH=crypto-config/ordererOrganizations/example.com/users/Admin@example.com/msp/
export CORE_PEER_ADDRESS=peer0.example.com:7051
export CORE_PEER_TLS_ROOTCERT_FILE=crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/ca.crt
export localMspId="SampleOrg"

In fact, Only the first item is necessary Since channel create/join need Admin. Peer will get config from core.yaml if the specified env var unset.

channel

channel create:

We have to specify the address(not IP:PORT) of orderer and its tlscacerts.
Note that the address of orderer shoudle be like orderer0.example.com:7050 but not 127.0.0.1:7050.
We have explain it in "Host"

peer channel create -c $CHANNEL_NAME -f channel-artifacts/mychan1.tx -o orderer0.example.com:7050 --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/msp/tlscacerts/tlsca.example.com-cert.pem

channel join

peer channel join -b mychan1.block -o orderer0.example.com:7050 --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/msp/tlscacerts/tlsca.example.com-cert.pem

chaincode

TLS

if TLS is enabled in peer, chaincode is also need to use TLS to establish secure connection.
Note that tls (client) key, cert and root cert of chaincode is generated by peer and pass them to chaincode with env variable.

Key and cert must be encoded to base64 format before fabric 2.0…

CORE_TLS_CLIENT_KEY_PATH
CORE_TLS_CLIENT_CERT_PATH

key and cert can be raw or base64 format after fabric2.0.
In order to be compatible with previous versions, fabric add 3 env vars to handle raw format.

CORE_TLS_CLIENT_KEY_FILE
CORE_TLS_CLIENT_CERT_FILE
CORE_PEER_TLS_ROOTCERT_FILE

They are passe into chaincode by externalBuilder/golang/bin/run.

#!/bin/bash

# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

if [ "$#" -ne 2 ]; then
    >&2 echo "Expected 2 directories got $#"
    exit 1
fi

OUTPUT=$1
ARTIFACTS=$2

# shellcheck disable=SC2155
export CORE_CHAINCODE_ID_NAME="$(jq -r .chaincode_id "$ARTIFACTS/chaincode.json")"
export CORE_PEER_TLS_ENABLED="true"
export CORE_TLS_CLIENT_CERT_FILE="$ARTIFACTS/client.crt"
export CORE_TLS_CLIENT_KEY_FILE="$ARTIFACTS/client.key"
export CORE_PEER_TLS_ROOTCERT_FILE="$ARTIFACTS/root.crt"
export CORE_PEER_LOCALMSPID="$(jq -r .mspid "$ARTIFACTS/chaincode.json")"

jq -r .client_cert "$ARTIFACTS/chaincode.json" > "$CORE_TLS_CLIENT_CERT_FILE"
jq -r .client_key  "$ARTIFACTS/chaincode.json" > "$CORE_TLS_CLIENT_KEY_FILE"
jq -r .root_cert   "$ARTIFACTS/chaincode.json" > "$CORE_PEER_TLS_ROOTCERT_FILE"

if [ -z "$(jq -r .client_cert "$ARTIFACTS/chaincode.json")" ]; then
    export CORE_PEER_TLS_ENABLED="false"
fi

exec "$OUTPUT/chaincode" -peer.address="$(jq -r .peer_address "$ARTIFACTS/chaincode.json")"

chaincode prepare

copy chaincode from integration. We will take chaincode/module as our chaincode.

cp fabric_src_path/integration/chaincode ./ -rp

Note the version of fabric-chaincode-go package which chaincode/module use is too old too use in fabric2.0. So we should update it.

cd chaincode/module/
rm -rf go.mod go.sum
go mod init
go build
rm -rf module
cd -

chaincode package

peer lifecycle chaincode package --label golang-external -l golang -p chaincode/module/ mychan1.tgz

chaincode install

install chaincode and then record package id in mycc1_package_id (env var), we will use package id below.

peer lifecycle chaincode install mychan1.tgz
2020-03-31 20:05:37.607 CST [cli.lifecycle.chaincode] submitInstallProposal -> INFO 031 Installed remotely: response:<status:200 payload:"\nPgolang-external:a311029de93cdf49ab56eaa2a77a3539774498c8187fa0f35f59fccaf56cf3a9\022\017golang-external" >
2020-03-31 20:05:37.607 CST [cli.lifecycle.chaincode] submitInstallProposal -> INFO 032 Chaincode code package identifier: golang-external:a311029de93cdf49ab56eaa2a77a3539774498c8187fa0f35f59fccaf56cf3a9

export mycc1_package_id=golang-external:a311029de93cdf49ab56eaa2a77a3539774498c8187fa0f35f59fccaf56cf3a9

chaincode approve

peer lifecycle chaincode approveformyorg --channelID $CHANNEL_NAME --name $CC_NAME --version 1.0 --init-required --package-id $mycc1_package_id --sequence 1 --waitForEvent --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt -o orderer0.example.com:7050

chaincode commit

peer lifecycle chaincode commit --channelID $CHANNEL_NAME --name $CC_NAME --version 1.0 --init-required --sequence 1 --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt -o orderer0.example.com:7050

chaincode invoke

init

peer chaincode invoke -C $CHANNEL_NAME -I -n $CC_NAME -c '{"Args": ["Init", "a", "1000", "b", "2000"]}' --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt -o orderer0.example.com:7050

query

peer chaincode invoke -C $CHANNEL_NAME -n $CC_NAME -c '{"Args": ["query", "a"]}' --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt -o orderer0.example.com:7050
2020-04-01 08:49:25.447 CST [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 001 Chaincode invoke successful. result: status:200 payload:"1000"