code version: fabric 2.0.1
foreword
I am study orderer of fabric2.0 recently, which need at least 3 node to constitute a cluster and TLS is also mandatory.
operation location
mkdir multiNodeConfig
cd multiNodeConfig
export FABRIC_CFG_PATH=$PWD
All our command below will exec under multiNodeConfig path.
config
Since sampleconfig is not enough to establish a 3 raft nodes cluster, I need to generate our msp.
step1: generate msp
create crypto-config.yaml:
OrdererOrgs:
- Name: SampleOrg
Domain: example.com
Specs:
- Hostname: orderer0
- Hostname: orderer1
- Hostname: orderer2
- Hostname: peer0
Users:
Count: 1
then generate msp
cryptogen generate --config=./crypto-config.yaml
step2: modify configtx.yaml
copy sampleconfig except msp
cp fabric_src_path/sampleconfig/* ./ -rp
rm -rf msp
modify the underlying config item
MSPDir:
MSPDir: crypto-config/ordererOrganizations/example.com/msp
Organizations:
We will run 3 orderer node in a machine, So we should ensure they use different ports.
OrdererEndpoints:
- orderer0.example.com:7050
- orderer1.example.com:8050
- orderer2.example.com:9050
AnchorPeers:
- Host: peer0.example.com
Port: 7051
Orderer: &OrdererDefaults
# EtcdRaft defines configuration which must be set when the "etcdraft"
# orderertype is chosen.
EtcdRaft:
# The set of Raft replicas for this network. For the etcd/raft-based
# implementation, we expect every replica to also be an OSN. Therefore,
# a subset of the host:port items enumerated in this list should be
# replicated under the Orderer.Addresses key above.
Consenters:
- Host: orderer0.example.com
ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.crt
ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.crt
- Host: orderer1.example.com
Port: 8050
ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer1.example.com/tls/server.crt
ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer1.example.com/tls/server.crt
- Host: orderer2.example.com
Port: 9050
ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/server.crt
ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/server.crt
generate genesis block and channel tx
env
export SYS_CHANNEL_PROFILE=SampleDevModeEtcdRaft
export SYS_CHANNEL_NAME=sys-chan
export CHANNEL_PROFILE=SampleSingleMSPChannel
export CHANNEL_NAME=mychan1
export CC_NAME=mycc1
genesis block
configtxgen -profile \${SYS_CHANNEL_PROFILE} -channelID \$SYS_CHANNEL_NAME -outputBlock ./channel-artifacts/genesis.block
channel tx
configtxgen -profile \$CHANNEL_PROFILE -channelID \$CHANNEL_NAME -outputCreateChannelTx ./channel-artifacts/channel.tx
step3: modify orderer.yaml
My plan is to debug orderer, So I run orderer directly Since docker is not so convenient to debug and also windows.
First, rename orderer.yaml to orderer0.yaml
mv orderer.yaml orderer0.yaml
then modify the underlying config item.
TLS:
# TLS: TLS settings for the GRPC server.
TLS:
Enabled: true
# PrivateKey governs the file location of the private key of the TLS certificate.
PrivateKey: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.key
# Certificate governs the file location of the server TLS certificate.
Certificate: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.crt
RootCAs:
- crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt
ClientAuthRequired: false
ClientRootCAs:
cluster:
It is important to setup ClientCertificate and ClientPrivateKey which is used to establish mutual TLS connections with other ordering service nodes.
We use the same certificate and private key pair when acting as a TLS server and client.
Cluster:
# SendBufferSize is the maximum number of messages in the egress buffer.
# Consensus messages are dropped if the buffer is full, and transaction
# messages are waiting for space to be freed.
SendBufferSize: 10
# ClientCertificate governs the file location of the client TLS certificate
# used to establish mutual TLS connections with other ordering service nodes.
ClientCertificate: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.crt
# ClientPrivateKey governs the file location of the private key of the client TLS certificate.
ClientPrivateKey: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/server.key
BootstrapFile & LocalMSPDir:
BootstrapFile: channel-artifacts/genesis.block
# LocalMSPDir is where to find the private crypto material needed by the
# orderer. It is set relative here as a default for dev environments but
# should be changed to the real location in production.
LocalMSPDir: crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/msp
FileLedger
FileLedger:
# Location: The directory to store the blocks in.
# NOTE: If this is unset, a new temporary location will be chosen every time
# the orderer is restarted, using the prefix specified by Prefix.
Location: production/orderer0
# The prefix to use when generating a ledger directory in temporary space.
# Otherwise, this value is ignored.
Prefix: hyperledger-fabric-ordererledger0
Operations:
Operations:
# host and port for the operations server
ListenAddress: 127.0.0.1:8443
Consensus:
Consensus:
# The allowed key-value pairs here depend on consensus plugin. For etcd/raft,
# we use following options:
# WALDir specifies the location at which Write Ahead Logs for etcd/raft are
# stored. Each channel will have its own subdir named after channel ID.
WALDir: production/orderer0/etcdraft/wal
# SnapDir specifies the location at which snapshots for etcd/raft are
# stored. Each channel will have its own subdir named after channel ID.
SnapDir: production/orderer0/etcdraft/snapshot
create orderer1.yaml and orderer2.yaml
cp orderer0.yaml orderer1.yaml
cp orderer0.yaml orderer2.yaml
What we need to do is just to modify the number suffix(0->1 or 0->2)
step4: modify core.yaml
id and listenAddress:
# The peer id provides a name for this peer instance and is used when
# naming docker resources.
id: peer0.example.com
# The networkId allows for logical separation of networks and is used when
# naming docker resources.
networkId: dev
# The Address at local network interface this Peer will listen on.
# By default, it will listen on all network interfaces
listenAddress: peer0.example.com:7051
address:
# When used as peer config, this represents the endpoint to other peers
# in the same organization. For peers in other organization, see
# gossip.externalEndpoint for more info.
# When used as CLI config, this means the peer's endpoint to interact with
address: peer0.example.com:7051
gossip:
gossip is not need since we have single org and peer.
gossip:
# Bootstrap set to initialize gossip with.
# This is a list of other peers that this peer reaches out to at startup.
# Important: The endpoints here have to be endpoints of peers in the same
# organization, because the peer would refuse connecting to these endpoints
# unless they are in the same organization as the peer.
bootstrap: peer0.example.com:7051
TLS:
# TLS Settings
tls:
# Require server-side TLS
enabled: true
# Require client certificates / mutual TLS.
# Note that clients that are not configured to use a certificate will
# fail to connect to the peer.
clientAuthRequired: false
# X.509 certificate used for TLS server
cert:
file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/server.crt
# Private key used for TLS server (and client if clientAuthEnabled
# is set to true
key:
file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/server.key
# Trusted root certificate chain for tls.cert
rootcert:
file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/ca.crt
# Set of root certificate authorities used to verify client certificates
clientRootCAs:
files:
- crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/ca.crt
# Private key used for TLS when making client connections. If
# not set, peer.tls.key.file will be used instead
clientKey:
file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/server.key
# X.509 certificate used for TLS when making client connections.
# If not set, peer.tls.cert.file will be used instead
clientCert:
file: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/server.crt
fileSystemPath:
fileSystemPath: production/peer
mspConfigPath:
# Path on the file system where peer will find MSP local configurations
mspConfigPath: crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/msp
vm:
endpoint:
externalBuilders:
If you run fabric2.0 in windows, you can see my another blob 《让fabric 2.0 external builder在windows上跑起来》
externalBuilders:
- path: E:\share_vir\go\src\github.com\hyperledger\fabric\multiNodeConfig\externalbuilders\golang
name: external-golang
environmentWhitelist:
- GOPROXY
- GOCACHE
- GOPATH
host
As smart as you are, Why we setup endpoints by xx.example.com instead of 127.0.0.
From my observation, fabric use grpc to communicate with remote node. and IP of remote node is parsed by dns. So we need to add our host entry in hosts file.
127.0.0.1 orderer0.example.com
127.0.0.1 orderer1.example.com
127.0.0.1 orderer2.example.com
127.0.0.1 peer0.example.com
start
rebuild orderer
Since orderer.yaml is the config of orderer which is hard in code, we should modify source code.
for example, replace from “orderer” to “orderer0” below and compile it, and then replaced by “orderer1” and “orderer2” step by step.
// Load parses the orderer YAML file and environment, producing
// a struct suitable for config use, returning error on failure.
func Load() (*TopLevel, error) {
config := viper.New()
coreconfig.InitViper(config, "orderer")
config.SetEnvPrefix(Prefix)
config.AutomaticEnv()
replacer := strings.NewReplacer(".", "_")
config.SetEnvKeyReplacer(replacer)
if err := config.ReadInConfig(); err != nil {
return nil, fmt.Errorf("Error reading configuration: %s", err)
}
var uconf TopLevel
if err := viperutil.EnhancedExactUnmarshal(config, &uconf); err != nil {
return nil, fmt.Errorf("Error unmarshaling config into struct: %s", err)
}
uconf.completeInitialization(filepath.Dir(config.ConfigFileUsed()))
return &uconf, nil
}
At last, we have orderer0, orderer1 and orderer2, and their config file is orderer0.yaml, orderer1.yaml and orderer2.yaml respectively.
exec
orderer start
exec they in 3 terminals
terminal0:
orderer0 start
terminal1:
orderer1 start
terminal2:
orderer2 start
You will see the log which indicates that cluster of raft node is working all the time.
peer start
cp fabric_src_path/integration/externalbuilders ./ -rp
GOCACHE=(go env GOCACHE) peer node start
env
export CORE_PEER_MSPCONFIGPATH=crypto-config/ordererOrganizations/example.com/users/Admin@example.com/msp/
export CORE_PEER_ADDRESS=peer0.example.com:7051
export CORE_PEER_TLS_ROOTCERT_FILE=crypto-config/ordererOrganizations/example.com/orderers/peer0.example.com/tls/ca.crt
export localMspId="SampleOrg"
In fact, Only the first item is necessary Since channel create/join need Admin. Peer will get config from core.yaml if the specified env var unset.
channel
channel create:
We have to specify the address(not IP:PORT) of orderer and its tlscacerts.
Note that the address of orderer shoudle be like orderer0.example.com:7050 but not 127.0.0.1:7050.
We have explain it in "Host"
peer channel create -c $CHANNEL_NAME -f channel-artifacts/mychan1.tx -o orderer0.example.com:7050 --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/msp/tlscacerts/tlsca.example.com-cert.pem
channel join
peer channel join -b mychan1.block -o orderer0.example.com:7050 --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/msp/tlscacerts/tlsca.example.com-cert.pem
chaincode
TLS
if TLS is enabled in peer, chaincode is also need to use TLS to establish secure connection.
Note that tls (client) key, cert and root cert of chaincode is generated by peer and pass them to chaincode with env variable.
Key and cert must be encoded to base64 format before fabric 2.0…
CORE_TLS_CLIENT_KEY_PATH
CORE_TLS_CLIENT_CERT_PATH
key and cert can be raw or base64 format after fabric2.0.
In order to be compatible with previous versions, fabric add 3 env vars to handle raw format.
CORE_TLS_CLIENT_KEY_FILE
CORE_TLS_CLIENT_CERT_FILE
CORE_PEER_TLS_ROOTCERT_FILE
They are passe into chaincode by externalBuilder/golang/bin/run.
#!/bin/bash
# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
set -euo pipefail
if [ "$#" -ne 2 ]; then
>&2 echo "Expected 2 directories got $#"
exit 1
fi
OUTPUT=$1
ARTIFACTS=$2
# shellcheck disable=SC2155
export CORE_CHAINCODE_ID_NAME="$(jq -r .chaincode_id "$ARTIFACTS/chaincode.json")"
export CORE_PEER_TLS_ENABLED="true"
export CORE_TLS_CLIENT_CERT_FILE="$ARTIFACTS/client.crt"
export CORE_TLS_CLIENT_KEY_FILE="$ARTIFACTS/client.key"
export CORE_PEER_TLS_ROOTCERT_FILE="$ARTIFACTS/root.crt"
export CORE_PEER_LOCALMSPID="$(jq -r .mspid "$ARTIFACTS/chaincode.json")"
jq -r .client_cert "$ARTIFACTS/chaincode.json" > "$CORE_TLS_CLIENT_CERT_FILE"
jq -r .client_key "$ARTIFACTS/chaincode.json" > "$CORE_TLS_CLIENT_KEY_FILE"
jq -r .root_cert "$ARTIFACTS/chaincode.json" > "$CORE_PEER_TLS_ROOTCERT_FILE"
if [ -z "$(jq -r .client_cert "$ARTIFACTS/chaincode.json")" ]; then
export CORE_PEER_TLS_ENABLED="false"
fi
exec "$OUTPUT/chaincode" -peer.address="$(jq -r .peer_address "$ARTIFACTS/chaincode.json")"
chaincode prepare
copy chaincode from integration. We will take chaincode/module as our chaincode.
cp fabric_src_path/integration/chaincode ./ -rp
Note the version of fabric-chaincode-go package which chaincode/module use is too old too use in fabric2.0. So we should update it.
cd chaincode/module/
rm -rf go.mod go.sum
go mod init
go build
rm -rf module
cd -
chaincode package
peer lifecycle chaincode package --label golang-external -l golang -p chaincode/module/ mychan1.tgz
chaincode install
install chaincode and then record package id in mycc1_package_id (env var), we will use package id below.
peer lifecycle chaincode install mychan1.tgz
2020-03-31 20:05:37.607 CST [cli.lifecycle.chaincode] submitInstallProposal -> INFO 031 Installed remotely: response:<status:200 payload:"\nPgolang-external:a311029de93cdf49ab56eaa2a77a3539774498c8187fa0f35f59fccaf56cf3a9\022\017golang-external" >
2020-03-31 20:05:37.607 CST [cli.lifecycle.chaincode] submitInstallProposal -> INFO 032 Chaincode code package identifier: golang-external:a311029de93cdf49ab56eaa2a77a3539774498c8187fa0f35f59fccaf56cf3a9
export mycc1_package_id=golang-external:a311029de93cdf49ab56eaa2a77a3539774498c8187fa0f35f59fccaf56cf3a9
chaincode approve
peer lifecycle chaincode approveformyorg --channelID $CHANNEL_NAME --name $CC_NAME --version 1.0 --init-required --package-id $mycc1_package_id --sequence 1 --waitForEvent --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt -o orderer0.example.com:7050
chaincode commit
peer lifecycle chaincode commit --channelID $CHANNEL_NAME --name $CC_NAME --version 1.0 --init-required --sequence 1 --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt -o orderer0.example.com:7050
chaincode invoke
init
peer chaincode invoke -C $CHANNEL_NAME -I -n $CC_NAME -c '{"Args": ["Init", "a", "1000", "b", "2000"]}' --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt -o orderer0.example.com:7050
query
peer chaincode invoke -C $CHANNEL_NAME -n $CC_NAME -c '{"Args": ["query", "a"]}' --tls --cafile crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt -o orderer0.example.com:7050
2020-04-01 08:49:25.447 CST [chaincodeCmd] chaincodeInvokeOrQuery -> INFO 001 Chaincode invoke successful. result: status:200 payload:"1000"