Running an etcd cluster on localhost

Purpose

• Run a cluster on localhost while investigating etcd
• Use a static cluster (So we have no external dependecies for bootstrapping)

Background information

• etcd source
• How to use etcdctl and etcd coreos's distributed key value store
• etcd clustering
• etcd documentation
• Consul which is a very popular alternative

Bootstrap

• Will use static bootstrapping
• Client connection port default is 2379 (Just supporting a single port per node), we will decerement the port for subsequent nodes so we do not get a port conflict
• Peer connection (Raft consensus) port default is 2380 (Just supporting a single port per node), we will increment the port for subsequent nodes so we do not get a port conflict
• Will use /tmp/etcdinv directory for the cluster - If you want the cluster to stick around use a different directory
  • If all nodes are stopped and then restarted the cluster will try to restart with this state, if the OS has not already purged this content
• Will write node logs to a file and run process in the background

# etcd bin directory
etcd_bin_dir=/home/pmcgrath/go/src/github.com/coreos/etcd/bin/

# Ensure we have a root directory for the cluster - Note we are using /tmp here, if you want the cluster to stick arounf use a different directory
mkdir -p /tmp/etcdinv

# Run node 1 $etcd_bin_dir./etcd \ -name node1 \ -data-dir /tmp/ectdinv/node1 \
　　-advertise-client-urls http://localhost:2379 \ -listen-peer-urls http://localhost:2380 \ -listen-client-urls http://localhost:2379 \ -initial-advertise-peer-urls http://localhost:2380 \ -initial-cluster-token MyEtcdCluster \ -initial-cluster node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382 \ -initial-cluster-state new &> /tmp/etcdinv/node1.log & # Run node 2 $etcd_bin_dir./etcd \ -name node2 \ -data-dir /tmp/ectdinv/node2 \
　　-advertise-client-urls http://localhost:2378 \ -listen-peer-urls http://localhost:2381 \ -listen-client-urls http://localhost:2378 \ -initial-advertise-peer-urls http://localhost:2381 \ -initial-cluster-token MyEtcdCluster \ -initial-cluster node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382 \ -initial-cluster-state new &> /tmp/etcdinv/node2.log & # Run node 3 $etcd_bin_dir./etcd \ -name node3 \ -data-dir /tmp/ectdinv/node3 \
 -advertise-client-urls http://localhost:2377 \ -listen-peer-urls http://localhost:2382 \ -listen-client-urls http://localhost:2377 \ -initial-advertise-peer-urls http://localhost:2382 \ -initial-cluster-token MyEtcdCluster \ -initial-cluster node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382 \ -initial-cluster-state new &> /tmp/etcdinv/node3.log & # List nodes ETCDCTL_PEERS=http://127.0.0.1:2379 $etcd_bin_dir/etcdctl member list 

• You can see the cluster node pids using
  • pidof etcd
  • ps aux | grep etcd

Interacting with the cluster using etcdctl

• Will use the client port 2379 based on this
• etcdctl defaults to 4001 at this time
• I could have added an extra client url for 4001 when bring up the nodes, but I'm guessing 4001 will be removed at some stage

# etcd bin directory
etcd_bin_dir=/home/pmcgrath/go/src/github.com/coreos/etcd/bin/

# Using node1
# Write a key ETCDCTL_PEERS=http://127.0.0.1:2379 $etcd_bin_dir/etcdctl set /dir1/key1 value1 # Should echo value1 # Read key ETCDCTL_PEERS=http://127.0.0.1:2379 $etcd_bin_dir/etcdctl get /dir1/key1 # Should echo value1 # Using node3 # Read key ETCDCTL_PEERS=http://127.0.0.1:2377 $etcd_bin_dir/etcdctl get /dir1/key1 # Should echo value1 

Kill one of the nodes

# etcd bin directory
etcd_bin_dir=/home/pmcgrath/go/src/github.com/coreos/etcd/bin/

# Kill node2
pidof etcd
# Should only have 3 pids kill $(ps aux | grep 'etcd \-name node2' | cut -d ' ' -f 2) pidof etcd # Should only have 2 pids # Read key using node1 ETCDCTL_PEERS=http://127.0.0.1:2379 $etcd_bin_dir/etcdctl get /dir1/key1 # Should echo value1 # Read key using node2 ETCDCTL_PEERS=http://127.0.0.1:2378 $etcd_bin_dir/etcdctl get /dir1/key1 # Should fail indicating cluster node could not available # Read key using node3 ETCDCTL_PEERS=http://127.0.0.1:2377 $etcd_bin_dir/etcdctl get /dir1/key1 # Should echo value1 

Using a proxy

# etcd bin directory
etcd_bin_dir=/home/pmcgrath/go/src/github.com/coreos/etcd/bin/

# Run a read write proxy - on 8080 
$etcd_bin_dir/etcd \ -proxy on \ -name proxy \ -listen-client-urls http://localhost:8080 \ -initial-cluster node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382 &> /tmp/etcdinv/proxy.log & # Read existing key ETCDCTL_PEERS=http://127.0.0.1:8080 $etcd_bin_dir/etcdctl get /dir1/key1 # Should echo value1 # Write a key ETCDCTL_PEERS=http://127.0.0.1:8080 $etcd_bin_dir/etcdctl set /dir1/key2 value2 # Should echo value2 # Read existing key ETCDCTL_PEERS=http://127.0.0.1:8080 $etcd_bin_dir/etcdctl get /dir1/key1 # Should echo value2




 ./etcdctl -peers 127.0.0.1:2379 member list