Fading Coder

One Final Commit for the Last Sprint

Home > Tech > Content

Deploying an etcd Cluster with HTTPS Communication Using Binary Packages on CentOS 7

Tech May 8 4

This guide walks through deploying a three-node etcd cluster on CentOS 7 using binary packages, with HTTPS communication enforced via a self-built Certificate Authority (CA). Prior familiarity with an HTTP-based etcd cluster is recommended.

For background on CA principles, see the articles on "How CA Works" and "Symmetric vs. Asymmetric Encryption". The certificate generation method used here is OpenSSL.

Host Information

Host IP Address
etcd-01 192.168.44.186
etcd-02 192.168.44.187
etcd-03 192.168.44.188

Environment Preparation

  1. Set hostnames (execute on each node)

    hostnamectl set-hostname etcd-01   # for etcd-01
hostnamectl set-hostname etcd-02   # for etcd-02
hostnamectl set-hostname etcd-03   # for etcd-03

  2. Configure local DNS resolution (execute on each node)

    cat >> /etc/hosts << EOF
192.168.44.186  etcd-01
192.168.44.187  etcd-02
192.168.44.188  etcd-03
EOF

  3. Disable firewalll and SELinux (execute on each node)

    systemctl disable firewalld --now
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

  4. Configure time synchronization (execute on each node)

    yum install chrony -y && systemctl enable --now chronyd

Download Binaries and Create Directory Structure

Execute the following on each node:

cd /opt
wget https://github.com/etcd-io/etcd/releases/download/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz
tar -zxvf etcd-v3.5.13-linux-amd64.tar.gz

# Create working directories: bin for binaries, cfg for config, ssl for certificates
mkdir -p /usr/local/etcd/{bin,cfg,ssl}

# Copy binaries
cp etcd-v3.5.13-linux-amd64/etcd* /usr/local/etcd/bin/

# Set environment variables
echo 'export ETCD_HOME=/usr/local/etcd' >> /etc/profile
echo 'export PATH=$PATH:$ETCD_HOME/bin' >> /etc/profile
source /etc/profile

# Verify version
etcd --version

Generate TLS Certificates

Generate certificates on etcd-01 and copy them to the other nodes.

cd /usr/local/etcd/ssl/

# Generate CA private key
openssl genrsa -out ca.key 2048

# Generate CA certificate (valid for 10000 days)
openssl req -x509 -new -nodes -key ca.key -subj "/CN=etcd" -days 10000 -out ca.crt

# Generate server private key
openssl genrsa -out server.key 2048

Create the certificate signing request (CSR) configuration file etcd-csr.conf. Ensure the [alt_names] block includes all cluster member IPs.

[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C = cn
ST = fujian
L = xiamen
O = etcd
OU = etcd
CN = etcd

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
IP.1 = 192.168.44.186
IP.2 = 192.168.44.187
IP.3 = 192.168.44.188

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names

Generate the CSR and server certificate:

openssl req -new -key server.key -out server.csr -config etcd-csr.conf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 10000 -extensions v3_ext -extfile etcd-csr.conf -sha256

List generated files:

ls -l
# Expected output: ca.crt, ca.key, ca.srl, etcd-csr.conf, server.crt, server.csr, server.key

Distribute certificates to other nodes:

scp * etcd-02:/usr/local/etcd/ssl/
scp * etcd-03:/usr/local/etcd/ssl/

Start the Cluster

1. Create etcd Configuration Files

For each node, create /usr/local/etcd/cfg/etcd.conf with the appropriate IP.

etcd-01:

cat > /usr/local/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.44.186:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.44.186:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.44.186:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.44.186:2379"
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.44.186:2380,etcd-02=https://192.168.44.187:2380,etcd-03=https://192.168.44.188:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

etcd-02:

cat > /usr/local/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.44.187:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.44.187:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.44.187:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.44.187:2379"
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.44.186:2380,etcd-02=https://192.168.44.187:2380,etcd-03=https://192.168.44.188:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

etcd-03:

cat > /usr/local/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.44.188:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.44.188:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.44.188:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.44.188:2379"
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.44.186:2380,etcd-02=https://192.168.44.187:2380,etcd-03=https://192.168.44.188:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

Configuration parameter explanation:

  • ETCD_NAME: Unique node name within the cluster.
  • ETCD_DATA_DIR: Directory for etcd data storage.
  • ETCD_LISTEN_CLIENT_URLS: URLs for client communication (default port 2379).
  • ETCD_LISTEN_PEER_URLS: URLs for inter-node communication (e.g., leader election, message passing; default port 2380).
  • ETCD_ADVERTISE_CLIENT_URLS: Advertised client URLs; should match listen-client-urls.
  • ETCD_INITIAL_ADVERTISE_PEER_URLS: Advertised peer URLs for cluster communication.
  • ETCD_INITIAL_CLUSTER: Comma-separated list of all initial peer URLs.
  • ETCD_INITIAL_CLUSTER_TOKEN: Unique cluster token to prevent accidental cluster merging.
  • ETCD_INITIAL_CLUSTER_STATE: Set to new for a fresh cluster, or existing when joining an existing one.

2. Create systemd Service File

Create /usr/lib/systemd/system/etcd.service on each node:

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/usr/local/etcd/cfg/etcd.conf
ExecStart=/usr/local/etcd/bin/etcd \
--cert-file=/usr/local/etcd/ssl/server.crt \
--key-file=/usr/local/etcd/ssl/server.key \
--peer-cert-file=/usr/local/etcd/ssl/server.crt \
--peer-key-file=/usr/local/etcd/ssl/server.key \
--trusted-ca-file=/usr/local/etcd/ssl/ca.crt \
--peer-trusted-ca-file=/usr/local/etcd/ssl/ca.crt \
--logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload

Service parameter explanation:

  • EnvironmentFile: Path to the etcd configuration file.
  • ExecStart: Path to the etcd binary with TLS flags:
    • --cert-file, --key-file: Server certificate and key for client connections.
    • --peer-cert-file, --peer-key-file: Certificate and key for peer (cluster) connections.
    • --trusted-ca-file, --peer-trusted-ca-file: CA certificate for verifying client and peer certificates.
    • --logger=zap: Enables structured logging with zap.

3. Start and Enable etcd

On each node:

systemctl start etcd
systemctl enable etcd

Verify Cluster Status

Check endpoint status

etcdctl \
 --cacert="/usr/local/etcd/ssl/ca.crt" \
 --cert="/usr/local/etcd/ssl/server.crt" \
 --key="/usr/local/etcd/ssl/server.key" \
 --endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.188:2379" \
 endpoint status --write-out=table

Example output:

+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.44.186:2379 | 21465eba80fe786a |  3.5.13 |   25 kB |     false |      false |         2 |          8 |                  8 |        |
| https://192.168.44.187:2379 | 5d144d32bb3de813 |  3.5.13 |   25 kB |     false |      false |         2 |          8 |                  8 |        |
| https://192.168.44.188:2379 | 51f535e47d80fd93 |  3.5.13 |   25 kB |     false |      false |         2 |          8 |                  8 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Check cluster health

etcdctl \
 --cacert="/usr/local/etcd/ssl/ca.crt" \
 --cert="/usr/local/etcd/ssl/server.crt" \
 --key="/usr/local/etcd/ssl/server.key" \
 --endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.188:2379" \
 endpoint health --write-out=table

Example output:

+-----------------------------+--------+-------------+-------+
|          ENDPOINT           | HEALTH |    TOOK     | ERROR |
+-----------------------------+--------+-------------+-------+
| https://192.168.44.187:2379 |   true |  17.89323ms |       |
| https://192.168.44.186:2379 |   true |  20.69533ms |       |
| https://192.168.44.188:2379 |   true | 21.574465ms |       |
+-----------------------------+--------+-------------+-------+

List cluster members

etcdctl \
 --cacert="/usr/local/etcd/ssl/ca.crt" \
 --cert="/usr/local/etcd/ssl/server.crt" \
 --key="/usr/local/etcd/ssl/server.key" \
 --endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.188:2379" \
 member list --write-out=table

Example output:

+------------------+---------+---------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |  NAME   |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+---------+-----------------------------+-----------------------------+------------+
| 21465eba80fe786a | started | etcd-01 | https://192.168.44.186:2380 | https://192.168.44.186:2379 |      false |
| 51f535e47d80fd93 | started | etcd-03 | https://192.168.44.188:2380 | https://192.168.44.188:2379 |      false |
| 5d144d32bb3de813 | started | etcd-02 | https://192.168.44.187:2380 | https://192.168.44.187:2379 |      false |
+------------------+---------+---------+-----------------------------+-----------------------------+------------+

Simplify etcdctl Usage with an Alias

To avoid typing the long certificate and endpoint flags each time, set an alias.

Temporary alias:

# To unalias: unalias etcdctl
alias etcdctl='etcdctl \
 --cacert="/usr/local/etcd/ssl/ca.crt" \
 --cert="/usr/local/etcd/ssl/server.crt" \
 --key="/usr/local/etcd/ssl/server.key" \
 --endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.188:2379"'

Permanent alias (user-level):

cat >> ~/.bashrc << EOF
alias etcdctl='etcdctl \
 --cacert="/usr/local/etcd/ssl/ca.crt" \
 --cert="/usr/local/etcd/ssl/server.crt" \
 --key="/usr/local/etcd/ssl/server.key" \
 --endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.188:2379"'
EOF
source ~/.bashrc

Permanent alias (system-wide):

cat >> /etc/profile << EOF
alias etcdctl='etcdctl \
 --cacert="/usr/local/etcd/ssl/ca.crt" \
 --cert="/usr/local/etcd/ssl/server.crt" \
 --key="/usr/local/etcd/ssl/server.key" \
 --endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.188:2379"'
EOF
source /etc/profile

After setting the alias, commands become simpler:

[root@etcd-01 ~]# etcdctl endpoint status --write-out=table
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.44.186:2379 | 21465eba80fe786a |  3.5.13 |   25 kB |     false |      false |         2 |         11 |                 11 |        |
| https://192.168.44.187:2379 | 5d144d32bb3de813 |  3.5.13 |   25 kB |     false |      false |         2 |         11 |                 11 |        |
| https://192.168.44.188:2379 | 51f535e47d80fd93 |  3.5.13 |   25 kB |     false |      false |         2 |         11 |                 11 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
Tags: etcdcluster
Back to List

Prev: Deploying and Managing Local Large Language Models with Ollama

Next: Diffusion Models Explained and Implemented with PyTorch

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.

Copyright © fadingcoder.top