Architecture and Node Planning

A distributed Apache DolphinScheduler setup relies on decoupled Master, Worker, and API server components. The following hostnames and roles will be used throughout this guide:

192.168.30.141 s141 (Master)
192.168.30.142 s142 (Master)
192.168.30.143 s143 (API Server)
192.168.30.144 s144 (Worker)
192.168.30.145 s145 (Worker)
192.168.30.146 s146 (Worker)
192.168.30.147 s147 (Worker)

Required Stack and Preparations

Ensure the following software is available and properly configured:

• PostgreSQL 8.2+ or MySQL 5.7 (along with JDBC driver 5.1.47+ if MySQL is selected)
• JDK 1.8+ with JAVA_HOME and PATH exported in /etc/profile
• ZooKeeper 3.4.6+
• Hadoop 2.6+ or MinIO (optional, only required if the built-in resource upload feature will store files there)

DolphinScheduler does not embed a full Hadoop or Spark distribution; it only invokes client binaries to submit jobs.

OS User Setup

Every machine in the cluster needs a dedciated user account with passwordless sudo. The following steps adjust the system accordingly.

# Execute as root
useradd dolphinscheduler
echo "111111" | passwd --stdin dolphinscheduler

# Grant passwordless sudo
echo 'dolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' >> /etc/sudoers

# Disable tty requirement if present
sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers

When a resource storage backend like HDFS or MinIO is used, grant the dolphinscheduler user read-write access there as well.

Hostname Resolution and SSH Key Exchange

Edit /etc/hosts on every node to include all cluster members:

192.168.30.141 s141
192.168.30.142 s142
192.168.30.143 s143
192.168.30.144 s144
192.168.30.145 s145
192.168.30.146 s146
192.168.30.147 s147

Generate SSH keys and establish trust from the main deployment node (s141) to itself and all remaining nodes. Perform this as the dolphinscheduler user.

su dolphinscheduler
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Propagate keys to all target hosts:

for host in s142 s143 s144 s145 s146 s147; do
    ssh-copy-id $host   # prompt for dolphinscheduler password once per host
done

Final, create the application directory and assign ownership:

mkdir -p /opt/dolphinscheduler
sudo chown -R dolphinscheduler:dolphinscheduler /opt/dolphinscheduler

Repeat the directory creation on all additional nodes.

Obtaining and Customizing the Distribution

Either download a prebuilt binary from the official release page or clone the source repository and build it manually. A source build makes it easier to pre-configure database connection details in datasource.properties and environment variables in dolphinscheduler_env.sh. After customizing the files, upload the compressed archive to /opt/dolphinscheduler on s141. Transfer will occur later via an automated deployment tool.

Automated Deployment and Startup Script

The script below unpacks the latest archive, stops a previously running Worker service (add or uncomment lines for Master and API services as needed), swaps the installation directory, adjusts permissions, and finally starts the target components. Only the Worker start line is active in this example; tailor the block to match the role assigned to each node.

#!/bin/sh
source_dir=`dirname $0`
source_dir=`cd ${source_dir}; pwd`

tarball_list=$(ls -l /opt/dolphinscheduler/ | awk '/^-/ {print $NF}' | grep 'apache-dolphinscheduler-incubating')
dist_file=''
for file in $tarball_list; do
    dist_file=$file
done

tar -zxvf $source_dir/$dist_file -C $source_dir
base_dir=${dist_file%%.tar.gz*}

# Stop desired components on update
# sh $source_dir/dolphinscheduler/bin/dolphinscheduler-daemon.sh stop master-server
sh $source_dir/dolphinscheduler/bin/dolphinscheduler-daemon.sh stop worker-server
# sh $source_dir/dolphinscheduler/bin/dolphinscheduler-daemon.sh stop api-server

rm -rf $source_dir/dolphinscheduler
mv $source_dir/$base_dir $source_dir/dolphinscheduler

chmod 755 $source_dir/dolphinscheduler/bin/*
sed -i 's/\r$//' $source_dir/dolphinscheduler/bin/dolphinscheduler-daemon.sh
sed -i 's/\r$//' $source_dir/dolphinscheduler/conf/env/dolphinscheduler_env.sh

# Start components according to the node role
# sh $source_dir/dolphinscheduler/bin/dolphinscheduler-daemon.sh start master-server
sh $source_dir/dolphinscheduler/bin/dolphinscheduler-daemon.sh start worker-server
# sh $source_dir/dolphinscheduler/bin/dolphinscheduler-daemon.sh start api-server

The script should be placed on s141 and executed selectively on the remaining nodes via a tool such as Alibaba Cloud Toolkit, which streamlines copying artifacts and running commands on multiple servers sequentially.

Verification

Once the script finishes on each machine, check that the corresponding processes started correctly. For the API node, open a browser and navigate to:

http://192.168.30.143:12345/dolphinscheduler

Log in with the default credentials to confirm the cluster is operational and all registered Worker agents appear in the monitoring panel.