Ansible自动化部署Flink、Hadoop

如果需要部署单机版本的flink,可前往Ansible自动化部署Flink单机版本-CSDN博客

一、准备环境

1、准备机器

本次以192.168.190.155,192.168.190.156,192.168.190.157,192.168.190.158四台虚机为例,本文操作全在管理机中。

机器IP

节点

CPU&内存

系统盘

存储

192.168.190.155

Ansible管理机

2C4G

50GB

50GB

192.168.190.158

Master/Worker(被管理机)

2C4G

50GB

50GB

192.168.190.157

Master/Worker(被管理机)

2C4G

50GB

50GB

192.168.190.156

Worker(被管理机)

2C4G

50GB

50GB

本次主要操作都在管理机:192.168.190.155上完成

2、为192.168.190.156,192.168.190.157,192.168.190.158三台虚机创建flink用户及免密(三台被管理机之间)

① 创建flink用户:

useradd flink(三台虚机都需要flink用户)

② 免密操作:

a 先切换到flink用户下:

su flink

b 执行免密指令:

##路径:/home/flink/.ssh

ssh-keygen -t rsa(一直按Enter即可)

cat id_rsa.pub >>authorized_keys  ##将名为id_rsa.pub的公钥文件的内容追加到authorized_keys文件中

chmod 700 .ssh  ##为目录.ssh赋予'0700'的权限

chmod 600 authorized_keys  ##为文件authorized_keys赋予'0600'的权限

3、关闭防火墙

systemctl status firewalld

若关闭,则检查下一项,若开启,则需要执行如下命令

systemctl stop firewalld  ##临时关闭防火墙,

systemctl disable firewalld  ##永久关闭防火墙

4、在管理机192.168.190.155上设置免密登录被管理机:

① 执行免密指令:

ssh-keygen -t rsa

② 使用ssh-copy-id传输公钥:

ssh-copy-id root@192.168.190.158  ##向flink-master传输公钥

ssh-copy-id root@192.168.190.156  ##向flink-worker1传输公钥

ssh-copy-id root@192.168.190.157  ##向flink-worker2传输公钥

5、在管理机192.168.190.155上创建必要的目录

① 创建ansible剧本目录:

mkdir -p /etc/ansible/roles/{flink,hadoop,java}/{defaults,tasks,templates}

② 创建tar包存放目录:

mkdir -p /data/{flink,hadoop,java}

6、在管理机的ansible的hosts文件中设置变量:

路径:vi /etc/ansible/hosts

[flink_servers]

flink-master ansible_host=192.168.190.158 MASTER=flink-master WORKER1=flink-worker1 WORKER2=flink-worker2

flink-worker1 ansible_host=192.168.190.156 MASTER=flink-master WORKER1=flink-worker1 WORKER2=flink-worker2

flink-worker2 ansible_host=192.168.190.157 MASTER=flink-master WORKER1=flink-worker1 WORKER2=flink-worker2

[flink_cluster]

flink-master ansible_host=192.168.190.158

flink-worker1 ansible_host=192.168.190.156

flink-worker2 ansible_host=192.168.190.157

二、编辑配置java剧本及变量

1、配置defaults目录下的配置文件(设置变量):main.yml

vi /etc/ansible/roles/flink/defaults/main.yml

内容:

---

JAVA_VERSION: 11.0.2

JAVA_INSTALL_DIR: /data/java

JAVA_HOME: "{{ JAVA_INSTALL_DIR }}/jdk-{{ JAVA_VERSION }}"

JAVA_TAR: jdk-{{ JAVA_VERSION }}_linux-x64_bin.tar.gz

2、配置tasks目录下的配置文件(剧本):java.yml

vi /etc/ansible/roles/flink/tasks/java.yml

内容:

---

- name: Deploy Java-JDK-11.0.2

  hosts: flink_cluster

  become: yes

  roles:

    - java

  tasks:

    - name: Create

      ansible.builtin.file:

        path: /data

        state: directory

        mode: '0755'

    - name: Create under /data/flink

      ansible.builtin.file:

        path: /data/flink

        state: directory

        mode: '0755'

    - name: Create under /data/java

      ansible.builtin.file:

        path: /data/java

        state: directory

        mode: '0755'

    - name: Create under /data/hadoop

      ansible.builtin.file:

        path: /data/hadoop

        state: directory

        mode: '0755'

    - name: Create under /data/data

      ansible.builtin.file:

        path: /data/data

        state: directory

        mode: '0755'

    - name: Create under /data/data/ts

      ansible.builtin.file:

        path: /data/data/ts

        state: directory

        mode: '0755'

    - name: Create under /data/hadoop/logs

      ansible.builtin.file:

        path: /data/hadoop/logs

        state: directory

        mode: '0755'

    - name: Copy Java

      copy:

        src: "{{ JAVA_INSTALL_DIR }}/{{ JAVA_TAR }}"

        dest: /tmp/{{ JAVA_TAR }}

    - name: Extrack Java

      unarchive:

        src: /tmp/{{ JAVA_TAR }}

        dest: "{{ JAVA_INSTALL_DIR }}"

        remote_src: yes

    - name: Add JAVA_HOME to /etc/profile.d

      blockinfile:

        path: /etc/profile.d/java.sh

        create: yes

        mode: '0644'

        block: |

          export JAVA_HOME={{ JAVA_HOME }}

          export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
          export PATH=$PATH:$JAVA_HOME/bin

        marker: "#{mark}ANSIBLE MANAGED BLOCK - JAVA_HOME"

- name: Deploy Hosts

  hosts: flink_cluster

  become: yes

  roles:

    - java

  tasks:

    - name: Ensure hosts enviroment variables are set in /etc/hosts

      blockinfile:

        path: /etc/hosts

        block: |

          {% for host in groups['flink_cluster'] %}

          {{ hostvars[host]['ansible_host'] }} {{ host }}

          {% endfor %}

        marker: "# {mark} ANSIBLE MANAGED BLOCK FOR hosts ENVIRONMENT VARIABLES"

        create: yes

        mode: '0644'

    - name: Template /etc/hostname

      ansible.builtin.template:

        src: /etc/ansible/roles/java/templates/hostname.j2

        dest: /etc/hostname

3、编辑配置jinja2文件(配合剧本使用,自动创建hadoop配置文件):

① hostname.j2

vi /etc/ansible/roles/java/templates/hostname.j2

内容:

{{ inventory_hostname}}

详细信息请参考以下文件:

[hostname.j2]

三、编辑配置hadoop剧本及变量

1、配置defaults目录下的配置文件(设置变量):main.yml

vi /etc/ansible/roles/hadoop/defaults/main.yml

内容:

---

HADOOP_VERSION: 3.3.3

JAVA_VERSION: 11.0.2

JAVA_INSTALL_DIR: /data/java

JAVA_HOME: "{{ JAVA_INSTALL_DIR }}/jdk-{{ JAVA_VERSION }}"

HADOOP_INSTALL_DIR: /data/hadoop

HADOOP_HOME: "{{ HADOOP_INSTALL_DIR }}/hadoop-{{ HADOOP_VERSION }}"

HADOOP_TAR: hadoop-{{ HADOOP_VERSION }}.tar.gz

HADOOP_JINJA2: /etc/ansible/roles/hadoop/templates

2、配置tasks目录下的配置文件(剧本):hadoop.yml

vi /etc/ansible/roles/hadoop/tasks/hadoop.yml

内容:

---

- name: Deploy Hadoop

  hosts: flink_servers

  become: yes

  roles:

    - hadoop

  tasks:

    - name: Copy Hadoop

      copy:

        src: "{{ HADOOP_INSTALL_DIR }}/{{ HADOOP_TAR }}"

        dest: /tmp/{{ HADOOP_TAR }}

    - name: Extrack Hadoop

      unarchive:

        src: /tmp/{{ HADOOP_TAR }}

        dest: "{{ HADOOP_INSTALL_DIR }}"

        remote_src: yes

    - name: Add HADOOP_HOME to /etc/profile.d

      blockinfile:

        path: /etc/profile.d/hadoop.sh

        create: yes

        mode: '0644'

        block: |

          export HADOOP_HOME={{ HADOOP_HOME }}

          export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
          export HADOOP_PID_DIR=${HADOOP_HOME}/pids

          export HADOOP_CLASSPATH=$(hadoop classpath)
          export HADOOP_CONF_DIR={{ HADOOP_HOME }}/etc/hadoop
          export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

          export FORMAT_MESSAGES_PATTERN_DISABLE_LOOKUPS=true

        marker: "# {mark} ANSIBLE MANAGED BLOCK - HADOOP_HOME"

    - name: Template core-site.j2

      ansible.builtin.template:

        src: "{{ HADOOP_JINJA2 }}/core-site.j2"

        dest: "{{ HADOOP_HOME }}/etc/hadoop/core-site.xml"

    - name: Template hadoop-env.j2

      ansible.builtin.template:

        src: "{{ HADOOP_JINJA2 }}/hadoop-env.j2"

        dest: "{{ HADOOP_HOME }}/etc/hadoop/hadoop-env.sh"

    - name: Template hadoop-workers.j2

      ansible.builtin.template:

        src: "{{ HADOOP_JINJA2 }}/hadoop-workers.j2"

        dest: "{{ HADOOP_HOME }}/etc/hadoop/workers"

    - name: Template hdfs-site.j2

      ansible.builtin.template:

        src: "{{ HADOOP_JINJA2 }}/hdfs-site.j2"

        dest: "{{ HADOOP_HOME }}/etc/hadoop/hdfs-site.xml"

    - name: Template mapred-site.j2

      ansible.builtin.template:

        src: "{{ HADOOP_JINJA2 }}/mapred-site.j2"

        dest: "{{ HADOOP_HOME }}/etc/hadoop/mapred-site.xml"

    - name: Template yarn-env.j2

      ansible.builtin.template:

        src: "{{ HADOOP_JINJA2 }}/yarn-env.j2"

        dest: "{{ HADOOP_HOME }}/etc/hadoop/yarn-env.sh"

    - name: Template yarn-site.j2

      ansible.builtin.template:

        src: "{{ HADOOP_JINJA2 }}/yarn-site.j2"

        dest: "{{ HADOOP_HOME }}/etc/hadoop/yarn-site.xml"

    - name: Template hadoop.service.j2

      ansible.builtin.template:

        src: "{{ HADOOP_JINJA2 }}/hadoop.service.j2"

        dest: /etc/systemd/system/hadoop.service

3、编辑配置jinja2文件(配合剧本使用,自动创建hadoop配置文件):

① core-site.j2

vi /etc/ansible/roles/hadoop/templates/core-site.j2

内容:

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<!-- 配置HDFS的NameNode所在节点服务器 -->

<name>fs.defaultFS</name>

<value>hdfs://{{ MASTER }}:9000</value>

</property>

<property>

<!-- 配置Hadoop的临时目录 -->

<name>hadoop.tmp.dir</name>

<value>{{ HADOOP_INSTALL_DIR }}/tmp</value>

<description>Abase for other temporary directories.</description>

</property>

</configuration>

详细信息请参考以下文件:

[core-site.j2]

② hadoop-env.j2

vi /etc/ansible/roles/hadoop/templates/hadoop-env.j2

内容:

export JAVA_HOME={{ JAVA_HOME }}

export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}

export HADOOP_CONF_DIR={{ HADOOP_HOME }}/etc/hadoop

export HADOOP_PID_DIR={{ HADOOP_INSTALL_DIR }}/ts

export HADOOP_LOG_DIR={{ HADOOP_INSTALL_DIR }}/logs

详细信息请参考以下文件:

[hadoop-env.j2]

③ hadoop.service.j2

vi /etc/ansible/roles/hadoop/templates/hadoop.service.j2

内容:

[Unit]  

Description=Hadoop Service  

After=network.target  

  

[Service]  

Type=forking  

User=flink

Group=flink

ExecStart={{ HADOOP_HOME }}/sbin/start-all.sh  

ExecStop={{ HADOOP_HOME }}/sbin/stop-all.sh  

Restart=on-abort  

  

[Install]  

WantedBy=multi-user.target

详细信息请参考以下文件:

[hadoop.service.j2]

④ hadoop-workers.j2

vi /etc/ansible/roles/hadoop/templates/hadoop-workers.j2

内容:

#localhost

{{ MASTER }}

{{ WORKER1 }}

{{ WORKER2 }}

详细信息请参考以下文件:

[hadoop-workers.j2]

⑤ hdfs-site.j2

vi /etc/ansible/roles/hadoop/templates/hdfs-site.j2

内容:

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

        <property>

        <!-- 配置HDFS的DataNode的备份数量 -->

        <name>dfs.replication</name>

        <value>3</value>

    </property>

    <property>

        <name>dfs.namenode.name.dir</name>

        <value>file:{{ HADOOP_INSTALL_DIR }}/hdfs/name</value>

    </property>

    <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:{{ HADOOP_INSTALL_DIR }}/hdfs/data</value>

    </property>

    <property>

        <!-- 配置HDFS的权限控制 -->

        <name>dfs.permissions.enabled</name>

        <value>false</value>

    </property>

    <property>

        <!-- 配置SecondaryNameNode的节点地址 -->

        <name>dfs.namenode.secondary.http-address</name>

        <value>{{ WORKER2 }}:9868</value>

    </property>

        <property>

          <name>dfs.http.address</name>

          <value>0.0.0.0:50070</value>

        </property>

</configuration>

详细信息请参考以下文件:

[hdfs-site.j2]

⑥ mapred-site.j2

vi /etc/ansible/roles/hadoop/templates/mapred-site.j2

内容:

详细信息请参考以下文件:

[mapred-site.j2]

⑦ yarn-env.j2

vi /etc/ansible/roles/hadoop/templates/yarn-env.j2

内容:

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<!-- 配置MR运行的环境 -->

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.application.classpath</name>

<value>

{{ HADOOP_HOME }}/etc/hadoop,

{{ HADOOP_HOME }}/share/hadoop/common/*,

{{ HADOOP_HOME }}/share/hadoop/common/lib/*,

{{ HADOOP_HOME }}/share/hadoop/hdfs/*,

{{ HADOOP_HOME }}/share/hadoop/hdfs/lib/*,

{{ HADOOP_HOME }}/share/hadoop/mapreduce/*,

{{ HADOOP_HOME }}/share/hadoop/mapreduce/lib/*,

{{ HADOOP_HOME }}/share/hadoop/yarn/*,

{{ HADOOP_HOME }}/share/hadoop/yarn/lib/*

</value>

</property>

</configuration>

详细信息请参考以下文件:

[yarn-env.j2]

⑧ yarn-site.j2

vi /etc/ansible/roles/hadoop/templates/yarn-site.j2

内容:

<?xml version="1.0"?>

<!--

  Licensed under the Apache License, Version 2.0 (the "License");

  you may not use this file except in compliance with the License.

  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<configuration>

<!-- Site specific YARN configuration properties -->

   <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <!-- 配置ResourceManager的服务节点 -->

        <name>yarn.resourcemanager.hostname</name>

        <value>{{ MASTER }}</value>

    </property>

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>{{ MASTER }}:8032</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address</name>

        <value>{{ MASTER }}:8088</value>

    </property>

        <!-- 配置当前节点yarn的可用资源 start -->

    <property>

        <name>yarn.nodemanager.resource.cpu-vcores</name>

           <value>15</value>

    </property>

        <property>

        <name>yarn.nodemanager.resource.memory-mb</name>

           <value>30000</value>

    </property>

        <!-- 配置当前节点yarn的可用资源 end -->

        <!-- 任务资源分配规则 start -->

    <property>

        <name>yarn.scheduler.minimum-allocation-vcores</name>

           <value>1</value>

    </property>

    <property>

        <name>yarn.scheduler.maximum-allocation-vcores</name>

           <value>4</value>

    </property>

        <!-- 任务资源分配规则 end -->

        <property>

        <name>yarn.log-aggregation-enable</name>

           <value>true</value>

    </property>

    <property>

          <name>yarn.nodemanager.vmem-check-enabled</name>

          <value>false</value>

          <description>Whether virtual memory limits will be enforced for containers.</description>

    </property>

        <property>

          <name>yarn.resourcemanager.am.max-attempts</name>

          <value>10</value>

          <description>

                The maximum number of application master execution attempts.

          </description>

        </property>

        <!-- 配置RM重启保存yarn上job状态 start -->

         <property>

           <description>Enable RM to recover state after starting. If true, then

           yarn.resourcemanager.store.class must be specified</description>

           <name>yarn.resourcemanager.recovery.enabled</name>

           <value>true</value>

         </property>

         <property>

           <description>The class to use as the persistent store.</description>

           <name>yarn.resourcemanager.store.class</name>

           <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>

         </property>

         <property>

           <name>yarn.resourcemanager.fs.state-store.uri</name>

           <value>${hadoop.tmp.dir}/yarn/system/rmstore</value>

         </property>

        <!-- 配置RM重启保存yarn上job状态 end -->

</configuration>

详细信息请参考以下文件:

[yarn-site.j2]

四、编辑配置flink剧本及变量

1、配置defaults目录下的配置文件(设置变量):main.yml

vi /etc/ansible/roles/flink/defaults/main.yml

内容:

---

FLINK_VERSION: 1.15.0

FLINK_TAR: flink-1.15.0-bin-scala_2.12.tgz

FLINK_HOME: /data/flink/flink-{{ FLINK_VERSION }}

FLINK_INSTALL_DIR: /data/flink

2、配置tasks目录下的配置文件(剧本):flink.yml

vi /etc/ansible/roles/flink/tasks/flink.yml

内容:

---

- name: Deploy Flink and Hadoop

  hosts: flink_servers

  become: yes

  roles:

    - flink

  tasks:

    - name: Copy flink

      copy:

        src: "{{ FLINK_INSTALL_DIR }}/{{ FLINK_TAR }}"

        dest: /tmp/{{ FLINK_TAR }}

    - name: Extrack flink

      unarchive:

        src: /tmp/{{ FLINK_TAR }}

        dest: "{{ FLINK_INSTALL_DIR }}"

        remote_src: yes

    - name: Template flink configuration file

      ansible.builtin.template:

        src: /etc/ansible/roles/flink/templates/flink-conf.j2

        dest: "{{ FLINK_HOME }}/conf/flink-conf.yaml"

    - name: Template flink.service.j2

      ansible.builtin.template:

        src: /etc/ansible/roles/flink/templates/flink-server.j2

        dest: /etc/systemd/system/flink.service

    - name: Template flink-masters.j2

      ansible.builtin.template:

        src: /etc/ansible/roles/flink/templates/flink-masters.j2

        dest: "{{ FLINK_HOME }}/conf/masters"

    - name: Template flink-workers.j2

      ansible.builtin.template:

        src: /etc/ansible/roles/flink/templates/flink-workers.j2

        dest: "{{ FLINK_HOME }}/conf/workers"

    - name: Add FLINK_HOME to /etc/profile.d

      blockinfile:

        path: /etc/profile.d/flink.sh

        create: yes

        mode: '0644'

        block: |

          export FLINK_HOME={{ FLINK_HOME }}

          export PATH=$PATH:$FLINK_HOME/bin

        marker: "# {mark} ANSIBLE MANAGED BLOCK - FLINK_HOME"

3、编辑配置jinja2文件(配合剧本使用,自动创建flink配置文件):

① flink-conf.j2

vi /etc/ansible/roles/flink/templates/flink-conf.j2

内容:

jobmanager.rpc.address: localhost

jobmanager.rpc.port: 6123

jobmanager.memory.process.size: 2048m

taskmanager.memory.process.size: 4096m

taskmanager.numberOfTaskSlots: 30

parallelism.default: 10

yarn.containers.vcores: 4

yarn.application-attempts: 10

metrics.scope.jm: <host>.jobmanager

metrics.scope.jm.job: <host>.jobmanager.<job_name>

metrics.scope.tm: <host>.taskmanager.<tm_id>

metrics.scope.tm.job: <host>.taskmanager.<tm_id>.<job_name>

metrics.scope.task: <host>.taskmanager.<tm_id>.<job_name>.<task_name>.<subtask_index>

metrics.scope.operator: <host>.taskmanager.<tm_id>.<job_name>.<operator_name>.<subtask_index>

state.backend: filesystem

state.checkpoints.dir: hdfs://{{ inventory_hostname }}:9000/flink-{{ FLINK_VERSION }}/flink-checkpoints

state.savepoints.dir: hdfs://{{ inventory_hostname }}:9000/flink-{{ FLINK_VERSION }}/flink-savepoints

state.backend.incremental: false

jobmanager.execution.failover-strategy: region

rest.port: 8081

rest.bind-port: 8085-8087

web.tmpdir: "{{ FLINK_HOME }}/flink-web"

web.upload.dir: "{{ FLINK_HOME }}/jars"

 taskmanager.memory.managed.fraction: 0.1

 taskmanager.memory.network.max: 200mb

 taskmanager.memory.jvm-metaspace.size: 256mb

jobmanager.archive.fs.dir: hdfs://{{ inventory_hostname }}:9000/flink-{{ FLINK_VERSION }}/completed-jobs

historyserver.archive.fs.dir: hdfs://{{ inventory_hostname }}:9000/flink-{{ FLINK_VERSION }}/completed-jobs

historyserver.archive.fs.refresh-interval: 10000

restart-strategy.failure-rate.max-failures-per-interval: 3

restart-strategy.failure-rate.failure-rate-interval: 5 min

restart-strategy.failure-rate.delay: 10 s

详细信息请参考以下文件:

[flink-conf.j2]

② flink.j2

vi /etc/ansible/roles/flink/templates/flink.j2

内容:

export FLINK_HOME={{ FLINK_HOME }}

export PATH=$PATH:$FLINK_HOME/bin

详细信息请参考以下文件:

[flink.j2]

③ flink-masters.j2

vi /etc/ansible/roles/flink/templates/flink-masters.j2

内容:

{{ MASTER }}:8081

{{ WORKER1 }}:8081

详细信息请参考以下文件:

[flink-masters.j2]

④ flink-server.j2

vi /etc/ansible/roles/flink/templates/flink-server.j2

内容:

[Unit]  

Description=Hadoop Service  

After=network.target  

  

[Service]  

Type=forking  

User=flink

Group=flink

ExecStart={{ FLINK_HOME }}/bin/start-cluster.sh  

ExecStop={{ FLINK_HOME }}/bin/stop-cluster.sh  

Restart=on-abort  

  

[Install]  

WantedBy=multi-user.target

详细信息请参考以下文件:

[flink-server.j2]

⑤ flink-workers.j2

vi /etc/ansible/roles/flink/templates/flink-workers.j2

内容:

{{ MASTER }}

{{ WORKER1 }}

{{ WORKER2 }}

详细信息请参考以下文件:

[flink-workers.j2]

五、执行ansible剧本指令

1、执行java.yml:

ansible-playbook java.yml  ##需要先cd /etc/ansible/roles/java/tasks

2、执行hadoop.yml:

ansible-playbook hadoop.yml  ##需要先cd /etc/ansible/roles/hadoop/tasks

3、执行flink.yml:

ansible-playbook flink.yml  ##需要先cd /etc/ansible/roles/flink/tasks

4、也可放在脚本中批量执行

① 创建脚本(名字自定义):

vi /etc/ansible/ansible-playbook.sh

内容(ps:脚本中引用的剧本yaml文件最好写绝对路径。):

#!bin/bash

ansible-playbook /etc/ansible/roles/java/tasks/java.yml

ansible-playbook /etc/ansible/roles/hadoop/tasks/hadoop.yml

ansible-playbook /etc/ansible/roles/flink/tasks/flink.yml

② 执行脚本:

sh ansible-playbook.sh  ##需要先cd /etc/ansible

六、在ansible剧本完成后,需前往被管理机执行下列语句(三台虚机都需要执行):

source /etc/profile.d/java.sh

source /etc/profile.d/hadoop.sh

source /etc/profile.d/flink.sh

七、备注注意项:

1、systemctl无法使用:

① 前往并修改配置文件:

cd /etc/systemd/system ,修改hadoop.service和flink.service

② 直接在hadoop或者flink目录的sbin/bin目录下启动:

hadoop: ./start-all.sh

flink: ./start-cluster.sh

  • 2
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
Hadoop是一个开源的分布式计算框架,可用于处理大数据集并提供高可靠性,高可用性和高性能。要进行详细的安装部署,需要运行以下步骤: 1. 安装Java:Hadoop基于Java编写,因此需要安装适当的Java版本。 2. 安装Hadoop:以二进制文件的形式下载Hadoop,并将其解压缩到目标位置。编辑Hadoop配置文件,并设置必要的参数,例如本地文件系统和Hadoop所依赖的其他组件。 3. 部署HDFS:使用bin/hdfs script启动HDFS守护进程并格式化NameNode。配置HDFS,并在数据节点上创建数据目录。 4. 部署YARN:使用bin/yarn script启动YARN守护进程,并在ResourceManager节点上运行MR程序的ApplicationMaster服务。重新配置YARN,并设置资源管理器和节点管理器。 5. 安装Spark:以二进制文件的形式下载Spark,并将其解压缩到目标位置。编辑Spark配置文件,并设置必要的参数,例如运行模式,内存设置和调试选项。 6. 安装Hive:以二进制文件的形式下载Hive,并按照说明进行安装。配置Hive,并设置Metastore和HiveServer2。 7. 安装HBase:以二进制文件的形式下载HBase,并按照说明进行安装。配置HBase,并设置区域服务器和HBase主服务器。 8. 安装Oozie:以二进制文件的形式下载Oozie,并按照说明进行安装。编辑Oozie配置文件,并设置必要的参数,例如数据库连接,属性和内存设置。 9. 安装Kafka:以二进制文件的形式下载Kafka,并按照说明进行安装。配置Kafka,并设置必要的参数,例如Zookeeper连接,日志存储位置和日志大小限制。 10. 安装Flume:以二进制文件的形式下载Flume,并按照说明进行安装。配置Flume,并设置必要的参数,例如Flume代理,事件类型和目标。 11. 安装Flink:以二进制文件的形式下载Flink,并按照说明进行安装。配置Flink,并设置必要的参数,例如集群模式,任务管理器,计算管道和作业提交方式。 12. 安装ES:以二进制文件的形式下载Elasticsearch,并按照说明进行安装。配置Elasticsearch,并设置必要的参数,例如节点类型,索引设置和查询配置。 13. 安装Redash:以二进制文件的形式下载Redash,并按照说明进行安装。配置Redash并设置必要的参数,例如数据库连接,权限和查询模式。 以上提到的大数据技术是开源的,所以可以在官网上找到相关二进制文件和详细的安装部署指南。也可以使用一些自动化部署工具,如Puppet和Ansible来简化整个过程。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

CTSXWT

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值