puppet 基于单节点master横向扩展

weixin_46546303

已于 2024-07-15 22:22:11 修改

阅读量646

点赞数 9

文章标签： puppet 运维自动化云计算

于 2024-07-15 22:17:10 首次发布

本文链接：https://blog.csdn.net/weixin_46546303/article/details/140448703

版权

本章是基于单台master使用apache代理转发后端两台虚拟机
服务器系统：ubuntu16.04
安装软件包：puppetmaster-passenger （3.8.5-2ubuntu0.1）
###备注：
ubuntu 16 puppet服务由：puppetmaster 提供master服务，puppet软件包提供agent服务，所以如果需要部署基于webrick http服务的puppet master直接安装软件包即可

apt-get install puppetmaster

序言：

当我们使用包安装puppetmaster之后，启动puppet master服务后。默认情况下，puppet使用基于Ruby的Webrick http服务器，puppet自带的	Webrick模是用户不需要单独安装http服务器就能运行master服务。webrick虽然方便，但是不具备扩展能力，因而只适用于对puppet进行测试、评估和开发。生产环境中，需要使用更健壮的web服务器（如apahce或nginx）来处理大量用户请求。

服务部署

部署前准备：

如果在部署前已经安装puppetmaster，需要先把此服务停止

/etc/init.d/puppetmaster  stop

安装软件包

我是在未安装puppetmaster，所以直接安装puppetmaster-passenger

root@xxxx: apt-get install puppetmaster-passenger
........
apache2_invoke: Enable module passenger
Setting up ruby-augeas (1:0.5.0-3build4) ...
Setting up ruby-nokogiri (1.6.7.2-3ubuntu0.1) ...
Setting up ruby-rgen (0.7.0-2) ...
Setting up ruby-safe-yaml (1.0.4-1) ...
Setting up ruby-shadow (2.4.1-1build4) ...
Setting up puppet-common (3.8.5-2ubuntu0.1) ...
Setting up puppetmaster-common (3.8.5-2ubuntu0.1) ...
Setting up puppetmaster-passenger (3.8.5-2ubuntu0.1) ...
apache2_invoke: Enable module ssl
apache2_invoke: Enable module headers
Notice: Signed certificate request for ca
Notice: cnabdabpdu0e-12-37 has a waiting certificate request
Notice: Signed certificate request for cnabdabpdu0e-12-37
Notice: Removing file Puppet::SSL::CertificateRequest cnabdabpdu0e-12-37 at '/var/lib/puppet/ssl/ca/requests/cnabdabpdu0e-12-37.pem'
Notice: Removing file Puppet::SSL::CertificateRequest cnabdabpdu0e-12-37 at '/var/lib/puppet/ssl/certificate_requests/cnabdabpdu0e-12-37.pem'
apache2_invoke: Enable site puppetmaster
Setting up ruby-i18n (0.7.0-2) ...
Setting up ruby-atomic (1.1.16-2build5) ...
Setting up ruby-thread-safe (0.3.5-3) ...
Setting up ruby-tzinfo (1.2.2-1) ...
Setting up ruby-activesupport (2:4.2.6-1) ...
Setting up ruby-blankslate (3.1.3-1) ...
Setting up ruby-builder (3.2.2-4) ...
Setting up ruby-activemodel (2:4.2.6-1) ...
Setting up ruby-arel (6.0.3-2) ...
Setting up ruby-activerecord (2:4.2.6-1) ...
Setting up ruby-activerecord-deprecated-finders (1.0.4-1) ...
Setting up ruby-selinux (2.4-3build2) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21.4) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.35-0ubuntu2) ...

######备注：
从安装软件包过程可以看出软件在安装时自动调用passenger、ssl模块，并为自己注册了证书。软件包安装完成之后会自动启动apache2服务，启动8140端口。

root@xxx~#: puppet  cert list -a
+ "xxxx" (SHA256) 76:72:24:FC:11:2A:BC:EE:B4:32:90:5C:86:80:DC:C6:F5:37:50:DE:BA:AF:99:FE:F6:92:B7:2F:E4:74:F7:47

业务场景

场景一：基础安装使用

当master安装好之后，服务自动启动，自动生成默认配置文件，服务就可以使用了

root@xxx:~# tree  /etc/puppet/
/etc/puppet/
├── auth.conf
├── etckeeper-commit-post
├── etckeeper-commit-pre
├── fileserver.conf
├── manifests
├── modules
└── puppet.conf

2 directories, 5 files

也可网页访问，返回如下界面：https://puppetmasterip:8140
在这里插入图片描述

场景二：基于场景一，修改数据存储目录

由于puppet存放认证文件和其他相关数据目录默认在/var/lib/puppet目录；数据存放在根目录下，由于业务特性puppet管理主机肯定不止几台或者几十台。所以需要提考虑存放数据的目录变更到空间更大的数据盘存放，最好不要存放在根目录。

停服务

未防止操作时有新的数据写入，所以先停止puppetmaster

root@xxx:~# /etc/init.d/apache2 stop                ##puppetmaster服务由apache启动，所以直接停止apache2即可
[ ok ] Stopping apache2 (via systemctl): apache2.service.

改配置

修改puppet.conf
配置文件如下：

root@xxx~#:  /etc/puppet# cat puppet.conf
[main]
#logdir=/var/log/puppet
logdir=/work/app/puppet/log

#vardir=/var/lib/puppet
vardir=/work/app/puppet

ssldir=/work/app/puppet/ssl
#ssldir=/var/lib/puppet/ssl

rundir=/run/puppet
factpath=$vardir/lib/facter
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post
autosign=true
autosign = /etc/puppet/autosign.conf
environmentpath = $confdir/environments       

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN 
ssl_client_verify_header = SSL_CLIENT_VERIFY
certname = pupprtmaster_hostname                                            ##这里需要修改成自己服务器主机名
ca      =  true

修改主机名映射：

vim  /etc/hosts
.....
10.xx.xx.xx.xx   server_hostname

修改rack配置文件config.ru

root@xxx~#: vim  /usr/share/puppet/rack/puppetmasterd/config.ru 
# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $LOAD_PATH.unshift('/opt/puppet/lib')

$0 = "master"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"

# Rack applications typically don't start as root.  Set --confdir and --vardir
# to prevent reading configuration from ~puppet/.puppet/puppet.conf and writing
# to ~puppet/.puppet
ARGV << "--confdir" << "/etc/puppet"
#ARGV << "--vardir"  << "/var/lib/puppet"
ARGV << "--vardir"  << "/work/app/puppet"         ###修改位置

# NOTE: it's unfortunate that we have to use the "CommandLine" class
#  here to launch the app, but it contains some initialization logic
#  (such as triggering the parsing of the config file) that is very
#  important.  We should do something less nasty here when we've
#  gotten our API and settings initialization logic cleaned up.
#
# Also note that the "$0 = master" line up near the top here is
#  the magic that allows the CommandLine class to know that it's
#  supposed to be running master.
#
# --cprice 2012-05-22

require 'puppet/util/command_line'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Util::CommandLine.new.execute

修改puppetmaster.conf

root@xxx~#: vim /etc/apache2/sites-available/puppetmaster.conf
# This Apache 2 virtual host config shows how to use Puppet as a Rack
# application via Passenger. See
# http://docs.puppetlabs.com/guides/passenger.html for more information.

# You can also use the included config.ru file to run Puppet with other Rack
# servers instead of Passenger.
#Add defaultCharset=utf-8

AddDefaultCharset utf8
# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120

Listen 8140

<VirtualHost *:8140>
        SSLEngine on
        SSLProtocol             ALL -SSLv2 -SSLv3
        SSLCipherSuite          EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
        SSLHonorCipherOrder     on

#        SSLCertificateFile      /var/lib/puppet/ssl/certs/xxxxxx.pem            ####把xxxxxx替换成自己主机名
        SSLCertificateFile      /work/app/puppet/ssl/certs/xxxxxx.pem

#       SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/xxxxxx.pem     
        SSLCertificateKeyFile   /work/app/puppet/ssl/private_keys/xxxxxx.pem

#        SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
        SSLCertificateChainFile /work/app/puppet/ssl/certs/ca.pem

#        SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem
        SSLCACertificateFile    /work/app/puppet/ssl/certs/ca.pem

        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.
#        SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
        SSLCARevocationFile     /work/app/puppet/ssl/ca/ca_crl.pem

        # Apache 2.4 introduces the SSLCARevocationCheck directive and sets it to none
        # which effectively disables CRL checking; if you are using Apache 2.4+ you must
        # specify 'SSLCARevocationCheck chain' to actually use the CRL.
        # SSLCARevocationCheck chain
        SSLVerifyClient optional
        SSLVerifyDepth  1
        # The `ExportCertData` option is needed for agent certificate expiration warnings
        SSLOptions +StdEnvVars +ExportCertData

        # This header needs to be set if using a loadbalancer or proxy
        RequestHeader unset X-Forwarded-For

        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

        DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
        RackBaseURI /
        <Directory /usr/share/puppet/rack/puppetmasterd/>
                Options None
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>
</VirtualHost>

创建数据路径

root@xxx~#: mkdir  -p /work/app/puppet
root@xxx~#: chown puppet:puppet   /work/app/puppet
root@xxx~#: mv /var/lib/puppet/*    /work/app/puppet
root@xxx~#:  /etc/init.d/apache2 start              ###重启验证服务

场景三，基于场景二扩展成代理+两个后端虚机

停服务

root@xxx~#: /etc/init.d/apache2 stop                ##puppetmaster服务由apache启动，所以直接停止apache2即可
[ ok ] Stopping apache2 (via systemctl): apache2.service.

创建配置文件

创建proxy配置文件

root@xxx~#: vim /etc/apache2/sites-enabled/puppetmaster_proxy.conf
# This Apache 2 virtual host config shows how to use Puppet as a Rack
# application via Passenger. See
# http://docs.puppetlabs.com/guides/passenger.html for more information.

# You can also use the included config.ru file to run Puppet with other Rack
# servers instead of Passenger.

# you probably want to tune these settings

PassengerHighPerformance on
PassengerMaxPoolSize 108
PassengerPoolIdleTime 1500
PassengerMaxRequests 1000
PassengerStatThrottleRate 120



<Proxy balancer://puppetmaster>
  BalancerMember http://127.0.0.1:8141
  BalancerMember http://127.0.0.1:8142
</Proxy>

Listen 8140

<VirtualHost *:8140>
        SSLEngine on
        SSLProtocol             ALL -SSLv2 -SSLv3
        SSLCipherSuite          EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA
        
        SSLHonorCipherOrder     on

#        SSLCertificateFile      /var/lib/puppet/ssl/certs/xxxxxx.pem      
        SSLCertificateFile      /work/app/puppet/ssl/certs/xxxxxx.pem   ##替换成自己的主机名

#       SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/xxxxxx.pem
        SSLCertificateKeyFile   /work/app/puppet/ssl/private_keys/xxxxxx.pem

#        SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
        SSLCertificateChainFile /work/app/puppet/ssl/certs/ca.pem

#        SSLCACertificateFile    /var/lib/puppet/ssl/certs/ca.pem
        SSLCACertificateFile    /work/app/puppet/ssl/certs/ca.pem

        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.

#        SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
        SSLCARevocationFile     /work/app/puppet/ssl/ca/ca_crl.pem

        # Apache 2.4 introduces the SSLCARevocationCheck directive and sets it to none
        # which effectively disables CRL checking; if you are using Apache 2.4+ you must
        # specify 'SSLCARevocationCheck chain' to actually use the CRL.
        
#        SSLCARevocationCheck none

        SSLVerifyClient optional
        SSLVerifyDepth  1
        # The `ExportCertData` option is needed for agent certificate expiration warnings
        SSLOptions +StdEnvVars +ExportCertData

        # This header needs to be set if using a loadbalancer or proxy
        RequestHeader unset X-Forwarded-For

        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e


        <Location />
          SetHandler balancer-manager
          Order allow,deny
          Allow from all
        </Location>

       ProxyPass / balancer://puppetmaster/
       ProxyPassReverse / balancer://puppetmaster/
       ProxyPreserveHost On

  # log settings
       ErrorLog /var/log/apache2/balancer_error.log
       CustomLog /var/log/apache2/balancer_access.log combined
       CustomLog /var/log/apache2/balancer_ssl_requests.log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"



#        DocumentRoot /usr/share/puppet/rack/puppetmasterd/public/
#        RackBaseURI /
#        <Directory /usr/share/puppet/rack/puppetmasterd/>
#                Options None
#                AllowOverride None
#                Order allow,deny
#                allow from all
#        </Directory>
</VirtualHost>

创建后端虚机
backend1

root@xxx~#:  vim /etc/apache2/sites-enabled/puppetmaster_backend_8141.conf
Listen 8141

<VirtualHost 127.0.0.1:8141>
        SSLEngine off
        SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1
        SetEnvIf X-SSL-Client-DN "(.*)" SSL_CLIENT_S_DN=$1

        PassengerEnabled On
        DocumentRoot /usr/share/puppet/rack/puppetmasterd_backend_8141/public/
        RackBaseURI /
        <Directory /usr/share/puppet/rack/puppetmasterd_backend_8141/>
                Options None
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>
        # log settings
        ErrorLog /var/log/apache2/puppetmasterd_backend_8141.error.log
        CustomLog /var/log/apache2/puppetmasterd_backend_8141_access.log combined

</VirtualHost>

backend2

root@xxx~#:  vim /etc/apache2/sites-enabled/puppetmaster_backend_8142.conf
Listen 8142

<VirtualHost 127.0.0.1:8142>
        SSLEngine off
        SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1
        SetEnvIf X-SSL-Client-DN "(.*)" SSL_CLIENT_S_DN=$1

        PassengerEnabled On
        DocumentRoot /usr/share/puppet/rack/puppetmasterd_backend_8142/public/
        RackBaseURI /
        <Directory /usr/share/puppet/rack/puppetmasterd_backend_8142/>
                Options None
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>
        # log settings
        ErrorLog /var/log/apache2/puppetmasterd_backend_8142.error.log
        CustomLog /var/log/apache2/puppetmasterd_backend_8142_access.log combined

</VirtualHost>

root@xxx~#:  mkdir -p /usr/share/puppet/rack/puppetmasterd_backend_814{1,2}/{public,tmp}
root@xxx~#:  cp  /usr/share/puppet/rack/puppetmasterd/config.ru /usr/share/puppet/rack/puppetmasterd_backend_814{1,2}/
root@xxx~#:  chown puppet:puppet   /usr/share/puppet/rack/puppetmasterd_backend_814{1,2}/config.ru

加载模块

root@xxx~#:   cd  /etc/apache2/mods-enabled
root@xxx~#:   ln -s ../mods-available/proxy.conf proxy.conf
root@xxx~#:   ln -s ../mods-available/proxy.load proxy.load
root@xxx~#:   ln -s ../mods-available/proxy_balancer.conf proxy_balancer.conf 
root@xxx~#:   ln -s ../mods-available/proxy_balancer.load proxy_balancer.load
root@xxx~#:   ln -s ../mods-available/proxy_http.load proxy_http.load
root@xxx~#:   ln -s ../mods-available/slotmem_shm.load  slotmem_shm.load
root@xxx~#:   ln -s ../mods-available/lbmethod_byrequests.load  lbmethod_byrequests.load

修改puppet.conf文件

root@xxx~#:  vim /etc/puppet/puppet.conf
[main]
#logdir=/var/log/puppet
logdir=/work/app/puppet/log

#vardir=/var/lib/puppet
vardir=/work/app/puppet

ssldir=/work/app/puppet/ssl
#ssldir=/var/lib/puppet/ssl

rundir=/run/puppet
factpath=$vardir/lib/facter
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post
autosign=true
autosign = /etc/puppet/autosign.conf
environmentpath = $confdir/environments       

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
#ssl_client_header = SSL_CLIENT_S_DN        ###注释这行
#ssl_client_verify_header = SSL_CLIENT_VERIFY  ###注释这行
certname = pupprtmaster_hostname                                            ##这里需要修改成自己服务器主机名
ca      =  true

启动服务

root@xxx~#:    /etc/init.d/apache2 start

问题分析

服务器无法启动

日志显示报错：

apache2: Could not reliably determine the server’s fully qualified domain name, using 10.xx.xx.xx. Set the ‘ServerName’ directive globally to suppress this message

Output
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: active (running) since Wed 2024-07-15 14:30:03 UTC; 33min ago
  Process: 34 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 46 (apache2)
    Tasks: 55 (limit: 2344)
   CGroup: /system.slice/apache2.service
           ├─46 /usr/sbin/apache2 -k start
           ├─47 /usr/sbin/apache2 -k start
           └─48 /usr/sbin/apache2 -k start

Jul 15 14:30:03 68e2cf19f3f1 systemd[1]: Starting The Apache HTTP Server...
Jul  1514:30:03 68e2cf19f3f1 apachectl[34]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.xx.xx.xx. Set the 'ServerName' directive globally to suppress this message
Jul 15 14:30:03 68e2cf19f3f1 systemd[1]: Started The Apache HTTP Server.

临时解决方案：

root@xxx~#: vim /etc/apache2/sites-enabled/puppetmaster_proxy.conf
ServerName  10.xx.xx.xx

2.模块缺失：

slotmem_shm
日志显示报错：
Failed to lookup provider ‘shm’ for ‘slotmem’: is mod_slotmem_shm loaded？？

解决方案:

root@xxx~#:   cd  /etc/apache2/mods-enabled
root@xxx~#:   ln -s ../mods-available/slotmem_shm.load  slotmem_shm.load

byrequests
日志显示报错：
Cannot find LB Method: byrequests
[proxy_balancer:emerg] [pid 503886:tid 140037069854592] (22)Invalid argument: AH01183: Cannot share balancer
解决方案

root@xxx~#:   cd  /etc/apache2/mods-enabled
root@xxx~#:   ln -s ../mods-available/lbmethod_byrequests.load  lbmethod_byrequests.load

3.agent同步报错：

报错信息：
Error: Could not request certificate: Error 403 on SERVER: Forbidden request: localhost(127.0.0.1) access to /certificate_revocation_list/ca [find] at :122
解决方案
注释下边两行

root@xxx~#:  vim /etc/puppet/puppet.conf
[master]
#ssl_client_header = SSL_CLIENT_S_DN        ###注释这行
#ssl_client_verify_header = SSL_CLIENT_VERIFY  ###注释这行

root@xxx~#:   /etc/init.d/apache2  restart