【在Cloud Foundry 上添加ECHO服务】
官方教程:https://github.com/cloudfoundry/oss-docs/tree/master/vcap/adding_a_system_service
不清楚是CF版本更新的原因还是别的什么,反正按照官方给出的教程添加echo是无法成功的。问题就在于官方提供配置文件缺少一些参数造成的。
下面的内容重点是熟悉这些配置信息是怎么回事。
首先我准备了两个节点:
【Node1】
预安装:cloud_controller、router 、health_manager、mongodb_gateway 、mysql_gateway、redis_gateway、mongodb_node、stager、redis_node。
IP地址:192.168.1.166
准备添加:echo_gateway
【Node2】
预安装:mysql、dea
IP地址:192.168.1.101
准备添加:echo_node
【添加gateway】
【步骤1】
内容:修改~/cloudfoundry/.deployments/devbox/config/vcap_components.json文件,在其中添加echo_gateway,如下
{"components":["router","health_manager","mongodb_gateway","cloud_controller","mysql_gateway","redis_gateway","mongodb_node","stager","redis_node","echo_gateway"]} |
功能:这个文件夹下存储的就是“开机启动项”,每次重启CF,它都是根据这个文件中的内容选择开启哪些内容。
【步骤2】
内容:修改vim cloudfoundry/.deployments/center/config/cloud_controller.yml文件,添加如下内容:
# Services we provide, and their tokens. Avoids bootstrapping DB. builtin_services: redis: token: changeredistoken mongodb: token: changemongodbtoken mysql: token: changemysqltoken echo: token: changeechotoken service_proxy: token: ["changebrokertoken"] |
功能:CC在与Services的gateway交互期间,需要添加1个令牌,两者在初始化配置文件中都需要一致。可以查看【步骤6】中echo_gateway的token配置信息。
【步骤3】
内容:修改~/cloudfoundry/vcap/services/tools/misc/bin/nuke_service.rb,添加如下内容:
default_configs = { :mongodb => File.expand_path("../../mongodb/config/mongodb_gateway.yml", __FILE__), :redis => File.expand_path("../../redis/config/redis_gateway.yml", __FILE__), :mysql => File.expand_path("../../mysql/config/mysql_gateway.yml", __FILE__), :neo4j => File.expand_path("../../neo4j/config/neo4j_gateway.yml", __FILE__), :vblob => File.expand_path("../../vblob/config/vblob_gateway.yml", __FILE__), :echo => File.expand_path("../../echo/config/echo_gateway.yml", __FILE__), } |
功能:这个是一个工具,用于从CF上删除一个service ,不能算作是添加Service中的一个环节。
【步骤4】
内容:编辑~/cloudfoundry/vcap/dev_setup/lib/vcap_components.rb文件,添加如下内容
## services: gateways & nodes %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch filesystem echo).each do |service| ServiceComponent.register("#{service}_gateway") end
%w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch ).each do |service| ServiceComponent.register("#{service}_node") end
%w(redis mysql mongodb postgresql).each do |service| ServiceComponent.register("#{service}_worker") end |
功能:看语句可以猜到,在启动过程中需要对每个组件进行注册操作。我们查看~/cloudfoundry/vcap/dev_setup/bin/vcap,可以看到这样一句,在【步骤1】中我们也看到启动的时候使用的vcap,而vcap又需要vcap_components库,这个库就是我们这个步骤中修改的文件:
require File.expand_path(File.join(“..”, “lib”, “vcap_components”), File.dirname(__FILE__)) |
【步骤5】
内容:执行下列命令
cd ~/cloudfoundry/vcap/services/echo;source $HOME/.cloudfoundry_deployment_profile && bundle package |
功能:【--?--应该是编译echo,或者是在解决依赖包关系,这部分内容暂时不太明白】
注:这部分内容必须分析,以后我们需要手动写这部分的内容
【步骤6】
内容:将 ~/cloudfoundry/vcap/services/echo/config/echo_gateway.yml文件拷贝到 ~/cloudfoundry/.deployments/mysqldea/config/目录下。并如下修改
--- cloud_controller_uri: api.vcap.me service: name: echo version: "1.0" description: 'Echo service' plans: ['free'] default_plan: free tags: ['echo', 'echo-1.0', 'echobased', 'demo'] timeout: 15 supported_versions: ['1.0'] version_aliases: "current": "1.0" ip_route: 192.168.1.166 index: 0 token: changeechotoken logging: level: debug mbus: nats://nats:nats@192.168.1.166:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2 |
功能:第一、如下所示,启动时候使用的配置信息在~/cloudfoundry/.deployments/mysqldea/config/目录下,所以我们需要将配置文件复制到该目录下。
Executing /home/vm/cloudfoundry/.deployments/center/deploy/rubies/ruby-1.9.2-p180/bin/ruby /home/vm/cloudfoundry/vcap/dev_setup/bin/vcap restart cloud_controller router health_manager mongodb_gateway mysql_gateway redis_gateway mongodb_node stager redis_node echo_gateway -c /home/vm/cloudfoundry/.deployments/center/config -v /home/vm/cloudfoundry -l /home/vm/cloudfoundry/.deployments/center/log |
第二、在配置信息方面,官网上给的有误,少了几个参数,后面我们会介绍如果缺少某些参数的时候我们要如何排错
【添加echo_node】
【步骤1】
内容:重复【添加echo_node】中的【步骤1】,但是这次添加的是“echo_node”
{"components":["dea","mysql_node","echo_node"]} |
功能:添加启动项
【步骤2】
内容:重复【添加echo_node】中的【步骤3】。
功能:与【添加echo_node】中的【步骤3】一致
【步骤3】
内容:重复【添加echo_node】中的【步骤4】,只是这次添加echo的位置不同
## services: gateways & nodes %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch filesystem).each do |service| ServiceComponent.register("#{service}_gateway") end
%w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch echo).each do |service| ServiceComponent.register("#{service}_node") end
%w(redis mysql mongodb postgresql).each do |service| ServiceComponent.register("#{service}_worker") end |
功能:与【添加echo_node】中的【步骤4】一致
【步骤4】
内容:重复【添加echo_node】中的【步骤5】
功能:与【添加echo_node】中的【步骤5】一致
【步骤5】
内容:将 ~/cloudfoundry/vcap/services/echo/config/echo_node.yml文件拷贝到 ~/cloudfoundry/.deployments/mysqldea/config/目录下。并如下修改
--- plan: free capacity: 100 local_db: sqlite3:/var/vcap/services/echo/echo_node.db mbus: nats://nats:nats@192.168.1.166:4222 base_dir: /var/vcap/services/echo/ index: 0 logging: level: debug pid: /var/vcap/sys/run/echo_node.pid node_id: echo_node_0 port: 5002 host: 192.168.1.101 |
功能:与与【添加echo_node】中的【步骤6】一致
【参数分析】
【分析1】
考虑【添加gateway】中【步骤1】中添加的参数
准备工作:将下列中灰色部分删除
{"components":["router","health_manager","mongodb_gateway","cloud_controller","mysql_gateway","redis_gateway","mongodb_node","stager","redis_node" |
结果:启动时不会检查echo_gateway,直接忽略。
结果分析:我们查看vim /home/vm/cloudfoundry/vcap/dev_setup/bin/vcap_dev,在最底部就可以看到,启动过程中的参数就包含了vcap_components["components"].join(" ")
puts "Using cloudfoundry config from #{deployment_config_path}" exec_cmd("#{ruby_binary} #{vcap_launch} #{command} #{vcap_components["components"].join(" ")} -c #{deployment_config_path} -v #{vcap_home} -l #{deployment_info["deployment_log_path"]}") |
而我们向面追查发现vcap_components的赋值
begin vcap_components = JSON.parse(File.read(Deployment.get_vcap_config_file(deployment_config_path))) deployment_info = JSON.parse(File.read(Deployment.get_deployment_info_file(deployment_config_path))) rescue => e STDERR.puts "#{e.inspect}. Could not parse deployment config files . Please check your deployment." exit 1 end |
我们直接将其打印出来:vcap_config_file/home/vm/cloudfoundry/.deployments/center/config/vcap_components.json。所以我们现在就知道了这个文件下存放着启动项的信息
【分析2】
考虑【添加gateway】中【步骤1】中的token参数,我们将分析gateway与CC之间的token是怎么回事
【分析2.1】
准备:在center节点上,我们编辑文件~/ cloudfoundry/.deployments/center/config/cloud_controller.yml,将token部分去除
# Services we provide, and their tokens. Avoids bootstrapping DB. builtin_services: redis: token: changeredistoken mongodb: token: changemongodbtoken mysql: token: changemysqltoken # echo: # token: changeechotoken service_proxy: token: ["changebrokertoken"] |
结果:正常运行,【--?--现在仅仅是log日志看着没问题,不清楚会不会影响正常使用】
CC的日志信息:
DEBUG -- Create service request: {"label":"echo-1.0","url":"http://192.168.1.166:35980","plans":["free"],"cf_plan_id":null,"tags":["echo","echo-1.0","echobased","demo"],"active":true,"description":"Echo service","plan_options":null,"acls":null,"timeout":15,"provider":null,"default_plan":"free","supported_versions":["1.0"],"version_aliases":{"current":"1.0"}} Found svc = #<Service id: 4, label: "echo-1.0", url: "http://192.168.1.166:32781", token: "changeechotoken", name: "echo", version: "1.0", description: "Echo service", info_url: nil, tags: ["echo", "echo-1.0", "echobased", "demo"], plans: ["free"], plan_options: nil, binding_options: nil, acls: nil, active: true, created_at: "2012-10-30 04:51:15", updated_at: "2012-10-30 05:23:56", timeout: 15, cf_plan_id: nil, provider: nil, supported_versions: ["1.0"], version_aliases: {"current"=>"1.0"}, default_plan: "free"> |
echo_gateway的日志信息:
INFO -- Sending info to cloud controller: http://api.vcap.me/services/v1/offerings INFO -- Fetching handles from cloud controller @ http://api.vcap.me/services/v1/offerings/echo-1.0/handles DEBUG -- [EchoaaS-Provisioner] Connected to node mbus.. DEBUG -- [EchoaaS-Provisioner] Received node announcement: {"available_capacity":100,"capacity_unit":1,"id":"echo_node_0","plan":"free","supported_versions":["1.0"]} INFO -- Successfully registered with cloud controller INFO -- Successfully fetched handles |
结果分析:说明修改~/ cloudfoundry/.deployments/center/config/cloud_controller.yml中的token没有起作 用。但是这个Token真的没有用么,见【分析1.3】
【分析2.2】
准备:修改vim cloudfoundry/.deployments/center/config/echo_gateway.yml,将token部分注释掉
--- cloud_controller_uri: api.vcap.me service: name: echo version: "1.0" description: 'Echo service' plans: ['free'] default_plan: free tags: ['echo', 'echo-1.0', 'echobased', 'demo'] timeout: 15 supported_versions: ['1.0'] version_aliases: "current": "1.0" ip_route: 192.168.1.166 index: 0 #token: changeechotoken logging: level: debug mbus: nats://nats:nats@192.168.1.166:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2 |
结果:在启动过程中提示“Token missing”,这个的log是echo_gateway的日志里面提示的信息,而在CC的log中找不到任何echo相关的字眼。
vm@vm-virtual-machine:~$ ~/cloudfoundry/vcap/dev_setup/bin/vcap_dev restart Targeting deployment "center" with cloudfoundry home "/home/vm/cloudfoundry" ……………… echo_gateway : STOPPED LOG: Couldn't read config file: Token missing |
结果分析:说明echo_gateway.yml中的token是必须写入的。
【分析2.3】
准备:修改vim cloudfoundry/.deployments/center/config/echo_gateway.yml,将token部分修改
--- cloud_controller_uri: api.vcap.me service: name: echo version: "1.0" description: 'Echo service' plans: ['free'] default_plan: free tags: ['echo', 'echo-1.0', 'echobased', 'demo'] timeout: 15 supported_versions: ['1.0'] version_aliases: "current": "1.0" ip_route: 192.168.1.166 index: 0 token: changemyechotoken logging: level: debug mbus: nats://nats:nats@192.168.1.166:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2 |
结果:
我们查看echo_gateway的日志发现:
[2012-10-30 13:57:24.994541] echo_gateway - pid=21159 tid=63c7 fid=5643 ERROR -- Failed registering with cloud controller, status=403 [2012-10-30 13:57:25.002911] echo_gateway - pid=21159 tid=63c7 fid=5643 ERROR -- Failed fetching handles, status=403 |
结果分析:CC配置文件中的token是需要的,一旦不一致,echo_gateway就会出现错误。
至于为什么在【分析2.1】中却没有出现问题,这点就不得而知了
【分析3】
考虑【添加gateway】中【步骤3】中的添加的语句,我们将其注释掉;
准备工作:我们将下面灰色部分语句注释
default_configs = { :mongodb => File.expand_path("../../mongodb/config/mongodb_gateway.yml", __FILE__), :redis => File.expand_path("../../redis/config/redis_gateway.yml", __FILE__), :mysql => File.expand_path("../../mysql/config/mysql_gateway.yml", __FILE__), :neo4j => File.expand_path("../../neo4j/config/neo4j_gateway.yml", __FILE__), :vblob => File.expand_path("../../vblob/config/vblob_gateway.yml", __FILE__), #:echo => File.expand_path("../../echo/config/echo_gateway.yml", __FILE__), } |
结果:正常运行,查看了想到的日志文件,运行都正常。
结果分析:这个配置时不影响应用正常启动的,更加证明这个是一个工具。
【分析4】
考虑【添加gateway】中【步骤3】中的添加的参数,我们将其删除:
准备工作:如下面所示,将echo删除
## services: gateways & nodes %w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch filesystem ServiceComponent.register("#{service}_gateway") end
%w(redis mysql mongodb rabbitmq postgresql vblob neo4j memcached couchdb elasticsearch).each do |service| ServiceComponent.register("#{service}_node") end |
结果:如下图所示,在启动echo_gateway的过程中,因为无法注册,所以直接忽略。
vm@vm-virtual-machine:~$ ~/cloudfoundry/vcap/dev_setup/bin/vcap_dev restart ……………… Executing /home/vm/cloudfoundry/.deployments/center/deploy/rubies/ruby-1.9.2-p180/bin/ruby /home/vm/cloudfoundry/vcap/dev_setup/bin/vcap restart cloud_controller router health_manager mongodb_gateway mysql_gateway redis_gateway mongodb_node stager redis_node echo_gateway -c /home/vm/cloudfoundry/.deployments/center/config -v /home/vm/cloudfoundry -l /home/vm/cloudfoundry/.deployments/center/log Skipping invalid component: echo_gateway_gateway Skipping invalid component: echo_gateway_node ……………… redis_node STOPPED Skipping invalid component: echo_gateway_gateway Skipping invalid component: echo_gateway_node Skipping invalid component: echo_gateway_gateway Skipping invalid component: echo_gateway_node cloud_controller RUNNING ……………… redis_node RUNNING |
结果分析:从配置的文件中,我们可以发现,echo需要注册后才能启动。
【分析5】
考虑【添加gateway】中【步骤5】中的添加的参数,我们将其删除:
准备工作:我们在配置完参数后,不执行bundle语句,直接启动
结果:我们发现,有如下的提示信息
vm@vm-virtual-machine:~$ ~/cloudfoundry/vcap/dev_setup/bin/vcap_dev restart Targeting deployment "center" with cloudfoundry home "/home/vm/cloudfoundry" ……………… redis_node : RUNNING echo_gateway : STOPPED LOG: /home/vm/cloudfoundry/.deployments/center/deploy/rubies/ruby-1.9.2-p180/lib/ruby/gems/1.9.1/gems/bundler-1.1.3/lib/bundler/spec_set.rb:90:in `block in materialize': Could not find ci_reporter-1.7.0 in any of the sources (Bundler::GemNotFound) ……………… from /home/vm/cloudfoundry/vcap/bin/../services/echo/bin/echo_gateway:7:in `<main>' |
结果分析:【--?--以后我们也要这样来?!】
【分析6】
考虑【添加gateway】中【步骤6】中的添加的参数,我们将试着一个个删除,看看会有什么样的表现
【分析6.1】
准备工作:删除ip_route: 192.168.1.166。
--- cloud_controller_uri: api.vcap.me service: name: echo version: "1.0" description: 'Echo service' plans: ['free'] default_plan: free tags: ['echo', 'echo-1.0', 'echobased', 'demo'] timeout: 15 supported_versions: ['1.0'] version_aliases: "current": "1.0"
index: 0 token: changeechotoken logging: level: debug mbus: nats://nats:nats@192.168.1.166:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2 |
结果:正常运行
结果分析:ip_route这个参数不是必须参数
【分析6.2】
准备工作:删除version部分内容
--- cloud_controller_uri: api.vcap.me service: name: echo version: "1.0" description: 'Echo service' plans: ['free'] default_plan: free tags: ['echo', 'echo-1.0', 'echobased', 'demo'] timeout: 15 supported_versions: ['1.0']
index: 0 token: changeechotoken logging: level: debug mbus: nats://nats:nats@192.168.1.166:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2 |
结果:
在echo_gateway的日志中可以看到
ERROR -- Failed registering with cloud controller, status=400 |
在CC的日志文件中可以看到
DEBUG -- Create service request: {"label":"echo-1.0","url":"http://192.168.1.166:37472","plans":["free"],"cf_plan_id":null,"tags":["echo","echo-1.0","echobased","demo"],"active":true,"description":"Echo service","plan_options":null,"acls":null,"timeout":15,"provider":null,"default_plan":"free","supported_versions":["1.0"],"version_aliases":null} ERROR -- Failure decoding service offering request: Field: version_aliases, Error: Missing field version_aliases |
结果分析:在添加服务的时候,需要确定参数:version_aliases。
【分析6.3】
准备工作
--- cloud_controller_uri: api.vcap.me service: name: echo version: "1.0" description: 'Echo service' plans: ['free'] default_plan: free tags: ['echo', 'echo-1.0', 'echobased', 'demo'] timeout: 15 version_aliases: "current": "1.0" index: 0 token: changeechotoken logging: level: debug mbus: nats://nats:nats@192.168.1.166:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2 |
结果:
在echo_gateway的日志中可以看到
ERROR -- Failed registering with cloud controller, status=400 |
在CC的日志文件中可以看到
DEBUG -- Create service request: {"label":"echo-1.0","url":"http://192.168.1.16 6:44294","plans":["free"],"cf_plan_id":null,"tags":["echo","echo-1.0","echobased","demo"],"active":true,"description":"Echo service","plan_op tions":null,"acls":null,"timeout":15,"provider":null,"default_plan":"free","supported_versions":null,"version_aliases":null} ERROR -- Failure decoding service offering request: Field: supported_versions, Error: Missing field supported_versions, Field: version_aliases, Error: Missing field version_aliases |
结果分析:
【分析6.4】
--- cloud_controller_uri: api.vcap.me service: name: echo version: "1.0" description: 'Echo service' plans: ['free'] default_plan: free tags: ['echo', 'echo-1.0', 'echobased', 'demo'] timeout: 15 supported_versions: ['1.0'] version_aliases: "current": "1.0" index: 0 token: changeechotoken logging: level: debug
mbus:nats://localhost:4222 pid: /var/vcap/sys/run/echo_service.pid node_timeout: 2 |
结果:在CC的日志中我们找不到任何和echo相关的内容,而在echo_gateway的日志文件中,我们可以找到
ERROR -- Exiting due to NATS error: Could not connect to server on nats://localhost:4222 ERROR -- Failed registering with cloud controller: ERROR -- Failed fetching handles: |
结果分析:CC请求Service需要CC在NATS上订阅了该services,然后这个service的gateway需要在NATS上进行注册以后,CC订阅到该service的信息,然后才会发送请求该service的请求。
【分析6】中的内容都是因为gateway配置错误而引起的,而gateway的配置文件需要配置那些信息呢?其实我们可以通过CC发送的request信息可以看出它需要读取那些参数信息,下表是mongodb的request经过整理的表格,我们整理了一部分必须的参数【--?--不排除其他参数必须】。
键 | 值 | 说明 |
"label" | "mongodb-1.8" | 请求的service的标签 |
"url" | "http://192.168.1.166:42208" | 【必须】链接的地址 |
"plans" | ["free"] |
|
"cf_plan_id" | null |
|
"tags" | ["nosql","document"] | 对于MongoDB的tags |
"active" | true |
|
"description" | "MongoDB NoSQL store" | 描述 |
"plan_options" | null |
|
"acls" | null |
|
"timeout" | 15 |
|
"provider" | null |
|
"default_plan" | "free" |
|
"supported_versions" | ["1.8","2.0"] | 【必须】可以支持的版本 |
"version_aliases" | {"next":"2.0","current":"1.8"} | 【必须】当前使用的版本,和备用的版本 |
从上面的表我们可能看的还是有些模糊,可以参考/home/vm/cloudfoundry/.deployments/center/deploy/rubies/ruby-1.9.2-p180/lib/ruby/gems/1.9.1/bundler/gems/vcap-services-base-e54a821c72f1/lib/base/asynchronous_service_gateway.rb的setup函数。我们可以发现下面的语句,发现很多参数都是匹配的:
43 def setup(opts) 44 missing_opts = REQ_OPTS.select {|o| !opts.has_key? o} 45 raise ArgumentError, "Missing options: #{missing_opts.join(', ')}" unless missing_opts.empty? 46 @service = opts[:service] 47 @token = opts[:token] 48 @logger = opts[:logger] || make_logger() 49 @cld_ctrl_uri = http_uri(opts[:cloud_controller_uri]) 50 @offering_uri = "#{@cld_ctrl_uri}/services/v1/offerings" 51 @provisioner = opts[:provisioner] 52 @hb_interval = opts[:heartbeat_interval] || 60 53 @node_timeout = opts[:node_timeout] 54 @handles_uri = "#{@cld_ctrl_uri}/services/v1/offerings/#{@service[:label]}/handles" 55 @handle_fetch_interval = opts[:handle_fetch_interval] || 1 56 @check_orphan_interval = opts[:check_orphan_interval] || -1 57 @double_check_orphan_interval = opts[:double_check_orphan_interval] || 300 58 @handle_fetched = false 59 @fetching_handles = false 60 @version_aliases = @service[:version_aliases] || {} 61 @svc_json = { 62 :label => @service[:label], 63 :url => @service[:url], 64 :plans => @service[:plans], 65 :cf_plan_id => @service[:cf_plan_id], 66 :tags => @service[:tags], 67 :active => true, 68 :description => @service[:description], 69 :plan_options => @service[:plan_options], 70 :acls => @service[:acls], 71 :timeout => @service[:timeout], 72 :provider => @service[:provider], 73 :default_plan => @service[:default_plan], 74 :supported_versions => @service[:supported_versions], 75 :version_aliases => @service[:version_aliases] 76 }.to_json |