使用nginx对grpc进行反向代理和负载均衡

5 篇文章 0 订阅
2 篇文章 0 订阅

前言

最近项目中需要给我们的grpc服务进行反向代理和负载均衡。故将这期间查阅的资料进行了整理记录

相关链接

介绍nginx负载均衡:https://blog.csdn.net/qq_51574197/article/details/117408823
nginx对grpc进行反向代理:https://blog.csdn.net/Dearmark/article/details/80066819
netty中http2的设计:https://my.oschina.net/u/1261452/blog/2997252
springboot对http2的支持:https://blog.csdn.net/luxinfeng666/article/details/107067260/
tomcat对http2的支持:https://www.cnblogs.com/xifenglou/p/9394948.html
nginx配置ssl和http2:http://www.web3.xin/index/article/567.html
springboot配置https(SSL证书):https://www.jb51.net/article/233123.htm#_lab2_1_4
springboot配置SSL:https://blog.csdn.net/csj50/article/details/123206007
keytool生成证书:https://blog.csdn.net/Guesshat/article/details/123024693
nginx对gprc进行负载均衡的java小demo:https://blog.csdn.net/weixin_42691149/article/details/119214146?utm_term=nginx-ingress%E8%A7%A3%E5%86%B3grpc%E9%95%BF%E8%BF%9E%E6%8E%A5%E8%B4%9F%E8%BD%BD%E4%B8%8D%E5%9D%87%E5%A4%B1%E8%B4%A5&utm_medium=distribute.pc_aggpage_search_result.none-task-blog-2allsobaiduweb~default-1-119214146-null-null&spm=3001.4430
nginx反向代理grpc长连接服务最佳实践:https://blog.csdn.net/zry5858/article/details/123609086
nginx安装http2模块:https://blog.csdn.net/ChinaHacker995/article/details/113538789
nginx安装http2模块:https://blog.csdn.net/weixin_43885541/article/details/121009451
postman支持grpc:https://blog.csdn.net/qq_60168783/article/details/123339858

先了解下grpc的一些前置条件

上配置

1. grpc

#此配置为负载均衡配置
upstream datacenter_load {
        server 127.0.0.1:9090;
}

server {
    #此处配置上http2,因为gprc是基于http2的
    listen 88 http2;
    server_name test.datacenter.technocore.network;
    #此处貌似不能像http转https一样,会报错,所以注释掉单独再写个grpcs的server配置
    #return 301 grpc://$server_name:880/$request_uri;

    location / {
        #此处配置grpc服务的反向代理
        grpc_pass  grpc://datacenter_load;
        grpc_set_header Host $host;
        grpc_set_header X-Real-IP $remote_addr;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

}

2. grpc + ssl

#此配置为负载均衡配置
upstream datacenter_load2 {
        server 127.0.0.1:9090;
}

server {
	#此处配置上http2和ssl,因为gprc是基于http2的
    listen 880 ssl http2;
    server_name test.technocore.network;
    #此处配置ssl证书的路径
    ssl_certificate /opt/metisnetwork/nginx/ssl/test.technocore.network.pem;
    ssl_certificate_key /opt/metisnetwork/nginx/ssl/test.technocore.network.key;

    location / {
    	#此处配置grpc服务的反向代理
        grpc_pass  grpc://datacenter_load2;
        grpc_set_header Host $host;
        grpc_set_header X-Real-IP $remote_addr;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

配置完之后验证

方法1,使用postman调用grpc

postman支持grpc:https://blog.csdn.net/qq_60168783/article/details/123339858

可能会遇到的报错信息:
Internal errorOperation couldn’t be completed because of an internal error at the server.Received RST_STREAM with code 2 triggered by internal client error: Protocol error
在这里插入图片描述
分析:
1.博主现在postman版本是Postman v9.19.0,暂时是不支持grpc+ssl的
2.可能是端口错误了

方法2,使用Java代码调用grpc

1.grpc客户端demo

package com.platon.metis.storage.grpc;


import com.platon.metis.storage.grpc.lib.api.ListPowerSummaryResponse;
import com.platon.metis.storage.grpc.lib.api.ResourceServiceGrpc;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import lombok.extern.slf4j.Slf4j;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.junit4.SpringRunner;




@RunWith(SpringRunner.class)
@Slf4j
public class GrpcClientDemo {




    @Test
    public void listPowerSummary() {
        log.info("start to test listPowerSummary()...");


        //普通的grpc需要使用明文,所以usePlaintext参数,表示明文进行请求。
        /**
         * 否则会报错:
         */
        ManagedChannel channel = ManagedChannelBuilder.forAddress("test.technocore.network",88)
                .usePlaintext()
                .build();
        


        com.google.protobuf.Empty request = com.google.protobuf.Empty.newBuilder().build();
        ResourceServiceGrpc.ResourceServiceBlockingStub resourceServiceBlockingStub = ResourceServiceGrpc.newBlockingStub(channel);
        ListPowerSummaryResponse response = resourceServiceBlockingStub.listPowerSummary(request);


        log.info("listPowerSummary(), response:{}", response);
    }


}
可能遇到的报错:

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception

原因:

构建channel的时候需要加上usePlaintext(),即表示通讯要使用明文通讯。

         /*
         * io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
         * Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
         *
         *     at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
         *     at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
         *     at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
         *     at com.platon.metis.storage.grpc.lib.api.ResourceServiceGrpc$ResourceServiceBlockingStub.listPowerSummary(ResourceServiceGrpc.java:522)
         *     at com.platon.metis.storage.grpc.GrpcClientDemo.listPowerSummary(GrpcClientDemo.java:39)
         *     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         *     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         *     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         *     at java.lang.reflect.Method.invoke(Method.java:498)
         *     at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
         *     at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
         *     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
         *     at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
         *     at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
         *     at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
         *     at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
         *     at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
         *     at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
         *     at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
         *     at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
         *     at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
         *     at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
         *     at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
         *     at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
         *     at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
         *     at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
         *     at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
         *     at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
         *     at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
         *     at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
         *     at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
         *     at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
         *     at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
         *     at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
         *     at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
         *     at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
         * Caused by: io.grpc.netty.shaded.io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 000012040000000000000300000080000400010000000500ffffff0000040800000000007fff00000000080700000000000000000000000001
         *     at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1232)
         *     at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1300)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
         *     at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         *     at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
         *     at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
         *     at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
         *     at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
         *     at java.lang.Thread.run(Thread.java:748)
         *
         */

2. grpc+ssl客户端demo

package com.platon.metis.storage.grpc;


import com.platon.metis.storage.grpc.lib.api.ListPowerSummaryResponse;
import com.platon.metis.storage.grpc.lib.api.ResourceServiceGrpc;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import lombok.extern.slf4j.Slf4j;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.junit4.SpringRunner;




@RunWith(SpringRunner.class)
@Slf4j
public class GrpcClientDemo {




    @Test
    public void listPowerSummary() {
        log.info("start to test listPowerSummary()...");

        //grpcs需要使用密文,所以要去掉usePlaintext
        /**
         * 否则会报错:
         */
        ManagedChannel channel = ManagedChannelBuilder.forAddress("test.technocore.network",880)
//                .usePlaintext()
                .build();


        com.google.protobuf.Empty request = com.google.protobuf.Empty.newBuilder().build();
        ResourceServiceGrpc.ResourceServiceBlockingStub resourceServiceBlockingStub = ResourceServiceGrpc.newBlockingStub(channel);
        ListPowerSummaryResponse response = resourceServiceBlockingStub.listPowerSummary(request);


        log.info("listPowerSummary(), response:{}", response);
    }


}
可能遇到的报错:

io.grpc.StatusRuntimeException: INTERNAL: http2 exception

原因:

构建channel的时候需要去掉usePlaintext(),即表示通讯要密文加密。

        /**
         * 否则会报错:
         * io.grpc.StatusRuntimeException: INTERNAL: http2 exception
         *
         *     at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
         *     at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
         *     at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
         *     at com.platon.metis.storage.grpc.lib.api.ResourceServiceGrpc$ResourceServiceBlockingStub.listPowerSummary(ResourceServiceGrpc.java:522)
         *     at com.platon.metis.storage.grpc.GrpcClientDemo.listPowerSummary(GrpcClientDemo.java:111)
         *     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         *     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         *     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         *     at java.lang.reflect.Method.invoke(Method.java:498)
         *     at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
         *     at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
         *     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
         *     at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
         *     at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
         *     at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
         *     at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
         *     at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
         *     at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
         *     at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
         *     at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
         *     at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
         *     at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
         *     at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
         *     at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
         *     at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
         *     at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
         *     at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
         *     at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
         *     at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
         *     at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
         *     at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
         *     at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
         *     at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
         *     at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
         *     at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
         *     at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
         * Caused by: io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
         *     at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:108)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:338)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:239)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
         *     at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
         *     at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
         *     at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
         *     at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
         *     at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
         *     at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
         *     at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
         *     at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
         *     at java.lang.Thread.run(Thread.java:748)
         */
  • 4
    点赞
  • 18
    收藏
    觉得还不错? 一键收藏
  • 4
    评论
评论 4
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值