Kubernetes生产实践系列之十九：ServiceMesh之使用Ingress Nginx暴露Kubernetes上的gRPC服务

Bertha 。 2023-03-13 05:47 6阅读 0赞

## **一、前言** ##

Ingress Nginx支持gRPC服务的暴露，低于0.21的版本使用如下的annotation支持gRPC：

nginx.ingress.kubernetes.io/grpc-backend: "true"

0.21版本以上使用如下annotation支持gRPC：

nginx.ingress.kubernetes.io/backend-protocol: "GRPC"

本文基于Ingress Ngixn版本0.30.0(quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0)进行部署和测试

转载自https://blog.csdn.net/cloudvtech

## **二、部署gRPC服务** ##

### 2.1 服务端代码和镜像构建 ###

**server.go **

package main
    
    import (
    	"context"
    	"fmt"
    	"os/signal"
    
    	"log"
    	"net"
    	"os"
    
    	"github.com/javiramos1/grpcapi"
    	"google.golang.org/grpc"
    	"google.golang.org/grpc/reflection"
    )
    
    type grpcServer struct{}
    
    func (*grpcServer) GrpcService(ctx context.Context, req *grpcapi.GrpcRequest) (*grpcapi.GrpcResponse, error) {
    	fmt.Printf("grpcServer %v\n", req)
    	name, _ := os.Hostname()
    
    	input := req.GetInput()
    	result := "Got input " + input + " server host: " + name
    	res := &grpcapi.GrpcResponse{
    		Response: result,
    	}
    	return res, nil
    }
    
    func main() {
    	fmt.Println("Starting Server...")
    
    	log.SetFlags(log.LstdFlags | log.Lshortfile)
    
    	hostname := os.Getenv("SVC_HOST_NAME")
    
    	if len(hostname) <= 0 {
    		hostname = "0.0.0.0"
    	}
    
    	port := os.Getenv("SVC_PORT")
    
    	if len(port) <= 0 {
    		port = "50051"
    	}
    
    	lis, err := net.Listen("tcp", hostname+":"+port)
    	if err != nil {
    		log.Fatalf("Failed to listen: %v", err)
    	}
    
    	opts := []grpc.ServerOption{}
    	s := grpc.NewServer(opts...)
    	grpcapi.RegisterGrpcServiceServer(s, &grpcServer{})
    
    	// reflection service on gRPC server.
    	reflection.Register(s)
    
    	go func() {
    		fmt.Println("Server running on ", (hostname + ":" + port))
    		if err := s.Serve(lis); err != nil {
    			log.Fatalf("failed to serve: %v", err)
    		}
    	}()
    
    	// Wait for Control C to exit
    	ch := make(chan os.Signal, 1)
    	signal.Notify(ch, os.Interrupt)
    
    	// Block until a signal is received
    	<-ch
    	fmt.Println("Stopping the server")
    	s.Stop()
    	fmt.Println("Closing the listener")
    	lis.Close()
    	fmt.Println("Server Shutdown")
    
    }

**Dockerfile**

FROM iron/go
    WORKDIR /app
    
    ADD grpc_server /app/
    
    CMD [ "./grpc_server" ]

**构建binary和镜像：**

CGO_ENABLED=0 GOOS=linux go build -o grpc_server -ldflags "-s -w -X 'main.build=$(git rev-parse --short HEAD)' -X 'main.buildDate=$(date --rfc-3339=seconds)'" -a -installsuffix cgo server.go
    docker build -t myregistry.com/grpc_server .
    docker push myregistry.comgrpc_server

2.2 部署gRPC服务

apiVersion: v1
    kind: Service
    metadata:
      name: grpcserver
    spec:
      ports:
      - port: 50051
        protocol: TCP
        targetPort: 50051
      selector:
        run: grpcserver
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        run: grpcserver
      name: grpcserver
    spec:
      replicas: 2
      selector:
        matchLabels:
          run: grpcserver
      template:
        metadata:
          labels:
            run: grpcserver
        spec:
          containers:
          - image: myregistry.com/grpc_server:latest
            name: grpcserver
            ports:
            - containerPort: 50051

包含两个replica和一个cluster service：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70][] ![2020051217180959.png][]

转载自https://blog.csdn.net/cloudvtech

## **三、通过Ingress Nginx暴露gRPC服务** ##

参考文档：

[https://kubernetes.github.io/ingress-nginx/examples/grpc/][https_kubernetes.github.io_ingress-nginx_examples_grpc]

### 3.1 Ingress Nginx暴露gRPC服务的TLS要求 ###

Ingress Nginx暴露gRPC服务的时候，暂时只支持TLS(HTTPS)的方式，而不能通过普通HTTP方式，所以我们要配置TLS secret

**生成key**

openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout sslforingress.key -out sslforingress.pem -subj "/CN=grpc.test.com"

**生成secret**

kubectl create secret tls grpcserver-secret  --cert sslforingress.pem --key sslforingress.key -n grpcserver

### 3.2 部署Ingress ###

暴露的地址是 **grpc.test.com:443**

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
      name: grpcserver
      namespace: grpcserver
    spec:
      rules:
        - host: grpc.test.com
          http:
            paths:
              - backend:
                  serviceName: grpcserver
                  servicePort: 50051
      tls:
    # This secret must exist beforehand
    # The cert must also contain the subj-name grpc.test.com
    # https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/PREREQUISITES.md#tls-certificates
        - secretName: grpcserver-secret
          hosts:
            - grpc.test.com

nginx.conf生成的内容：

## start server grpc.test.com                                   
            server {                                                                                     
                    server_name grpc.test.com ;                                                            
                                                                                                                               
                    listen 80  ;                                                                                               
                    listen [::]:80  ;                                                                                          
                    listen 443  ssl http2 ;                                                                                    
                    listen [::]:443  ssl http2 ;                                                                               
                                                                                                                               
                    set $proxy_upstream_name "-";                                                                              
                                                                                                                               
                    ssl_certificate_by_lua_block {                                                                             
                            certificate.call()                                                                                 
                    }
    
                    location / {                                                  
                                                                                                         
                            set $namespace      "grpcserver";             
                            set $ingress_name   "grpcserver";                                      
                            set $service_name   "grpcserver";                   
                            set $service_port   "50051";                                                 
                            set $location_path  "/";                                                
    
            ... ...
                            # Allow websocket connections                                                
                            grpc_set_header                        Upgrade           $http_upgrade;      
                                                                                                         
                            grpc_set_header                        Connection        $connection_upgrade;
                                                                                       
                            grpc_set_header X-Request-ID           $req_id;            
                            grpc_set_header X-Real-IP              $remote_addr;       
                                                                                       
                            grpc_set_header X-Forwarded-For        $remote_addr;       
                                                                                           
                            grpc_set_header X-Forwarded-Host       $best_http_host;        
                            grpc_set_header X-Forwarded-Port       $pass_port;             
                            grpc_set_header X-Forwarded-Proto      $pass_access_scheme;                  
                                                                                                         
                            grpc_set_header X-Scheme               $pass_access_scheme;                  
                                                                                                         
                            # Pass the original X-Forwarded-For                                          
                            grpc_set_header X-Original-Forwarded-For $http_x_forwarded_for;              
                                                                                                         
                            # mitigate HTTPoxy Vulnerability                                             
                            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                            grpc_set_header Proxy                  "";    
                                                                        
                            # Custom headers to proxied server           
            
            ... ...
                    
                    }                                            
                                                                  
            }                                                            
            ## end server grpc.test.com

转载自https://blog.csdn.net/cloudvtech

## **四、通过Ingress Nginx访问gRPC服务** ##

### 4.1 安装grpcurl ###

[https://github.com/fullstorydev/grpcurl][https_github.com_fullstorydev_grpcurl]

go get github.com/fullstorydev/grpcurl
    go install github.com/fullstorydev/grpcurl/cmd/grpcurl

运行之后在go的PATH里面会有grpcurl的binary，将grpcurl拷贝到/usr/local/bin

### 4.2 通过grpcurl访问Kubernetes的gRPC服务 ###

查看服务暴露的gRPC方法：

grpcurl -insecure grpc.test.com:443 list
    greet.GrpcService
    grpc.reflection.v1alpha.ServerReflection

调用服务：

[root@k8s-node-01 ~]# grpcurl  -insecure grpc.test.com:443 greet.GrpcService/grpcService
    {
      "response": "Got input  server host: grpcserver-5bfd56f94b-bc6fg"
    }
    [root@k8s-node-01 ~]# grpcurl  -insecure grpc.test.com:443 greet.GrpcService/grpcService
    {
      "response": "Got input  server host: grpcserver-5bfd56f94b-w7frq"
    }

可以看到服务在两个节点之间轮转

## **五、通过API访问Ingress Nginx暴露的gRPC服务** ##

### 5.1 访问基于TLS的gRPC服务 ###

建立连接的时候加上如下代码，可以忽略SSL证书的有效性：

>         config := &tls.Config\{
> 
>                 InsecureSkipVerify: true,
> 
>         \}
> 
>         cc, err := grpc.Dial(hostname+":"+port, grpc.WithTransportCredentials(credentials.NewTLS(config)))

### 5.2 通过短连接访问gRPC服务 ###

**client\_short\_connection.go**

package main
    
    import (
    	"context"
    	"fmt"
    	"log"
    	"os"
    	"github.com/javiramos1/grpcapi"
    	"google.golang.org/grpc"
            "google.golang.org/grpc/credentials"
            "crypto/tls"
    )
    
    func main() {
    
    	fmt.Println("Starting client...")
    
    	hostname := os.Getenv("SVC_HOST_NAME")
    
    	if len(hostname) <= 0 {
    		hostname = "0.0.0.0"
    	}
    
    	port := os.Getenv("SVC_PORT")
    
    	if len(port) <= 0 {
    		port = "50051"
    	}
    
    	config := &tls.Config{
        		InsecureSkipVerify: true,
    	}
    	cc, err := grpc.Dial(hostname+":"+port, grpc.WithTransportCredentials(credentials.NewTLS(config)))
    
    
    	if err != nil {
    		log.Fatalf("could not connect: %v", err)
    	}
    	defer cc.Close()
    
    	c := grpcapi.NewGrpcServiceClient(cc)
    	fmt.Printf("Created client: %f", c)
    		callService(c)
    
    }
    
    func callService(c grpcapi.GrpcServiceClient) {
    	fmt.Println("callService...")
    	req := &grpcapi.GrpcRequest{
    		Input: "test",
    	}
    	res, err := c.GrpcService(context.Background(), req)
    	if err != nil {
    		log.Fatalf("error while calling gRPC: %v", err)
    	}
    	log.Printf("Response from Service: %v", res.Response)
    }

编译：

CGO_ENABLED=0 GOOS=linux go build -o grpc_client_short_connection -ldflags "-s -w -X 'main.build=$(git rev-parse --short HEAD)' -X 'main.buildDate=$(date --rfc-3339=seconds)'" -a -installsuffix cgo client.go

运行：

export SVC_HOST_NAME=grpc.test.com
    export SVC_PORT=443
    
    while true; do ./grpc_client_short_connection; sleep 1;done
    
    2020/05/12 13:21:39 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    2020/05/12 13:21:40 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    2020/05/12 13:21:41 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    2020/05/12 13:21:42 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    2020/05/12 13:21:43 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    2020/05/12 13:21:44 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    2020/05/12 13:21:45 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    2020/05/12 13:21:46 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    2020/05/12 13:21:47 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    2020/05/12 13:21:48 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    2020/05/12 13:21:49 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    2020/05/12 13:21:50 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg

同时可以看到系统存在大量端口被短连接占用之后等待关闭：

netstat -anp | grep 443 -w
    tcp        0      0 172.3.0.11:41984        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41948        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41910        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41974        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41934        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41970        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41928        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41912        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41940        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41942        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41976        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41994        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41980        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41988        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41998        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41916        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41978        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41924        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41922        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41996        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41986        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41962        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41906        172.2.2.11:443          TIME_WAIT   -                   
    tcp        0      0 172.3.0.11:41960        172.2.2.11:443          TIME_WAIT   -

### 5.3 通过长连接访问gRPC服务 ###

**client\_longlive\_connection.go**

package main
    
    import (
    	"context"
    	"fmt"
    	"log"
    	"os"
            "time"
    	"github.com/javiramos1/grpcapi"
    	"google.golang.org/grpc"
            "google.golang.org/grpc/credentials"
            "crypto/tls"
    )
    
    func main() {
    
    	fmt.Println("Starting client...")
    
    	hostname := os.Getenv("SVC_HOST_NAME")
    
    	if len(hostname) <= 0 {
    		hostname = "0.0.0.0"
    	}
    
    	port := os.Getenv("SVC_PORT")
    
    	if len(port) <= 0 {
    		port = "50051"
    	}
    
    	config := &tls.Config{
        		InsecureSkipVerify: true,
    	}
    	cc, err := grpc.Dial(hostname+":"+port, grpc.WithTransportCredentials(credentials.NewTLS(config)))
    
    
    	if err != nil {
    		log.Fatalf("could not connect: %v", err)
    	}
    	defer cc.Close()
    
    	c := grpcapi.NewGrpcServiceClient(cc)
    	fmt.Printf("Created client: %f", c)
            for i := 1; i <= 100; i++ {
    		callService(c)
          		time.Sleep(2000 * time.Millisecond)
            }
    
    }
    
    func callService(c grpcapi.GrpcServiceClient) {
    	fmt.Println("callService...")
    	req := &grpcapi.GrpcRequest{
    		Input: "test",
    	}
    	res, err := c.GrpcService(context.Background(), req)
    	if err != nil {
    		log.Fatalf("error while calling gRPC: %v", err)
    	}
    	log.Printf("Response from Service: %v", res.Response)
    }

编译并运行：

CGO_ENABLED=0 GOOS=linux go build -o grpc_client_longlive_connection -ldflags "-s -w -X 'main.build=$(git rev-parse --short HEAD)' -X 'main.buildDate=$(date --rfc-3339=seconds)'" -a -installsuffix cgo client.go
    
    export SVC_HOST_NAME=grpc.test.com
    export SVC_PORT=443
    
    ./grpc_client_longlive 
    2020/05/12 13:40:06 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    callService...
    2020/05/12 13:40:08 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    callService...
    2020/05/12 13:40:10 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    callService...
    2020/05/12 13:40:12 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    callService...
    2020/05/12 13:40:14 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    callService...
    2020/05/12 13:40:16 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    callService...
    2020/05/12 13:40:18 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    callService...
    2020/05/12 13:40:20 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq
    callService...
    2020/05/12 13:40:22 Response from Service: Got input test server host: grpcserver-5bfd56f94b-bc6fg
    callService...
    2020/05/12 13:40:24 Response from Service: Got input test server host: grpcserver-5bfd56f94b-w7frq

整个运行过程中只有一个到443端口的长连接：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70 1][]

这说明客户端到Ingress Nginx建立了gRPC长连接，而Ingress Nginx解码了七层gRPC数据包并进行负载均衡到gRPC 服务的两个replica POD

转载自https://blog.csdn.net/cloudvtech

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200512171755631.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g=,size_16,color_FFFFFF,t_70
[2020051217180959.png]: /images/20230312/b652479cb80845f9851db930be664563.png
[https_kubernetes.github.io_ingress-nginx_examples_grpc]: https://kubernetes.github.io/ingress-nginx/examples/grpc/
[https_github.com_fullstorydev_grpcurl]: https://github.com/fullstorydev/grpcurl
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70 1]: /images/20230312/8b7b6702ad7f45eaa738b4614e55d944.png

发表评论取消回复

表情：

评论列表（有 0 条评论，6人围观）

还没有评论，来说两句吧...

相关阅读

相关【Kubernetes】Kubernetes对外服务之Ingress

Ingress 一、Ingress 的概念 1. Ingress 简介 2. K8S 对外提供服务的方案 2.1 No

「爱情、让人受尽委屈。」/ 2024年03月24日 18:17/ 0 赞/ 74 阅读

相关 Kubernetes之service服务暴露

> service的IP地址仅在集群内部可达，然而，总会有些服务需要暴露到外部网络中接受各类客户端的访问，此时就需要在集群的边缘为其添加一层转发机制，以实现外部流量进入到集群的

我不是女神ヾ/ 2023年07月06日 11:54/ 0 赞/ 25 阅读

相关 Kubernetes生产实践系列之十：安装Kubernetes原生的airflow

一、前言 Airflow是Airbnb的基于DAG(有向无环图)的任务管理系统，是进行任务分割、调度处理的利器。在生产实践中，有业务部门需要使用airflow来进行大批量

今天药忘吃喽~/ 2023年05月22日 22:09/ 0 赞/ 25 阅读

相关 Kubernetes生产实践系列之九：通过RPM安装Kubernetes HA集群

一、前言高可用的Kubernetes集群会有3个master节点，运行3个APIServer实例，这三个APIServer实例可以挂载再同一个负载均衡器后端，实现

悠悠/ 2023年05月21日 13:56/ 0 赞/ 14 阅读

相关 Kubernetes生产实践系列之十九：ServiceMesh之使用Ingress Nginx暴露Kubernetes上的gRPC服务

一、前言 Ingress Nginx支持gRPC服务的暴露，低于0.21的版本使用如下的annotation支持gRPC： nginx.ingress.kube

Bertha 。/ 2023年03月13日 05:47/ 0 赞/ 7 阅读

相关 Kubernetes生产实践系列之十七：ServiceMesh之gRPC基本原理和实践

一、前言 gRPC是一款高效的RPC框架，基于HTTP/2进行设计并支持ProtoBuf序列化协议。gRPC使用长连接和单连接内部高并发来提高性能，这对于数据中心内部服务

绝地灬酷狼/ 2023年03月12日 14:28/ 0 赞/ 38 阅读

相关 Kubernetes生产实践系列之十八：ServiceMesh之在Kubernetes中对gRPC流量进行负载均衡

一、前言 gRPC服务在部署到Kubernetes之后，有来自Kubernetes内部和外部的gRPC访问请求，所以对负责请求接入和负载均衡的LB提出了更高的要求，除了实

超、凢脫俗/ 2023年03月12日 14:28/ 0 赞/ 29 阅读

相关 Kubernetes生产实践系列之二十九：Kubernetes基础技术之容器关键技术实践

一、前言在文章[《Kubernetes生产实践系列之二十八：Kubernetes基础技术之容器关键技术介绍》][Kubernetes_Kubernetes]中，对于Do

雨点打透心脏的1/2处/ 2022年11月26日 15:52/ 0 赞/ 373 阅读

相关 Kubernetes使用Nginx Ingress暴露Dashboard

Kubernetes使用Nginx Ingress暴露Dashboard 文章目录 Kubernetes使用Nginx Ingress暴露Dashboard

超、凢脫俗/ 2022年05月11日 06:28/ 0 赞/ 215 阅读

相关【kubernetes 系列5】Kubernetes服务发现 ingress & ingress controller

Kubernetes服务发现 ingress & ingress controller 1、介绍 kubernetes Ingess 是有2部分组成，Ingress

迈不过友情╰/ 2022年01月28日 04:27/ 0 赞/ 268 阅读