k8s ingress 配置问题

比眉伴天荒 2022-09-09 10:59 185阅读 0赞

**目录**

Ingress-Nginx Deployment

开启TLS/ HTTPS

四层反向代理配置

客户端地址记录

ingress匹配优先级

跨域配置

白名单及请求速率限制

支持websocket配置

七层负载均衡算法

通过configmap nginx-configuration定义一些全局常规参数

--------------------

### **Ingress-Nginx Deployment ** ###

containers:
            - name: nginx-ingress-controller
              image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
              args:
                - /nginx-ingress-controller
                - --configmap=$(POD_NAMESPACE)/nginx-configuration
                - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
                - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
                - --publish-service=$(POD_NAMESPACE)/ingress-nginx
                - --annotations-prefix=nginx.ingress.kubernetes.io

> \--ingress-class：声明ingress入口名称，如果ingress资源要绑该ingress controller，需要在 annotation 中定义 kubernetes.io/ingress.class: "nginx"

### 开启TLS/ HTTPS ###

创建ssl证书 secret

> kubectl -n default create secret tls tls-https  --key ./tls.key  --cert ./tls.crt  
> secret/tls-https created

默认情况下，如果ingress对象入口启用了TLS，则ingress-controller将使用308永久重定向响应将HTTP客户端重定向到HTTPS端口443

[root@test-k8s-wuwjg static]# curl -I   http://test.haha.com/
     HTTP/1.1 308 Permanent Redirect

可以在特定ingress资源的metadata.annotations中通过配置nginx.ingress.kubernetes.io/ssl-redirect: "false" 使用注释禁用此功能

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: ingress-dt
      annotations:
        kubernetes.io/ingress.class: "nginx" # 绑定ingress-class
        nginx.ingress.kubernetes.io/ssl-redirect: "false" #关闭SSL跳转
    spec:
      tls:
      - hosts:
        - test.haha.com
        secretName: tls-https 
      rules:
        - host: test.haha.com
          http:
            paths:
              - path: /
                backend:
                  serviceName: test-front
                  servicePort: 80

###     四层反向代理配置 ###

> Ingress Controller启动时会去watch两个configmap(一个tcp，一个 udp),即开头deployment模板中args字段配置 --tcp-services-configmap --udp-services-configmap所定义,以tcp代理相关configmap如下

kubectl edit cm  tcp-services -o yaml -n ingress-nginx 
    apiVersion: v1
    kind: ConfigMap
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      name: tcp-services
      namespace: ingress-nginx
    data:     #通过data字段添加四层反向代理的service
      "6666": default/nginx-06:1992  #key为代理端口
     
    wq!
     
    [root@test-k8s-01]#telnet 192.168.1.18 6666
    Trying 192.168.2.18...
    Connected to 192.168.2.18.
    Escape character is '^]'.

### 客户端地址记录 ###

> 可以在特定ingress资源下通过metadata.annotations字段下通过nginx.ingress.kubernetes.io/configuration-snippet 参数来定义

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: ingress-dt
      annotations:
        kubernetes.io/ingress.class: "nginx"
        nginx.ingress.kubernetes.io/configuration-snippet: |
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;

###    ingress匹配优先级 ###

> 首先在nginx中我们知道location的匹配优先级大致为精准匹配 > 前缀匹配 > 正则匹配> /
> 
> 其中,前缀匹配:^~，精准匹配 =，正则匹配细分为：
> 
> ~ 区分大小写（大小写敏感）匹配成功；~\* 不区分大小写匹配成功；!~ 区分大小写匹配失败；!~\*  不区分大小写匹配失败
> 
> 而ingress资源对象中，spec.rules.http.paths.path字段默认只支持不区分大小写的正则匹配,但前提需要设置nginx.ingress.kubernetes.io/use-regex注释设置为true(默认值为false)来启用此功能                  
>

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: ingress-dt
      annotations:
        kubernetes.io/ingress.class: "nginx"
        nginx.ingress.kubernetes.io/use-regex: "true"
    spec:
      rules:
        - host: test.haha.com
          http:
            paths:
              - path: /
                backend:
                  serviceName: nginx-front
                  servicePort: 80
              - path: /wifi
                backend:
                  serviceName: nginx-wifi
                  servicePort: 80

>  进入到ingress-controller的pod中，观察nginx配置文件，发现location的正则匹配已生效

[root@test-k8s-wuwjg static]# kubectl exec -it nginx-ingress-controller-79886bd49b-zf5h5 -n ingress-nginx /bin/sh -c "ca nginx.conf"
       ...
       ...
            ## start server test.haha.com                                                                                                                                                                
            server {                                                                                                                                                                                          
                    server_name test.haha.com ;                                                                                                                                                          
                                                                                                                                                                                                              
                    listen 80  ;                                                                                                                                                                              
                    listen [::]:80  ;                                                                                                                                                                        
                                                                                                                                                                                                              
                    set $proxy_upstream_name "-";                                                                                                                                                             
                                                                                                                                                                                                              
                    ssl_certificate_by_lua_block {                                                                                                                                                            
                            certificate.call()                                                                                                                                                                
                    }                                                                                                                                                                                         
                                                                                                                                                                                                              
                    location ~* "^/wifi" {                                                                                                                                                                    
                                                                                                                                                                                                              
                            set $namespace      "default";                                                                                                                                                    
                            set $ingress_name   "ingress-dt";                                                                                                                                                 
                            set $service_name   "nginx-wifi";                                                                                                                                            
                            set $service_port   "80";                                                                                                                                                         
                            set $location_path  "/wifi";

### **跨域配置**  ###

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: ingress-dt
      annotations:
        nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
        nginx.ingress.kubernetes.io/cors-allow-headers："DNT,X-CustomHeader,Keep-Alive,User- 
    Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization"
        nginx.ingress.kubernetes.io/cors-allow-origin: "*"

###    白名单及请求速率限制 ###

> 设置 test.haha.com/login 登陆页为每秒100个连接数，192.168.1.0/24,192.168.2.8 IP段不在限速范围

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: ingress-dt
      annotations:
        kubernetes.io/ingress.class: "nginx"
        nginx.ingress.kubernetes.io/whitelist-source-range: 192.168.1.0/24,192.168.2.8
        nginx.ingress.kubernetes.io/limit-rps: '100'
        nginx.ingress.kubernetes.io/use-regex: "true"
        nginx.ingress.kubernetes.io/ssl-redirect: "false"
    spec:
      tls:
      - hosts:
        - test.haha.com
        secretName: tls-https
      rules:
        - host: test.haha.com
          http:
            paths:
              - path: /login
                backend:
                  serviceName: nginx-front
                  servicePort: 80

###     支持websocket配置  ###

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: ingress-dt
      annotations:
        kubernetes.io/ingress.class: "nginx"
        nginx.ingress.kubernetes.io/configuration-snippet: |
        proxy_set_header Upgrade "websocket";
        proxy_set_header Connection "Upgrade";
        nginx.ingress.kubernetes.io/proxy-read-timeout 3600; 
        nginx.ingress.kubernetes.io/proxy-send-timeout 3600;

###    七层负载均衡算法 ###

> 默认为round-robin，在具体ingress资源中通过ingress metadata.annotations字段可具体设置

>  通过会话cookie进行一致性hash均衡算法

ingress.kubernetes.io/affinity: "cookie"
    ingress.kubernetes.io/session-cookie-name: "route"
    ingress.kubernetes.io/session-cookie-hash: "sha1"

> 通过客户端ip进行一致性hash的均衡算法

nginx.ingress.kubernetes.io/upstream-hash-by: "${remote_addr}"

>  通过请求uri进行一致性hash的均衡算法

nginx.ingress.kubernetes.io/upstream-hash-by: "${request_uri}"

###    通过configmap nginx-configuration定义一些全局常规参数  ###

[root@test-k8s-01]# kubectl get cm nginx-configuration -n ingress-nginx -o yaml |grep data |grep -v "metadata" -A 500
    data:
      multi_accept: on;
      use: epoll;
      user: www;
      worker_connections: 65535;
      worker_cpu_affinity: auto;
      worker_processes: auto;
      worker_rlimit_nofile: 300000;
      
      # 把真实IP地址传给后端
      compute-full-forwarded-for: "true"
      forwarded-for-header: "X-Forwarded-For"
      use-forwarded-headers: "true"
      # 关闭版本显示
      server-tokens: "false"
      # 客户端请求头的缓冲区大小 
      client-header-buffer-size: "512k"
      # 设置用于读取大型客户端请求标头的最大值number和size缓冲区
      large-client-header-buffers: "16 512k"
      # 读取客户端请求body的缓冲区大小
      client-body-buffer-size: "968k"
      # 代理缓冲区大小
      proxy-buffer-size: "1024k"
      # 代理body大小
      proxy-body-size: "50m"
      # 服务器名称哈希大小
      server-name-hash-bucket-size: "128"
      # map哈希大小
      map-hash-bucket-size: "128"
      # SSL加密套件
      ssl-ciphers: "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA"
      # ssl 协议
      ssl-protocols: "TLSv1 TLSv1.1 TLSv1.2"
    #定义json 访问日志格式
    log-format-upstream: '{"time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr", "x-forward-for": "$proxy_add_x_forwarded_for", "request_id": "$req_id", "remote_user": "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":$status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length, "duration": $request_time,"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent"}'