k8s使用openebs实现动态持久化存储

红太狼 2022-03-08 04:52 1380阅读 0赞

### 简介 ###

本文章介绍如何使用openebs为k8s提供动态申请pv的功能。iscsi提供底层存储功能，openebs管理iscsi。目前只支持pv的`ReadWriteOnce`访问模式

访问模式只是能力描述，并不是强制执行的，对于没有按pvc声明的方式使用pv，存储提供者应该负责访问时的运行错误。例如如果设置pvc的访问模式为`ReadOnlyMany` ，pod挂载后依然可写，如果需要真正的不可写，申请pvc是需要指定 `readOnly: true` 参数

### 安装 ###

实验用的Vagrantfile

# -*- mode: ruby -*-
    # vi: set ft=ruby :
    
    ENV["LC_ALL"] = "en_US.UTF-8"
    
    Vagrant.configure("2") do |config|
        (1..3).each do |i|
          config.vm.define "lab#{i}" do |node|
            node.vm.box = "centos-7.4-docker-17"
            node.ssh.insert_key = false
            node.vm.hostname = "lab#{i}"
            node.vm.network "private_network", ip: "11.11.11.11#{i}"
            node.vm.provision "shell",
              inline: "echo hello from node #{i}"
            node.vm.provider "virtualbox" do |v|
              v.cpus = 2
              v.customize ["modifyvm", :id, "--name", "lab#{i}", "--memory", "3096"]
              file_to_disk = "lab#{i}_vdb.vdi"
              unless File.exist?(file_to_disk)
                # 50GB
                v.customize ['createhd', '--filename', file_to_disk, '--size', 50 * 1024]
              end
              v.customize ['storageattach', :id, '--storagectl', 'IDE', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', file_to_disk]
            end
          end
        end
    end

安装配置iscsi

# 安装 iscsi
    yum install iscsi-initiator-utils -y
    
    # 查看 InitiatorName 是否正常配置
    cat /etc/iscsi/initiatorname.iscsi
    
    # 启动查看状态
    systemctl start iscsid.service
    systemctl status iscsid.service
    
    systemctl start iscsi.service
    systemctl status iscsi.service

安装openebs

# 部署
    mkdir openebs && cd openebs
    wget https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-operator.yaml
    wget https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-storageclasses.yaml
    kubectl apply -f openebs-operator.yaml
    kubectl apply -f openebs-storageclasses.yaml
    
    # 查看 openebs 状态
    kubectl get pods -n openebs -o wide
    kubectl get svc -n openebs
    kubectl get crd

### 测试 ###

# 查看 storage class
    kubectl get sc
    
    # 创建pvc测试
    cat >openebs-pvc-test.yaml<<EOF
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
     name: openebs1
    spec:
     storageClassName: openebs-standard
     accessModes:
      - ReadWriteOnce
     resources:
       requests:
         storage: 5Gi
    EOF
    kubectl apply -f openebs-pvc-test.yaml
     
    # 查看
    kubectl get pvc
    kubectl get pv
     
    # 创建 nginx pod 挂载测试
    cat >nginx-pod.yaml<<EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx-pod1
      labels:
        name: nginx-pod1
    spec:
      containers:
      - name: nginx-pod1
        image: nginx:alpine
        ports:
        - name: web
          containerPort: 80
        volumeMounts:
        - name: openebs1-vol1
          mountPath: /usr/share/nginx/html
      volumes:
      - name: openebs1-vol1
        persistentVolumeClaim:
          claimName: openebs1
    EOF
    kubectl apply -f nginx-pod.yaml
     
    # 查看
    kubectl get pods -o wide
     
    # 修改文件内容
    kubectl exec -ti nginx-pod1 -- /bin/sh -c 'echo Hello World from Openebs!!! > /usr/share/nginx/html/index.html'
     
    # 访问测试
    POD_ID=$(kubectl get pods -o wide | grep nginx-pod1 | awk '{print $(NF-1)}')
    curl http://$POD_ID

### 参考文档 ###

*  [https://github.com/heketi/heketi/blob/master/docs/admin/install-kubernetes.md][https_github.com_heketi_heketi_blob_master_docs_admin_install-kubernetes.md]
 *  [https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md][https_github.com_gluster_gluster-kubernetes_blob_master_docs_setup-guide.md]
 *  [https://github.com/gluster/gluster-kubernetes/blob/master/docs/examples/hello\_world/README.md][https_github.com_gluster_gluster-kubernetes_blob_master_docs_examples_hello_world_README.md]
 *  [https://jimmysong.io/kubernetes-handbook/practice/using-heketi-gluster-for-persistent-storage.html][https_jimmysong.io_kubernetes-handbook_practice_using-heketi-gluster-for-persistent-storage.html]
 *  [https://kubernetes.io/docs/concepts/storage/persistent-volumes/][https_kubernetes.io_docs_concepts_storage_persistent-volumes]
 *  [https://docs.openshift.com/enterprise/3.1/architecture/additional\_concepts/storage.html\#pv-access-modes][https_docs.openshift.com_enterprise_3.1_architecture_additional_concepts_storage.html_pv-access-modes]

作者：CountingStars\_  
链接：https://www.jianshu.com/p/f0ac69afc418

[https_github.com_heketi_heketi_blob_master_docs_admin_install-kubernetes.md]: https://github.com/heketi/heketi/blob/master/docs/admin/install-kubernetes.md
[https_github.com_gluster_gluster-kubernetes_blob_master_docs_setup-guide.md]: https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md
[https_github.com_gluster_gluster-kubernetes_blob_master_docs_examples_hello_world_README.md]: https://github.com/gluster/gluster-kubernetes/blob/master/docs/examples/hello_world/README.md
[https_jimmysong.io_kubernetes-handbook_practice_using-heketi-gluster-for-persistent-storage.html]: https://jimmysong.io/kubernetes-handbook/practice/using-heketi-gluster-for-persistent-storage.html
[https_kubernetes.io_docs_concepts_storage_persistent-volumes]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
[https_docs.openshift.com_enterprise_3.1_architecture_additional_concepts_storage.html_pv-access-modes]: https://docs.openshift.com/enterprise/3.1/architecture/additional_concepts/storage.html#pv-access-modes