【kubernetes/k8s概念】rook

今天药忘吃喽~ 2022-02-05 07:17 307阅读 0赞

[https://github.com/rook/rook][https_github.com_rook_rook]

>      [云原生计算基金会][Link 1](CNCF)的孵化级项目，是Kubernetes的开源**云本地存储协调**器，为各种存储解决方案提供平台，框架和支持，以便与云原生环境本地集成
> 
>     Ceph is a highly scalable distributed storage solution for **block storage**, **object storage**, and **shared file systems** with years of production deployments.
> 
>     实现了一个自动管理的、自动扩容的、自动修复的分布式存储服务。Rook 支持 Ceph 存储，基于 Kubernetes 使用 Rook 可以大大简化 Ceph 存储集群的搭建以及使用

# Desigin #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70][]

引用：[https://github.com/rook/rook/blob/master/Documentation/ceph-storage.md][https_github.com_rook_rook_blob_master_Documentation_ceph-storage.md]

operator：作用启动监控存储集群，包括mon，osd，mgr等服务，可以支持 block 块存储，file system 文件存储，object 对象存储 (S3/Swift)。 启动包括ceph agent 与 discover服务，agent 包含 Flexvolume plugin，包括attaching network storage devices, mounting volumes, and formating the filesystem 三种操作。

# Rook 架构 #

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 1][]

> # Block 块存储 #
> 
>        Ceph 块设备也可以简称为 RBD 或 RADOS 块设备
> 
>        Rook 创建 Ceph 块设备，先创建 CephBlockPool 和 StorageClass，CephBlockPool 是 Rook 的 CRD （创建一个 Block 池），在创建一个 StorageClass，这个是动态卷配置 (Dynamic provisioning) 
> 
> # File System 文件系统 #
> 
>       A shared file system can be mounted with read/write permission from multiple pods. This may be useful for applications which can be clustered using a shared filesystem.
> 
>       ROOK\_ALLOW\_MULTIPLE\_FILESYSTEM定义允许多个文件系统共享
> 
> # Object 对象存储      #
> 
>      Ceph RGW 是基于 librados，为应用提供 RESTful 类型的对象存储接口，其接口方式支持 `S3`（兼容 Amazon S3 RESTful API） 和 `Swift`（兼容 OpenStack Swift API） 两种类型

# 组件Mon #

>        Mon作用是监控、管理和协调整个分布式系统环境中其它各个OSD/PG、Client、MDS角色的工作，保证整个分布环境中的数据一致性
> 
>        MON采用主备模式（leader/follower），即使系统中有多个MON角色，实际工作的也只有一个MON，其它MON都处于standby状态
> 
>        **当Ceph失去了Leader MON后，其它MON会基于PaxOS算法，投票选出新的Leader**

# 组件osd #

>     存储落地操作的节点叫做OSD（Object Storage Device），基于它工作的RADOS、PG等模块

# 1. rook快速部署： #

路径rook/cluster/example/kubernetes/ceph

1. kubectl apply -f common.yaml

2. kubectl apply -f operator.yaml

3. kubectl aplply -f cluster.yaml

*  **dataDirHostPath**: 在宿主机上生成的，保存的是ceph的相关的配置文件，再重新生成集群目录为空，否则mon会无法启动
 *  **useAllDevices**: 使用所有的设备，建议为false，否则会把宿主机所有可用的磁盘都干掉
 *  **useAllNodes**：使用所有的node节点，建议为false，肯定不会用k8s集群内的所有node来搭建ceph的
 *  **databaseSizeMB和journalSizeMB**：当磁盘大于100G的时候，就注释这俩项就行了

# 2. rook csi部署 #

###   2.1 需求 ###

1.  a Kubernetes v1.13+ is needed in order to support CSI Spec 1.0.
2.  `--allow-privileged` flag set to true in [kubelet][] and your [API server][]

需要的镜像：

quay.io/k8scsi/csi-provisioner:v1.0.1

quay.io/cephcsi/rbdplugin:v1.0.0

quay.io/k8scsi/csi-attacher:v1.0.1

quay.io/k8scsi/csi-snapshotter:v1.0.1

quay.io/k8scsi/csi-node-driver-registrar:v1.0.2

quay.io/cephcsi/cephfsplugin:v1.0.0

###   2.2 为CSI驱动create RBRC ###

绑定的ClusterRole的rule未设置，是不是被骗了，搞这么一大堆

kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/rbd/csi-nodeplugin-rbac.yaml

apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: rook-csi-rbd-plugin-sa
      namespace: rook-ceph
    
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: rbd-csi-nodeplugin
    aggregationRule:
      clusterRoleSelectors:
      - matchLabels:
          rbac.ceph.rook.io/aggregate-to-rbd-csi-nodeplugin: "true"
    rules: []
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: rbd-csi-nodeplugin-rules
      labels:
        rbac.ceph.rook.io/aggregate-to-rbd-csi-nodeplugin: "true"
    rules:
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "update"]
      - apiGroups: [""]
        resources: ["namespaces"]
        verbs: ["get", "list"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["configmaps"]
        verbs: ["get", "list"]
    
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: rbd-csi-nodeplugin
    subjects:
      - kind: ServiceAccount
        name: rook-csi-rbd-plugin-sa
        namespace: rook-ceph
    roleRef:
      kind: ClusterRole
      name: rbd-csi-nodeplugin
      apiGroup: rbac.authorization.k8s.io

kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/rbd/csi-provisioner-rbac.yaml

apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: rook-csi-rbd-provisioner-sa
      namespace: rook-ceph
    
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: rbd-external-provisioner-runner
    aggregationRule:
      clusterRoleSelectors:
      - matchLabels:
          rbac.ceph.rook.io/aggregate-to-rbd-external-provisioner-runner: "true"
    rules: []
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: rbd-external-provisioner-runner-rules
      labels:
        rbac.ceph.rook.io/aggregate-to-rbd-external-provisioner-runner: "true"
    rules:
      - apiGroups: [""]
        resources: ["secrets"]
        verbs: ["get", "list"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "create", "delete", "update"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["volumeattachments"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["list", "watch", "create", "update", "patch"]
      - apiGroups: [""]
        resources: ["endpoints"]
        verbs: ["get", "create", "update"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["volumesnapshots"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: [""]
        resources: ["configmaps"]
        verbs: ["get", "list", "create", "delete"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["volumesnapshotcontents"]
        verbs: ["create", "get", "list", "watch", "update", "delete"]
      - apiGroups: ["snapshot.storage.k8s.io"]
        resources: ["volumesnapshotclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["customresourcedefinitions"]
        verbs: ["create"]
    
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: rbd-csi-provisioner-role
    subjects:
      - kind: ServiceAccount
        name: rook-csi-rbd-provisioner-sa
        namespace: rook-ceph
    roleRef:
      kind: ClusterRole
      name: rbd-external-provisioner-runner
      apiGroup: rbac.authorization.k8s.io

同理创建cephfs的rbac，就不展示了

kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/cephfs

###     2.3 创建operator-with-csi.yaml ###

kubectl apply -f operaor-with-csi.yaml

#################################################################################################################
    # The deployment for the rook operator that enables the ceph-csi driver for beta testing.
    # For example, to create the rook-ceph cluster:
    #   kubectl create -f common.yaml
    #   kubectl create -f operator-with-csi.yaml
    #   kubectl create -f cluster.yaml
    #################################################################################################################
    
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: rook-ceph-operator
      namespace: rook-ceph
      labels:
        operator: rook
        storage-backend: ceph
    spec:
      selector:
        matchLabels:
          app: rook-ceph-operator
      replicas: 1
      template:
        metadata:
          labels:
            app: rook-ceph-operator
        spec:
          serviceAccountName: rook-ceph-system
          containers:
          - name: rook-ceph-operator
            image: rook/ceph:master
            args: ["ceph", "operator"]
            volumeMounts:
            - mountPath: /var/lib/rook
              name: rook-config
            - mountPath: /etc/ceph
              name: default-config-dir
            env:
            - name: ROOK_CURRENT_NAMESPACE_ONLY
              value: "true"
            # CSI enablement
            - name: ROOK_CSI_ENABLE_CEPHFS
              value: "true"
            - name: ROOK_CSI_CEPHFS_IMAGE
              value: "quay.io/cephcsi/cephfsplugin:v1.0.0"
            - name: ROOK_CSI_ENABLE_RBD
              value: "true"
            - name: ROOK_CSI_RBD_IMAGE
              value: "quay.io/cephcsi/rbdplugin:v1.0.0"
            - name: ROOK_CSI_REGISTRAR_IMAGE
              value: "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
            - name: ROOK_CSI_PROVISIONER_IMAGE
              value: "quay.io/k8scsi/csi-provisioner:v1.0.1"
            - name: ROOK_CSI_SNAPSHOTTER_IMAGE
              value: "quay.io/k8scsi/csi-snapshotter:v1.0.1"
            - name: ROOK_CSI_ATTACHER_IMAGE
              value: "quay.io/k8scsi/csi-attacher:v1.0.1"
            # The name of the node to pass with the downward API
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # The pod name to pass with the downward API
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            # The pod namespace to pass with the downward API
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumes:
          - name: rook-config
            emptyDir: {}
          - name: default-config-dir
            emptyDir: {}

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 3][]

核心是RADOS，也即高可用、自动化的分布式对象存储，该模块负责对OSD的自动化运行，保证整个存储系统的可用性。同时，该模块通过一个名为librados的公共库对外提供存储服务，块存储和对象存储都依赖该动态库。

## **参考资料** ##

*  [https://github.com/rook/rook/blob/master/Documentation/ceph-block.md][https_github.com_rook_rook_blob_master_Documentation_ceph-block.md]
 *  [https://github.com/rook/rook/blob/master/Documentation/ceph-filesystem.md][https_github.com_rook_rook_blob_master_Documentation_ceph-filesystem.md]
 *  [Rook Ceph Object Storage][]
 *  [https://github.com/rook/rook/blob/master/Documentation/ceph-storage.md][https_github.com_rook_rook_blob_master_Documentation_ceph-storage.md]

https://github.com/rook/rook

https://rook.io/

https://edgefs.io/

[https_github.com_rook_rook]: https://github.com/rook/rook
[Link 1]: http://a/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70]: /images/20220205/db9814b0f9354211ab17fc84df8a114a.png
[https_github.com_rook_rook_blob_master_Documentation_ceph-storage.md]: https://github.com/rook/rook/blob/master/Documentation/ceph-storage.md
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 1]: /images/20220205/557ce4b932e24c94be5722c26847d126.png
[kubelet]: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
[API server]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 2]: /images/20220205/691f6532208c44cf8600acbe153949af.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob25nbGluemhhbmc_size_16_color_FFFFFF_t_70 3]: /images/20220205/06e2d6152cfe4e9aa3ef978dbd8175d5.png
[https_github.com_rook_rook_blob_master_Documentation_ceph-block.md]: https://github.com/rook/rook/blob/master/Documentation/ceph-block.md
[https_github.com_rook_rook_blob_master_Documentation_ceph-filesystem.md]: https://github.com/rook/rook/blob/master/Documentation/ceph-filesystem.md
[Rook Ceph Object Storage]: https://rook.github.io/docs/rook/v0.9/ceph-object.html