基于Kubernetes的框架和应用系列之五：在Kubernetes安装基于Ceph PVC的cassandra集群

港控/mmm° 2022-01-15 15:29 328阅读 0赞

### 一、前言 ###

在数据库领域，针对不同的业务需求，有不同类型的数据库方案，比如：

*  关系型数据库，如Mysql、Oracle、DB2、Sqlserver
 *  非关系型数据库
    
     *  Key-Value数据库，如Memcached、Redis、MemcacheDB
     *  Document-based数据库，如MongoDB、CouchDB
     *  Column-based数据库，如Cassandra、HBase
     *  图形数据库，如Neo4J、InforGrid

下图是在DB排名网站[https://db-engines.com/en/ranking][https_db-engines.com_en_ranking]的TOP20数据库：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70][]

可见Cassandra是列数据库中最热门的，Cassandra有如下一些特点：

*  极高的读写性能，写入操作只涉及到顺序写入和内存操作，具有很高的写性能；而进行读操作时，Cassandra实现数据分层存储，将热点数据放在Memtable和相对小的SSTable中，所以具有很高的读性能。
 *  简单的部署结构，Cassandra是去中心化的P2P结构，所有节点完全一样没有单点问题

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70 1][]

上图是但数据中心consistency level 是1的数据读取过程，节点10是Coordinator，负责和客户端交互，真正的数据有三个副本分别在1，3，6节点，由于consistency level 是1，所以只要从节点1读取数据并返回就可以了。更复杂的场景可以[在这里找到分析][Link 1]。

本文讲述如何在Kubernetes部署带持久存储的高可用Cassandra集群。

转载自https://blog.csdn.net/cloudvtech

### 二、安装Cassandra集群 ###

**2.1 准备工作**

创建namespace

kubectl create ns cassandra

创建Ceph StorageClass的access key

kubectl create ns cassandra
    kubectl -n ceph get secrets/pvc-ceph-client-key -o json | jq '.metadata.namespace = "cassandra"' | kubectl create -f -
    kubectl get secret -n cassandra
    NAME                  TYPE                                  DATA      AGE
    default-token-7ldwg   kubernetes.io/service-account-token   3         1m
    pvc-ceph-client-key   kubernetes.io/rbd                     1         14s

**2.2 部署文件**

cassandra-service.yaml，提供集群访问接口

apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: cassandra
      name: cassandra
      namespace: cassandra
    spec:
      clusterIP: None
      ports:
        - port: 9042
      selector:
        app: cassandra

cassandra-statefulset.yaml，基于statefulset的9节点集群的部署

apiVersion: "apps/v1beta1"
    kind: StatefulSet
    metadata:
      name: cassandra
      namespace: cassandra
    spec:
      serviceName: cassandra
      replicas: 9
      template:
        metadata:
          labels:
            app: cassandra
        spec:
          containers:
            - name: cassandra
              image: cassandra:3
              imagePullPolicy: IfNotPresent
              ports:
                - containerPort: 7000
                  name: intra-node
                - containerPort: 7001
                  name: tls-intra-node
                - containerPort: 7199
                  name: jmx
                - containerPort: 9042
                  name: cql
              env:
                - name: CASSANDRA_SEEDS
                  value: cassandra-0.cassandra.cassandra.svc.cluster.local # cassandra-0.cassandra.%%NAMESPACE%%.svc.cluster.local
                - name: MAX_HEAP_SIZE
                  value: 256M
                - name: HEAP_NEWSIZE
                  value: 100M
                - name: CASSANDRA_CLUSTER_NAME
                  value: "Cassandra"
                - name: CASSANDRA_DC
                  value: "DC1"
                - name: CASSANDRA_RACK
                  value: "Rack1"
                - name: CASSANDRA_ENDPOINT_SNITCH
                  value: GossipingPropertyFileSnitch
              volumeMounts:
                - name: cassandra-data
                  mountPath: /var/lib/cassandra/data
      volumeClaimTemplates:
        - metadata:
            name: cassandra-data
            annotations:  # comment line if you want to use a StorageClass
              # or specify which StorageClass
              volume.beta.kubernetes.io/storage-class: "ceph-rbd"   # comment line if you
              # want to use a StorageClass or specify which StorageClass
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 100Gi

**2.3 查看部署结果**

所有的POD

[root@k8s-master-01 cassandra-ceph]# kubectl get pods -n cassandra
    NAME          READY     STATUS    RESTARTS   AGE
    cassandra-0   1/1       Running   0          8m
    cassandra-1   1/1       Running   0          7m
    cassandra-2   1/1       Running   6          7m
    cassandra-3   1/1       Running   0          7m
    cassandra-4   1/1       Running   1          7m
    cassandra-5   1/1       Running   3          6m
    cassandra-6   1/1       Running   4          6m
    cassandra-7   1/1       Running   0          6m
    cassandra-8   1/1       Running   1          5m

所有的PV

[root@k8s-master-01 cassandra-ceph]# kubectl get pv -n cassandra | grep cassandra
    pvc-41daebbf-cd35-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-2   ceph-rbd                 10m
    pvc-4b1866b0-cd35-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-3   ceph-rbd                 9m
    pvc-59be4568-cd35-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-4   ceph-rbd                 9m
    pvc-67c8c038-cd35-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-5   ceph-rbd                 8m
    pvc-6e4e1b94-cd31-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-0   ceph-rbd                 37m
    pvc-74b73065-cd31-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-1   ceph-rbd                 37m
    pvc-7848907c-cd35-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-6   ceph-rbd                 8m
    pvc-7ef9c6bc-cd35-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-7   ceph-rbd                 8m
    pvc-99379af4-cd35-11e8-9135-fa163ebda1b8   100Gi      RWO            Delete           Bound         cassandra/cassandra-data-cassandra-8   ceph-rbd                 7m

相互之间数据交互关系

**![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70 2][]**

**2.4 查看Cassandra集群状态**

[root@k8s-master-01 cassandra-ceph]# kubectl exec -ti cassandra-0 -n cassandra -- nodetool status 
    Datacenter: DC1
    ===============
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
    UN  10.244.44.214   193.95 KiB  256          20.9%             f81e3aad-5d13-49a3-a0d7-77a505833f42  Rack1
    UN  10.244.89.150   114.71 KiB  256          22.0%             d3299f20-ae10-443f-97d0-25e5462833ca  Rack1
    UN  10.244.154.215  82.01 KiB  256          22.7%             d1261c1b-8f6a-4f0a-ae0b-21541dd99f77  Rack1
    UN  10.244.44.199   88.89 KiB  256          22.3%             62e2e592-9b80-47ec-98a4-6e8d909f1955  Rack1
    UN  10.244.44.210   69.92 KiB  256          22.9%             ab0e1502-995b-41a4-a363-a9cf42fe6bb2  Rack1
    UN  10.244.89.147   77.78 KiB  256          21.9%             ee1d368a-1599-49e7-8eb6-a16c7ef8dbf9  Rack1
    UN  10.244.89.190   114.72 KiB  256          22.7%             28e27ba4-5ecd-4126-aec2-c3444c287442  Rack1
    UN  10.244.154.223  69.93 KiB  256          23.0%             55d126e8-adfa-435b-a5d1-c2b0e074910b  Rack1
    UN  10.244.154.207  108.62 KiB  256          21.6%             ff0f5f91-fc98-4a58-956d-336586e048fb  Rack1

转载自https://blog.csdn.net/cloudvtech

[https_db-engines.com_en_ranking]: https://db-engines.com/en/ranking
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70]: /images/20220115/2ee1198b4423471bbf6ddb6ce79e0a28.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70 1]: /images/20220115/c250202c182c4a63ab145fc184d9bc9a.png
[Link 1]: https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/dml/dmlClientRequestsReadExp.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g_size_16_color_FFFFFF_t_70 2]: /images/20220115/76713aa96fad44f599172c0be3a78883.png