Elasticsearch索引的数据存储路径是如何确定的

女爷i 2023-07-15 02:53 29阅读 0赞

Elasticsearch中，在node的配置中可以指定`path.data`用来作为节点数据的存储目录，而且我们可以指定多个值来作为数据存储的路径，那么Elasticsearch是如何判断应该存储到哪个路径下呢？今天我就记录一下这个问题。

## Elasticsearch的索引创建过程 ##

1.  集群master收到创建索引的请求后，经过创建索引的一些步骤，最终会将索引创建完成的请求提交到ClusterState
2.  master将根据ClusterState分发给所有节点
3.  涉及创建shard的节点会读取本地可用的`path.data`，然后依据一定的规则获取路径。
4.  创建基本shard路径，保存基本的shard信息。

## 如何确定在哪个目录下 ##

### 源码 ###

主要调用的是ShardPath的selectNewPathForShard方法

for (NodeEnvironment.NodePath nodePath : env.nodePaths()) {
                    totFreeSpace = totFreeSpace.add(BigInteger.valueOf(nodePath.fileStore.getUsableSpace()));
                }
    
                // TODO: this is a hack!!  We should instead keep track of incoming (relocated) shards since we know
                // how large they will be once they're done copying, instead of a silly guess for such cases:
    
                // Very rough heuristic of how much disk space we expect the shard will use over its lifetime, the max of current average
                // shard size across the cluster and 5% of the total available free space on this node:
                BigInteger estShardSizeInBytes = BigInteger.valueOf(avgShardSizeInBytes).max(totFreeSpace.divide(BigInteger.valueOf(20)));
    
                // TODO - do we need something more extensible? Yet, this does the job for now...
                final NodeEnvironment.NodePath[] paths = env.nodePaths();
    
                // If no better path is chosen, use the one with the most space by default
                NodeEnvironment.NodePath bestPath = getPathWithMostFreeSpace(env);
    
                if (paths.length != 1) {
                    Map<NodeEnvironment.NodePath, Long> pathToShardCount = env.shardCountPerPath(shardId.getIndex());
    
                    // Compute how much space there is on each path
                    final Map<NodeEnvironment.NodePath, BigInteger> pathsToSpace = new HashMap<>(paths.length);
                    for (NodeEnvironment.NodePath nodePath : paths) {
                        FileStore fileStore = nodePath.fileStore;
                        BigInteger usableBytes = BigInteger.valueOf(fileStore.getUsableSpace());
                        pathsToSpace.put(nodePath, usableBytes);
                    }
    
                    bestPath = Arrays.stream(paths)
                            // Filter out paths that have enough space
                            .filter((path) -> pathsToSpace.get(path).subtract(estShardSizeInBytes).compareTo(BigInteger.ZERO) > 0)
                            // Sort by the number of shards for this index
                            .sorted((p1, p2) -> {
                                    int cmp = Long.compare(pathToShardCount.getOrDefault(p1, 0L),
                                        pathToShardCount.getOrDefault(p2, 0L));
                                    if (cmp == 0) {
                                        // if the number of shards is equal, tie-break with the number of total shards
                                        cmp = Integer.compare(dataPathToShardCount.getOrDefault(p1.path, 0),
                                                dataPathToShardCount.getOrDefault(p2.path, 0));
                                        if (cmp == 0) {
                                            // if the number of shards is equal, tie-break with the usable bytes
                                            cmp = pathsToSpace.get(p2).compareTo(pathsToSpace.get(p1));
                                        }
                                    }
                                    return cmp;
                                })
                            // Return the first result
                            .findFirst()
                            // Or the existing best path if there aren't any that fit the criteria
                            .orElse(bestPath);
                }
    
                statePath = bestPath.resolve(shardId);
                dataPath = statePath;
            }

### 过程分析 ###

1.  首先判断是否自定义了`path.data`，没有自定义就在默认路径下创建
2.  自定义的情况下确保节点下最少有5%的空间可以使用
3.  获取所有的paths，
4.  然后设置默认最佳的path是当前拥有最多空间的path
5.  遍历所有的paths，首先过滤掉没有空间的path，如果最终没有符合的，就返回4步骤的path，否则继续6步骤
6.  按照规则对paths排序，首先判断每个path下该索引的shard数，优先返回含有本索引的shard数最少的path；  
    当条件结果相同，对比每个path中包含有的shard总数（所有索引的），返回包含shard数最少的path；  
    当2条件结果相同，对比可用空间，返回可用空间最大的path
7.  生成相应的路径，创建目录等信息。