使用zookeeper封装组件curator的锁,发现zookeeper大量临时节点没有被删除

叁歲伎倆 2022-03-22 18:42 703阅读 0赞

转自：[https://www.cnblogs.com/xiaodu1993/articles/xiaodu1993.html][https_www.cnblogs.com_xiaodu1993_articles_xiaodu1993.html]

现象：zookeeper集群大量临时节点没有释放掉，导致集群响应很慢

分析过程：  
通过工具排查，发现大量创建lock对象的节点没有释放，奇怪的是release的时候应该删除的呀！只能看源码罗。

private static final String LOCK_NAME = "lock-";
    
    internals = new LockInternals(client, driver, path, lockName, maxLeases);
    
    this.path = ZKPaths.makePath(path, lockName);
    
    public static String makePath(String parent, String child)
    {
        StringBuilder path = new StringBuilder();
    
        joinPath(path, parent, child);
    
        return path.toString();
    }

发现了一段代码逻辑：InterProcessMutex加锁的时候，是创建'..../name/lock-'这样的节点，创建了一个父节点，父节点下面只有一个子节点，release的时候只删除了子节点'...../lock-'，'..../name'并没有删除，坑于此。明白了坑的地方，接下来就明了了：unlock的时候，顺便删除'..../name'这个节点。

思考为什么要这么设计？ 先来看一下底层实现。发现InterProcessSemaphore也是这样设计的，但是为什么要设置子节点还是没有搞懂，直到我看到了InterProcessReadWriteLock的是现实，里面有两个字段

private static final String READ_LOCK_NAME  = "__READ__";
     private static final String WRITE_LOCK_NAME = "__WRIT__";

再看到初始化读写锁的时候

public InterProcessReadWriteLock(CuratorFramework client, String basePath, byte[] lockData)
    {
        lockData = (lockData == null) ? null : Arrays.copyOf(lockData, lockData.length);
    
        writeMutex = new InternalInterProcessMutex
        (
            client,
            basePath,
            WRITE_LOCK_NAME,
            lockData,
            1,
            new SortingLockInternalsDriver()
            {
                @Override
                public PredicateResults getsTheLock(CuratorFramework client, List<String> children, String sequenceNodeName, int maxLeases) throws Exception
                {
                    return super.getsTheLock(client, children, sequenceNodeName, maxLeases);
                }
            }
        );
    
        readMutex = new InternalInterProcessMutex
        (
            client,
            basePath,
            READ_LOCK_NAME,
            lockData,
            Integer.MAX_VALUE,
            new SortingLockInternalsDriver()
            {
                @Override
                public PredicateResults getsTheLock(CuratorFramework client, List<String> children, String sequenceNodeName, int maxLeases) throws Exception
                {
                    return readLockPredicate(children, sequenceNodeName);
                }
            }
        );
    }

明白了，读写锁需要在一个路径下创建两个节点。至此一切明了。

之前怀疑作者想做缓存，没有删除父节点，看了一下源码？原来时curator兼容老版本的bug,因为为了支持'CONTAINER'这种类型，如果zookeeper版本低，本来应该是临时节点的，被存储为了持久化节点

String    fixForNamespace(String path, boolean isSequential)
    {
        if ( ensurePathNeeded.get() )
        {
            try
            {
                final CuratorZookeeperClient zookeeperClient = client.getZookeeperClient();
                RetryLoop.callWithRetry
                (
                    zookeeperClient,
                    new Callable<Object>()
                    {
                        @Override
                        public Object call() throws Exception
                        {
                            ZKPaths.mkdirs(zookeeperClient.getZooKeeper(), ZKPaths.makePath("/", namespace), true, client.getAclProvider(), true);
                            return null;
                        }
                    }
                );
                ensurePathNeeded.set(false);
            }
            catch ( Exception e )
            {
                client.logError("Ensure path threw exception", e);
            }
        }
    
        return ZKPaths.fixForNamespace(namespace, path, isSequential);
    }
    //这是创建父节点的入口
    ZKPaths.mkdirs(zookeeperClient.getZooKeeper(), ZKPaths.makePath("/", namespace), true, client.getAclProvider(), true);
    
    zookeeper.create(subPath, new byte[0], acl, getCreateMode(asContainers));
    // asContainers = true 获取的CONTAINER类型的节点
    return asContainers ? getContainerCreateMode() : CreateMode.PERSISTENT;
    
    //兼容版本有段代码比较坑
    try
            {
                localCreateMode = CreateMode.valueOf("CONTAINER");
            }
            catch ( IllegalArgumentException ignore )
            {
                localCreateMode = NON_CONTAINER_MODE;
                log.warn("The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.");
            }
    低版本的没有这种类型，就处理成了
    private static final CreateMode NON_CONTAINER_MODE = CreateMode.PERSISTENT;

这算是curator在处理版本兼容时的bug，踩了版本兼容的坑.

[https_www.cnblogs.com_xiaodu1993_articles_xiaodu1993.html]: https://www.cnblogs.com/xiaodu1993/articles/xiaodu1993.html