几种常见的OOM异常解析

野性酷女 2022-02-17 08:23 326阅读 0赞

OOM为out of memory的简称，称之为内存溢出。

程序中常见的打印有如下几类：

## 一： ##

如图：

![SouthEast][]

Java应用程序在启动时会指定所需要的内存大小，其主要被分割成两个不同的部分，分别为Head space(堆空间-Xmx指定)和Permegen(永久代-XX:MaxPermSize指定)，

通常来说，造成如上图异常的基本上程序代码问题而造成的内存泄露。这种异常，通过dump+EMA可以轻松定位。（EMA虽功能强大，但对机器性能内存要求极高）

## 二：  Java.lang.OutOfMemeoryError:GC overhead limit exceeded  如上异常，即程序在垃圾回收上花费了98%的时间，却收集不回2%的空间，通常这样的异常伴随着CPU的冲高。定位方法同上。      ##

## 三：  Java.lang.OutOfMemoryError: PermGen space（JAVA8引入了Metaspace区域）  永久代内存被耗尽，永久代的作用是存储每个类的信息，如类加载器引用、运行池常量池、字段数据、方法数据、方法代码、方法字节码等。基本可以推断PermGen占用大小取决于被加载的数量以及类的大小。定位方法同上。       还有很多OOM异常，甚至会触发操作系统的OOM killer去杀掉其它进程。      ##

## 四：  本节主要讨论下面一种OOM，如图  ![SouthEast 1][]     产生这种异常的原因是由于系统在不停地创建大量的线程，且不进行释放。系统的内存是有限的，分配给JAVA应用的程序也是有限的，系统自身能允许创建的最大线程数计算规则：  (MaxProcessMemory-JVMMemory-ReservedOsMemory)/ThreadStackSize  其中  MaxProcessMemory：指的是一个进程的最大内存  JVMMemory ：JVM内存  ReservedOsMemory：保留的操作系统内存  ThreadStackSize：线程栈的大小  从公式中可以得出结论，系统可创建线程数量与分配给JVM内存大小成反比。       在java程序中，创建一个线程，虚拟机会在JVM内存创建一个Thread对象同时创建一个操作系统线程，且系统线程的内存不占用JVMMemory，而占用系统中剩下的内存，即(MaxProcessMemory - JVMMemory - ReservedOsMemory)。  结论很明显：程序创建的线程不可能无限大。       先讨论第一种情况，即经jstack或dump后，线程的数量确在系统要求的阀值内，报上面异常，该如何？  1.参考之前的参数，可以修改两个变量JVMMemory和ThreadStackSize来满足更多线程创建的要求。  \-Xss1024k -Xms4096m -Xmx4096m  2.查看是否操作系统限制了可创建线程数量  执行ulimit -u 可以查看当前用户可创建线程量，如果不满足要求，可以通过修改配置文件调整其大小：  相关配置文件在etc/security/limit.d/XX-nproc.conf中       由于上面配置不合理造成而产生异常，个人认为应属小概率事件。       第二种情况，是程序中存在线程泄露，代码本身有问题，在某些场景，条件下存在线程不断创建而不销毁的BUG。  先看一段代码（取自业务真实代码稍加改写）： ##

package com.zte.sunquan.demo.netty.server;
    import com.google.common.collect.Maps;
    import java.util.Map;
    import java.util.concurrent.ExecutorService;
    import java.util.concurrent.Executors;
    public class MainController { 
        private ExecutorService executor = Executors.newSingleThreadExecutor();
        private static Map<String, MainController> map = Maps.newConcurrentMap();
        public void submitTask() { 
            for (int i = 0; i < 10; i++) { 
                executor.submit(new Runnable() { 
                    @Override
                    public void run() { 
           System.out.println(Thread.currentThread().getName() + "正处理");
                    }
                });
            }
        }
        public static void main(String[] args) throws InterruptedException { 
            while (true) { 
                MainController ct1 = new MainController();
                map.put("1", ct1);
                ct1.submitTask();
                map.remove("1");
            }
        }
    }

如上表的程序就存在线程泄露，开发人员误以为从map中移除，对象就可以释放，最终交由GC回收，但其实在创建的对象中创建的线程池，由于未关闭，执行完任务后，进入waiting状态，是不会释放的，随着程序运行，线程会越积越多，最终导致OOM的异常。

又或者将main函数，改为一个监听入口，在一般甚至短暂的压力测试中，虽然线程较多，但系统仍可以正常进行，误以为系统运行正常，但大业务、多节点、网络震荡、长时间，商用等能够大量触发该监听的场景中，这类问题才以最终kill系统进程暴露出来。

此外，如上表中的代码，线程池创建的进程，采用Executors中DefaultThreadFactory提供的pool-XXX-thread-XXX命名规则，诚然可以通过jstack打印线程调用栈，但如下面的打印：

.....
    "pool-2577-thread-1" #8681 prio=5 os_prio=0 tid=0x00007f8ea4de7800 nid=0x6a0b waiting on condition [0x00007f8cae227000]
       java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000057136f4b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
     
    "pool-2576-thread-1" #8680 prio=5 os_prio=0 tid=0x00007f8ea4de6000 nid=0x6a0a waiting on condition [0x00007f8cae72c000]
       java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000057136fa20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    ......

完整打印一直是从pool-1-thread-1到pool-5406-thread-1，很明显代码一直在创建线程池，但从打印输出，却无法分析出罪魁祸首是谁？接下来自然，会去系统日志中搜索pool-开头的线程打印，期望能找到线索，遗憾的是，什么信息也没有。此时问题定位，卡壳了。但可以肯定，代码肯定有前面例子中类似的写法。此时，走查代码，不失为一个好办法。

线索就此断掉了。。。。。

但考虑到是使用JDK并发包提供的功能，那在JDK中Executors类中，创建线程时，增加打印，强制将调用栈的信息打印出来，是否可以找到蛛丝马迹？

确定JDK版本--->获取源码---->修改代码--->编译-------->合入rt.jar ------->复现

我们将最终信息输入到了var/log目录下，需要提前将该目录的权限开放chmod 777 -R

![SouthEast 2][]

输出日志中，对一些连续创建的线程，观察打印信息，幸运女神出现了，OpticalTopologyAdapter，抓到你了。问题解决！

这种问题的出现，归根到底，是没有对线程池进行管理，开发人员对于线程池的滥用，影响了程序稳定性。

修改的源文件如下：

package java.util.concurrent;
     
    import sun.security.util.SecurityConstants;
     
    import java.io.File;
    import java.io.FileWriter;
    import java.io.IOException;
    import java.security.AccessControlContext;
    import java.security.AccessControlException;
    import java.security.AccessController;
    import java.security.PrivilegedAction;
    import java.security.PrivilegedActionException;
    import java.security.PrivilegedExceptionAction;
    import java.util.Collection;
    import java.util.List;
    import java.util.concurrent.atomic.AtomicInteger;
     
    public class Executors { 
     
       
        public static ExecutorService newFixedThreadPool(int nThreads) { 
            StackTraceElement[] elements = Thread.currentThread().getStackTrace();
            recordThreadStackInfo("SQ:FixThreadPool: " + Thread.currentThread().getName() + nThreads, elements);
            return new ThreadPoolExecutor(nThreads, nThreads,
                    0L, TimeUnit.MILLISECONDS,
                    new LinkedBlockingQueue<Runnable>());
        }
     
       
        public static ExecutorService newWorkStealingPool(int parallelism) { 
            return new ForkJoinPool
                    (parallelism,
                            ForkJoinPool.defaultForkJoinWorkerThreadFactory,
                            null, true);
        }
     
       
        public static ExecutorService newWorkStealingPool() { 
            return new ForkJoinPool
                    (Runtime.getRuntime().availableProcessors(),
                            ForkJoinPool.defaultForkJoinWorkerThreadFactory,
                            null, true);
        }
     
       
        public static ExecutorService newFixedThreadPool(int nThreads, ThreadFactory threadFactory) { 
            StackTraceElement[] elements = Thread.currentThread().getStackTrace();
            recordThreadStackInfo("SQ2:FixThreadPool: " + Thread.currentThread().getName() + nThreads, elements);
            return new ThreadPoolExecutor(nThreads, nThreads,
                    0L, TimeUnit.MILLISECONDS,
                    new LinkedBlockingQueue<Runnable>(),
                    threadFactory);
        }
     
        
        public static ExecutorService newSingleThreadExecutor() { 
            StackTraceElement[] elements = Thread.currentThread().getStackTrace();
            recordThreadStackInfo("SQ1:SingleThread: " + Thread.currentThread().getName(), elements);
            return new FinalizableDelegatedExecutorService
                    (new ThreadPoolExecutor(1, 1,
                            0L, TimeUnit.MILLISECONDS,
                            new LinkedBlockingQueue<Runnable>()));
        }
     
        
        public static ExecutorService newSingleThreadExecutor(ThreadFactory threadFactory) { 
            StackTraceElement[] elements = Thread.currentThread().getStackTrace();
            recordThreadStackInfo("SQ2:SingleThread: " + Thread.currentThread().getName(), elements);
            return new FinalizableDelegatedExecutorService
                    (new ThreadPoolExecutor(1, 1,
                            0L, TimeUnit.MILLISECONDS,
                            new LinkedBlockingQueue<Runnable>(),
                            threadFactory));
        }
     
        
        public static ExecutorService newCachedThreadPool() { 
            return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
                    60L, TimeUnit.SECONDS,
                    new SynchronousQueue<Runnable>());
        }
     
        
        public static ExecutorService newCachedThreadPool(ThreadFactory threadFactory) { 
            return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
                    60L, TimeUnit.SECONDS,
                    new SynchronousQueue<Runnable>(),
                    threadFactory);
        }
     
        
        public static ScheduledExecutorService newSingleThreadScheduledExecutor() { 
            return new DelegatedScheduledExecutorService
                    (new ScheduledThreadPoolExecutor(1));
        }
     
        
        public static ScheduledExecutorService newSingleThreadScheduledExecutor(ThreadFactory threadFactory) { 
            return new DelegatedScheduledExecutorService
                    (new ScheduledThreadPoolExecutor(1, threadFactory));
        }
     
        
        public static ScheduledExecutorService newScheduledThreadPool(int corePoolSize) { 
            return new ScheduledThreadPoolExecutor(corePoolSize);
        }
     
       
        public static ScheduledExecutorService newScheduledThreadPool(
                int corePoolSize, ThreadFactory threadFactory) { 
            return new ScheduledThreadPoolExecutor(corePoolSize, threadFactory);
        }
     
        
        public static ExecutorService unconfigurableExecutorService(ExecutorService executor) { 
            if (executor == null)
                throw new NullPointerException();
            return new DelegatedExecutorService(executor);
        }
     
        
        public static ScheduledExecutorService unconfigurableScheduledExecutorService(ScheduledExecutorService executor) { 
            if (executor == null)
                throw new NullPointerException();
            return new DelegatedScheduledExecutorService(executor);
        }
     
        
        public static ThreadFactory defaultThreadFactory() { 
            return new DefaultThreadFactory();
        }
     
        
        public static ThreadFactory privilegedThreadFactory() { 
            return new PrivilegedThreadFactory();
        }
     
        
        public static <T> Callable<T> callable(Runnable task, T result) { 
            if (task == null)
                throw new NullPointerException();
            return new RunnableAdapter<T>(task, result);
        }
     
       
        public static Callable<Object> callable(Runnable task) { 
            if (task == null)
                throw new NullPointerException();
            return new RunnableAdapter<Object>(task, null);
        }
     
        
        public static Callable<Object> callable(final PrivilegedAction<?> action) { 
            if (action == null)
                throw new NullPointerException();
            return new Callable<Object>() { 
                public Object call() { 
                    return action.run();
                }
            };
        }
     
       
        public static Callable<Object> callable(final PrivilegedExceptionAction<?> action) { 
            if (action == null)
                throw new NullPointerException();
            return new Callable<Object>() { 
                public Object call() throws Exception { 
                    return action.run();
                }
            };
        }
     
        
        public static <T> Callable<T> privilegedCallable(Callable<T> callable) { 
            if (callable == null)
                throw new NullPointerException();
            return new PrivilegedCallable<T>(callable);
        }
     
        
        public static <T> Callable<T> privilegedCallableUsingCurrentClassLoader(Callable<T> callable) { 
            if (callable == null)
                throw new NullPointerException();
            return new PrivilegedCallableUsingCurrentClassLoader<T>(callable);
        }
     
        // Non-public classes supporting the public methods
     
        
        static final class RunnableAdapter<T> implements Callable<T> { 
            final Runnable task;
            final T result;
     
            RunnableAdapter(Runnable task, T result) { 
                this.task = task;
                this.result = result;
            }
     
            public T call() { 
                task.run();
                return result;
            }
        }
     
        
        static final class PrivilegedCallable<T> implements Callable<T> { 
            private final Callable<T> task;
            private final AccessControlContext acc;
     
            PrivilegedCallable(Callable<T> task) { 
                this.task = task;
                this.acc = AccessController.getContext();
            }
     
            public T call() throws Exception { 
                try { 
                    return AccessController.doPrivileged(
                            new PrivilegedExceptionAction<T>() { 
                                public T run() throws Exception { 
                                    return task.call();
                                }
                            }, acc);
                } catch (PrivilegedActionException e) { 
                    throw e.getException();
                }
            }
        }
     
       
        static final class PrivilegedCallableUsingCurrentClassLoader<T> implements Callable<T> { 
            private final Callable<T> task;
            private final AccessControlContext acc;
            private final ClassLoader ccl;
     
            PrivilegedCallableUsingCurrentClassLoader(Callable<T> task) { 
                SecurityManager sm = System.getSecurityManager();
                if (sm != null) { 
                    // Calls to getContextClassLoader from this class
                    // never trigger a security check, but we check
                    // whether our callers have this permission anyways.
                    sm.checkPermission(SecurityConstants.GET_CLASSLOADER_PERMISSION);
     
                    // Whether setContextClassLoader turns out to be necessary
                    // or not, we fail fast if permission is not available.
                    sm.checkPermission(new RuntimePermission("setContextClassLoader"));
                }
                this.task = task;
                this.acc = AccessController.getContext();
                this.ccl = Thread.currentThread().getContextClassLoader();
            }
     
            public T call() throws Exception { 
                try { 
                    return AccessController.doPrivileged(
                            new PrivilegedExceptionAction<T>() { 
                                public T run() throws Exception { 
                                    Thread t = Thread.currentThread();
                                    ClassLoader cl = t.getContextClassLoader();
                                    if (ccl == cl) { 
                                        return task.call();
                                    } else { 
                                        t.setContextClassLoader(ccl);
                                        try { 
                                            return task.call();
                                        } finally { 
                                            t.setContextClassLoader(cl);
                                        }
                                    }
                                }
                            }, acc);
                } catch (PrivilegedActionException e) { 
                    throw e.getException();
                }
            }
        }
     
     
        private static void recordThreadStackInfo(String headInfo, StackTraceElement[] elements) { 
            StringBuffer buf = new StringBuffer(headInfo);
            for (int i = 0; i < elements.length; i++) { 
                buf.append("\n "
                        + elements[i].getClassName()
                        + "."
                        + elements[i].getMethodName()
                        + "("
                        + elements[i].getFileName()
                        + ":"
                        + elements[i].getLineNumber()
                        + ")\n");
            }
            recordInfo(buf.toString());
        }
     
        private static void recordInfo(String info) { 
            File f = new File("/var/log/threadsInfo.txt");
            if (!f.exists()) { 
                try { 
                    f.createNewFile();
                } catch (IOException e1) { 
                    e1.printStackTrace();
                }
            }
            FileWriter fileWriter = null;
            try { 
                fileWriter = new FileWriter(f, true);
                fileWriter.write(info);
                fileWriter.flush();
            } catch (IOException e1) { 
                e1.printStackTrace();
            } finally { 
                if (fileWriter != null)
                    try { 
                        fileWriter.close();
                    } catch (IOException e1) { 
                        e1.printStackTrace();
                    }
            }
        }
     
        private static void printStackTrace(String emsg) { 
            Exception e = new Exception(emsg);
            e.printStackTrace();
        }
     
        /** * The default thread factory */
        static class DefaultThreadFactory implements ThreadFactory { 
            private static final AtomicInteger poolNumber = new AtomicInteger(1);
            private final ThreadGroup group;
            private final AtomicInteger threadNumber = new AtomicInteger(1);
            private final String namePrefix;
     
            DefaultThreadFactory() { 
                SecurityManager s = System.getSecurityManager();
                group = (s != null) ? s.getThreadGroup() :
                        Thread.currentThread().getThreadGroup();
                namePrefix = "pool-" +
                        poolNumber.getAndIncrement() +
                        "-thread-";
            }
     
            public Thread newThread(Runnable r) { 
                Thread t = new Thread(group, r,
                        namePrefix + threadNumber.getAndIncrement(),
                        0);
                ///
                String threadNum = namePrefix + threadNumber.get();
                if (poolNumber.get() > 20 && namePrefix.startsWith("pool-")) { 
                    StackTraceElement[] elements = Thread.currentThread().getStackTrace();
                    recordThreadStackInfo("SQ2:SingleThread: " + threadNum + " name:" + t.getName(), elements);
                    Exception e = new Exception("SQ newThread Exception");
                    e.printStackTrace();
                }
                ///
                if (t.isDaemon())
                    t.setDaemon(false);
                if (t.getPriority() != Thread.NORM_PRIORITY)
                    t.setPriority(Thread.NORM_PRIORITY);
                return t;
            }
        }
     
        /** * Thread factory capturing access control context and class loader */
        static class PrivilegedThreadFactory extends DefaultThreadFactory { 
            private final AccessControlContext acc;
            private final ClassLoader ccl;
     
            PrivilegedThreadFactory() { 
                super();
                SecurityManager sm = System.getSecurityManager();
                if (sm != null) { 
                    // Calls to getContextClassLoader from this class
                    // never trigger a security check, but we check
                    // whether our callers have this permission anyways.
                    sm.checkPermission(SecurityConstants.GET_CLASSLOADER_PERMISSION);
     
                    // Fail fast
                    sm.checkPermission(new RuntimePermission("setContextClassLoader"));
                }
                this.acc = AccessController.getContext();
                this.ccl = Thread.currentThread().getContextClassLoader();
            }
     
            public Thread newThread(final Runnable r) { 
                return super.newThread(new Runnable() { 
                    public void run() { 
                        AccessController.doPrivileged(new PrivilegedAction<Void>() { 
                            public Void run() { 
                                Thread.currentThread().setContextClassLoader(ccl);
                                r.run();
                                return null;
                            }
                        }, acc);
                    }
                });
            }
        }
     
        /** * A wrapper class that exposes only the ExecutorService methods * of an ExecutorService implementation. */
        static class DelegatedExecutorService extends AbstractExecutorService { 
            private final ExecutorService e;
     
            DelegatedExecutorService(ExecutorService executor) { 
                e = executor;
            }
     
            public void execute(Runnable command) { 
                e.execute(command);
            }
     
            public void shutdown() { 
                e.shutdown();
            }
     
            public List<Runnable> shutdownNow() { 
                return e.shutdownNow();
            }
     
            public boolean isShutdown() { 
                return e.isShutdown();
            }
     
            public boolean isTerminated() { 
                return e.isTerminated();
            }
     
            public boolean awaitTermination(long timeout, TimeUnit unit)
                    throws InterruptedException { 
                return e.awaitTermination(timeout, unit);
            }
     
            public Future<?> submit(Runnable task) { 
                return e.submit(task);
            }
     
            public <T> Future<T> submit(Callable<T> task) { 
                return e.submit(task);
            }
     
            public <T> Future<T> submit(Runnable task, T result) { 
                return e.submit(task, result);
            }
     
            public <T> List<Future<T>> invokeAll(Collection<? extends Callable<T>> tasks)
                    throws InterruptedException { 
                return e.invokeAll(tasks);
            }
     
            public <T> List<Future<T>> invokeAll(Collection<? extends Callable<T>> tasks,
                                                 long timeout, TimeUnit unit)
                    throws InterruptedException { 
                return e.invokeAll(tasks, timeout, unit);
            }
     
            public <T> T invokeAny(Collection<? extends Callable<T>> tasks)
                    throws InterruptedException, ExecutionException { 
                return e.invokeAny(tasks);
            }
     
            public <T> T invokeAny(Collection<? extends Callable<T>> tasks,
                                   long timeout, TimeUnit unit)
                    throws InterruptedException, ExecutionException, TimeoutException { 
                return e.invokeAny(tasks, timeout, unit);
            }
        }
     
        static class FinalizableDelegatedExecutorService
                extends DelegatedExecutorService { 
            FinalizableDelegatedExecutorService(ExecutorService executor) { 
                super(executor);
            }
     
            protected void finalize() { 
                super.shutdown();
            }
        }
     
        /** * A wrapper class that exposes only the ScheduledExecutorService * methods of a ScheduledExecutorService implementation. */
        static class DelegatedScheduledExecutorService
                extends DelegatedExecutorService
                implements ScheduledExecutorService { 
            private final ScheduledExecutorService e;
     
            DelegatedScheduledExecutorService(ScheduledExecutorService executor) { 
                super(executor);
                e = executor;
                System.out.println("b");
            }
     
            public ScheduledFuture<?> schedule(Runnable command, long delay, TimeUnit unit) { 
                return e.schedule(command, delay, unit);
            }
     
            public <V> ScheduledFuture<V> schedule(Callable<V> callable, long delay, TimeUnit unit) { 
                return e.schedule(callable, delay, unit);
            }
     
            public ScheduledFuture<?> scheduleAtFixedRate(Runnable command, long initialDelay, long period, TimeUnit unit) { 
                return e.scheduleAtFixedRate(command, initialDelay, period, unit);
            }
     
            public ScheduledFuture<?> scheduleWithFixedDelay(Runnable command, long initialDelay, long delay, TimeUnit unit) { 
                return e.scheduleWithFixedDelay(command, initialDelay, delay, unit);
            }
        }
     
        /** * Cannot instantiate. */
        private Executors() { 
        }
    }

因此有了下面的几点建议：

1.如果使用JDK创建线程池，必须指定命名规则，参考

ThreadFactory build=new ThreadFactoryBuilder().setPriority(Thread.NORM_PRIORITY)
                                     .setDaemon(false)
                                     .setNameFormat("SQ-Creat-%d")
                                     .build();
    Executors.newSingleThreadScheduledExecutor(build);

但使用上面的方式只能设置线程名，而线程池的数目，是无法判断的。简而言之，即有SQ-Create-1,SQ-Create-2.SQ-Create-3，无法判断出这是一个线程池中三个线程，还是存在于两个线程池中。

其实，更简单的方法，只要实现ThreadFactory，稍等改动，则可以实现。（使用UserThreadFactory强制使用必须传入线程池名）

package com.zte.sunquan.demo.executor;
     
    import java.util.concurrent.ThreadFactory;
    import java.util.concurrent.atomic.AtomicInteger;
     
    /** * Created by 10184538 on 2018/1/4. */
    public class UserThreadFactory implements ThreadFactory { 
        private static final AtomicInteger poolNumber = new AtomicInteger(1);
        private final ThreadGroup group;
        private final AtomicInteger threadNumber = new AtomicInteger(1);
        private final String namePrefix;
     
        private UserThreadFactory(String threadPoolName) { 
            SecurityManager s = System.getSecurityManager();
            group = (s != null) ? s.getThreadGroup() :
                    Thread.currentThread().getThreadGroup();
            namePrefix = threadPoolName + "-" +
                    poolNumber.getAndIncrement() +
                    "-thread-";
        }
     
        public Thread newThread(Runnable r) { 
            Thread t = new Thread(group, r,
                    namePrefix + threadNumber.getAndIncrement(),
                    0);
            if (t.isDaemon())
                t.setDaemon(false);
            if (t.getPriority() != Thread.NORM_PRIORITY)
                t.setPriority(Thread.NORM_PRIORITY);
            return t;
        }
     
        public static UserThreadFactory build(String poolThreadName) { 
            return new UserThreadFactory(poolThreadName);
        }
    }

2.略

3.最后简单介绍下ONOS线程池的封装使用(以下参考ONOS1.5.2版本)

Onos中考虑到线程的管理，使用了：

SharedExecutors

SharedExecutorService

SharedScheduledExecutors

ShardScheduleExecutorServce

对JDK的Executors进行了封装，增加了部分管理。同时其特点提供的GroupedThreadFactory支持线程的子父节关系定义管理。如：

@Test
    public void test() throws InterruptedException { 
        //对每一个事件都要定义其所在的事件组，事件名，并保持相关继承关系
        CountDownLatch countDownLatch = new CountDownLatch(1);
        ExecutorService executorService =
                Executors.newFixedThreadPool(2, getFactory("sqGroup/a/b", "sqThread%d"));
        executorService.execute(() -> { 
            System.out.println("parent thread group name: " + Thread.currentThread().getThreadGroup().getParent().getName());
            System.out.println("current thread group name: " + Thread.currentThread().getThreadGroup().getName());
            System.out.println("current thread name: " + Thread.currentThread().getName());
            countDownLatch.countDown();
        });
        countDownLatch.await();
        System.out.println(Thread.currentThread().getThreadGroup().getName());
    }

打印：

parent thread group name: sqGroup/a
    
    current thread group name: sqGroup/a/b
    
    current thread name: sqThread0
    
    main

总结：本人故障定位中查到的N多OOM问题，原因不外乎以下3类

1、线程池不管理式滥用

2、本地缓存滥用（如只需要使用Node的id,但将整个Node所有信息都缓存)

3、特殊场景考虑不足（如采用单线程处理复杂业务，环境震荡+多设备下积压任务暴增）

--------------------

## 利用gc.log进行OOM监控 ##

针对JVM的监听，JDK默认提供了如jconsole、jvisualVM工具都非常好用，本节介绍利用打开gc的日志功能，进行OOM的监控。

首先需要对JAVA程序添加执行参数：

\-XX:+PrintGC 输出GC日志

\-XX:+PrintGCDetails 输出GC的详细日志

\-XX:+PrintGCTimeStamps 输出GC的时间戳（以基准时间的形式）

\-XX:+PrintGCDateStamps 输出GC的时间戳（以日期的形式，如 2013-05-04T21:53:59.234+0800）

\-XX:+PrintHeapAtGC 在进行GC的前后打印出堆的信息

\-Xloggc:../logs/gc.log 日志文件的输出路径

示例参数配置：

\-xmx100M -XX:+PrintGCDetails -Xloggc:../logs/gc.log -XX:+PrintGCDetails   -Xmx设置堆内存

输出日志的详细介绍：

![SouthEast 3][]

总共分配了100M堆内存空间，老年代+年轻代=堆

就堆内存，此次GC共回收了79186-75828=3356=3.279M内存，能回收空间已经非常少了。

同时从PSYoungGen回收10612-7254=3558=3.474M内存

3558-3356=202K，说明有202K的空间在年轻代释放，却传入年老代继续占用

下一条日志输出：

2018-01-03T10:53:52.281+0800: 11.052: \[Full GC (Allocation Failure) \[PSYoungGen: 7254K->7254K(24064K)\] \[ParOldGen: 68574K->68486K(68608K)\] 75828K->75740K(92672K), \[Metaspace: 3357K->3357K(1056768K)\], 0.6010057 secs\] \[Times: user=2.08 sys=0.00, real=0.60 secs\]

68574-68486=88K

75828-75740=88K

此次FullGC 只能释放老年代的88K空间

ps.对于分析gc.log，提供一种图形化工具辅助分析

gcviewer-1.3.6

[SouthEast]: /images/20220217/53f8fd4f136f4cfb9c8d6a8b3e8e82a3.png
[SouthEast 1]: /images/20220217/e7f8e92bfe6f482dbe60960ee9159929.png
[SouthEast 2]: /images/20220217/51374c9290944dfc9355069d632a20ba.png
[SouthEast 3]: /images/20220217/fdb5f367d505488cb4590bf309aaff84.png