java生产环境调优(6) 模拟一次死锁,以及死锁的定位

阳光穿透心脏的1/2处 2023-06-16 02:25 20阅读 0赞

以前在看慕课网考试视频的时候, 分析cpu飙高 ,很自然的想到使用jstack去进行定位分析,相对而言,这一块还是比价的容易去定位的.  
后来在一次面试的时候,面试官问我你在现实环境中遇到过死锁么.  
当时还是比较蒙的,现实生活都很难遇到多线程,何来死锁.  
后来面试官又问 你知道怎么定位死锁么.这个可把我难住了,面试官接着说 就是你刚才说的jstack,那个时候才恍然大悟,死锁本质不就是cpu卡住了么.jstack不就是看cpu执行栈的么,惭愧啊,惭愧.

--------------------

ok ,回到正题,最近看到极客时间的老师的讲到死锁的问题,决定搞代码试一试

死锁本质上是2个线程抢2个资源,要么都抢到,要么都抢不到,偏偏有一种可能2个线程各自抢到一个,互相等对方释放,这个时候就会造成僵局,造成永久阻塞

ok,上代码.

public class DeadLock extends Thread { 
    
        private String firstKey;
        private String secondKey;
    
        public DeadLock(String firstKey,String secondKey){ 
            this.firstKey=firstKey;
            this.secondKey=secondKey;
        }
    
        @Override
        public void run() { 
            synchronized (firstKey){ 
                System.out.println(Thread.currentThread().getName()+"获得第一个锁,马上尝试获取第二个锁");
    
                try { 
                    Thread.sleep(1000);
                    synchronized (secondKey){ 
                        System.out.println(Thread.currentThread().getName()+"获取到第二个锁");
                    }
                } catch (InterruptedException e) { 
                    e.printStackTrace();
                }
            }
    
        }
    
    
        public static void main(String[] args) { 
    
            String firstKey="firstKey";
            String secondKey="secondKey";
            DeadLock deadLockTest1=new DeadLock(firstKey,secondKey);
            DeadLock deadLockTest2=new DeadLock(secondKey,firstKey);
    
            deadLockTest1.start();
            deadLockTest2.start();
        }
    }

首先定义两把锁,firstKey和secondKey 分别开2个线程,  
一个是先获取firstKey 再获取secondKey  
一个是先获取secondKey 再获取 firstKey

为了防止任何一个线程获取成功后继续获取第二个锁,中间加了一个延时.

ok 跑起来看一看

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2E4OTcxODA2NzM_size_16_color_FFFFFF_t_70]

可以看到两个线程都没有继续往下走,死锁演示成功了.

那么如何定位了?之前说了很简单 jstack

先jps 看下这个应用的进程id,再jstack导出线程栈看看

2019-11-23 22:31:39
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode):
    
    "Attach Listener" #14 daemon prio=9 os_prio=31 tid=0x00007fb7fc8e9800 nid=0x5903 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "DestroyJavaVM" #13 prio=5 os_prio=31 tid=0x00007fb7fc0a3000 nid=0x2703 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "Thread-1" #12 prio=5 os_prio=31 tid=0x00007fb7fc0a2000 nid=0x5803 waiting for monitor entry [0x000070000c45d000]
       java.lang.Thread.State: BLOCKED (on object monitor)
    	at com.example.demo.jikexueyuan.DeadLock.run(DeadLock.java:21)
    	- waiting to lock <0x000000076acf7da0> (a java.lang.String)
    	- locked <0x000000076acf7dd8> (a java.lang.String)
    
    "Thread-0" #11 prio=5 os_prio=31 tid=0x00007fb7fc05a800 nid=0x5703 waiting for monitor entry [0x000070000c35a000]
       java.lang.Thread.State: BLOCKED (on object monitor)
    	at com.example.demo.jikexueyuan.DeadLock.run(DeadLock.java:21)
    	- waiting to lock <0x000000076acf7dd8> (a java.lang.String)
    	- locked <0x000000076acf7da0> (a java.lang.String)
    
    "Service Thread" #10 daemon prio=9 os_prio=31 tid=0x00007fb7fe939800 nid=0xa803 runnable [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C1 CompilerThread3" #9 daemon prio=9 os_prio=31 tid=0x00007fb7fe029800 nid=0x5503 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C2 CompilerThread2" #8 daemon prio=9 os_prio=31 tid=0x00007fb7fd846000 nid=0x4203 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C2 CompilerThread1" #7 daemon prio=9 os_prio=31 tid=0x00007fb7fd845800 nid=0x3f03 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C2 CompilerThread0" #6 daemon prio=9 os_prio=31 tid=0x00007fb7fc05a000 nid=0x4403 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "Monitor Ctrl-Break" #5 daemon prio=5 os_prio=31 tid=0x00007fb7fc064000 nid=0x3e03 runnable [0x000070000bc45000]
       java.lang.Thread.State: RUNNABLE
    	at java.net.SocketInputStream.socketRead0(Native Method)
    	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    	at java.net.SocketInputStream.read(SocketInputStream.java:171)
    	at java.net.SocketInputStream.read(SocketInputStream.java:141)
    	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    	- locked <0x000000076adcb018> (a java.io.InputStreamReader)
    	at java.io.InputStreamReader.read(InputStreamReader.java:184)
    	at java.io.BufferedReader.fill(BufferedReader.java:161)
    	at java.io.BufferedReader.readLine(BufferedReader.java:324)
    	- locked <0x000000076adcb018> (a java.io.InputStreamReader)
    	at java.io.BufferedReader.readLine(BufferedReader.java:389)
    	at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)
    
    "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fb7fc039000 nid=0x3c03 runnable [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fb7fb00c000 nid=0x4d03 in Object.wait() [0x000070000ba3f000]
       java.lang.Thread.State: WAITING (on object monitor)
    	at java.lang.Object.wait(Native Method)
    	- waiting on <0x000000076ab08ed0> (a java.lang.ref.ReferenceQueue$Lock)
    	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
    	- locked <0x000000076ab08ed0> (a java.lang.ref.ReferenceQueue$Lock)
    	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
    	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
    
    "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fb7fc814800 nid=0x4e03 in Object.wait() [0x000070000b93c000]
       java.lang.Thread.State: WAITING (on object monitor)
    	at java.lang.Object.wait(Native Method)
    	- waiting on <0x000000076ab06bf8> (a java.lang.ref.Reference$Lock)
    	at java.lang.Object.wait(Object.java:502)
    	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    	- locked <0x000000076ab06bf8> (a java.lang.ref.Reference$Lock)
    	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
    
    "VM Thread" os_prio=31 tid=0x00007fb7fe024800 nid=0x3703 runnable 
    
    "GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fb7fc004000 nid=0x2007 runnable 
    
    "GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fb7fc004800 nid=0x2a03 runnable 
    
    "GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fb7fc005000 nid=0x2c03 runnable 
    
    "GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fb7fc005800 nid=0x2e03 runnable 
    
    "GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007fb7fd800000 nid=0x5303 runnable 
    
    "GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007fb7fc006800 nid=0x2f03 runnable 
    
    "GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007fb7fd801000 nid=0x3103 runnable 
    
    "GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007fb7fd801800 nid=0x3203 runnable 
    
    "GC task thread#8 (ParallelGC)" os_prio=31 tid=0x00007fb7fd802000 nid=0x3403 runnable 
    
    "GC task thread#9 (ParallelGC)" os_prio=31 tid=0x00007fb7fc007000 nid=0x3603 runnable 
    
    "VM Periodic Task Thread" os_prio=31 tid=0x00007fb7fc8e9000 nid=0xa603 waiting on condition 
    
    JNI global references: 15
    
    
    Found one Java-level deadlock:
    =============================
    "Thread-1":
      waiting to lock monitor 0x00007fb7fc023358 (object 0x000000076acf7da0, a java.lang.String),
      which is held by "Thread-0"
    "Thread-0":
      waiting to lock monitor 0x00007fb7fc020968 (object 0x000000076acf7dd8, a java.lang.String),
      which is held by "Thread-1"
    
    Java stack information for the threads listed above:
    ===================================================
    "Thread-1":
    	at com.example.demo.jikexueyuan.DeadLock.run(DeadLock.java:21)
    	- waiting to lock <0x000000076acf7da0> (a java.lang.String)
    	- locked <0x000000076acf7dd8> (a java.lang.String)
    "Thread-0":
    	at com.example.demo.jikexueyuan.DeadLock.run(DeadLock.java:21)
    	- waiting to lock <0x000000076acf7dd8> (a java.lang.String)
    	- locked <0x000000076acf7da0> (a java.lang.String)
    
    Found 1 deadlock.

看最后一行,jstack真的是很智能,它自动的告诉我们发现了一个死锁,这个对开发这而言很友好,因为不需要开发者专门去定位了.

看到这里我立刻想到了另外一个问题,那就是使用非synchronize锁,以前面试官问我,java锁的分类,那个时候不太清楚.面试官问我除了jvm原生的,还有啥,当时我不知道啊.哎.

现在好歹知道了一些,今天突然来了兴趣试一下ReentrantLock

看看用这个来实现死锁,jstack会有什么反应

ok,上代码

import java.util.concurrent.locks.ReentrantLock;
    
    public class DeadLock2 extends Thread { 
        ReentrantLock firstLock;
        ReentrantLock secondLock;
    
        public DeadLock2(ReentrantLock firstLock, ReentrantLock secondLock) { 
            this.firstLock = firstLock;
            this.secondLock = secondLock;
        }
    
        @Override
        public void run() { 
            firstLock.lock();
            System.out.println(Thread.currentThread().getName() + "获得第一个锁,马上尝试获取第二个锁");
            try { 
                Thread.sleep(1000);
                secondLock.lock();
                System.out.println(Thread.currentThread().getName() + "获取到第二个锁");
            } catch (InterruptedException e) { 
                e.printStackTrace();
            }
        }
    
        public static void main(String[] args) { 
            ReentrantLock firstLock = new ReentrantLock();
            ReentrantLock secondLock = new ReentrantLock();
    
            DeadLock2 deadLockTest1=new DeadLock2(firstLock,secondLock);
            DeadLock2 deadLockTest2=new DeadLock2(secondLock,firstLock);
    
            deadLockTest1.start();
            deadLockTest2.start();
        }
    }

ok啦,跑起来看看

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2E4OTcxODA2NzM_size_16_color_FFFFFF_t_70 1]  
效果是一样的,也出现了死锁,老规矩,jstack 看下

2019-11-23 22:51:05
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode):
    
    "Attach Listener" #14 daemon prio=9 os_prio=31 tid=0x00007fc5c004b800 nid=0x5903 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "DestroyJavaVM" #13 prio=5 os_prio=31 tid=0x00007fc5bf889800 nid=0x2803 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "Thread-1" #12 prio=5 os_prio=31 tid=0x00007fc5bf830800 nid=0x5803 waiting on condition [0x000070000af4d000]
       java.lang.Thread.State: WAITING (parking)
    	at sun.misc.Unsafe.park(Native Method)
    	- parking to wait for  <0x000000076acfda50> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    	at com.example.demo.jikexueyuan.DeadLock2.run(DeadLock2.java:20)
    
    "Thread-0" #11 prio=5 os_prio=31 tid=0x00007fc5c2800000 nid=0xa503 waiting on condition [0x000070000ae4a000]
       java.lang.Thread.State: WAITING (parking)
    	at sun.misc.Unsafe.park(Native Method)
    	- parking to wait for  <0x000000076acfda80> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    	at com.example.demo.jikexueyuan.DeadLock2.run(DeadLock2.java:20)
    
    "Service Thread" #10 daemon prio=9 os_prio=31 tid=0x00007fc5bda46800 nid=0x5603 runnable [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C1 CompilerThread3" #9 daemon prio=9 os_prio=31 tid=0x00007fc5bda45800 nid=0x5503 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C2 CompilerThread2" #8 daemon prio=9 os_prio=31 tid=0x00007fc5bda45000 nid=0x3f03 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C2 CompilerThread1" #7 daemon prio=9 os_prio=31 tid=0x00007fc5bf82c800 nid=0x4203 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "C2 CompilerThread0" #6 daemon prio=9 os_prio=31 tid=0x00007fc5bf82b800 nid=0x4403 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "Monitor Ctrl-Break" #5 daemon prio=5 os_prio=31 tid=0x00007fc5bf82a000 nid=0x3d03 runnable [0x000070000a735000]
       java.lang.Thread.State: RUNNABLE
    	at java.net.SocketInputStream.socketRead0(Native Method)
    	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    	at java.net.SocketInputStream.read(SocketInputStream.java:171)
    	at java.net.SocketInputStream.read(SocketInputStream.java:141)
    	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    	- locked <0x000000076adcb018> (a java.io.InputStreamReader)
    	at java.io.InputStreamReader.read(InputStreamReader.java:184)
    	at java.io.BufferedReader.fill(BufferedReader.java:161)
    	at java.io.BufferedReader.readLine(BufferedReader.java:324)
    	- locked <0x000000076adcb018> (a java.io.InputStreamReader)
    	at java.io.BufferedReader.readLine(BufferedReader.java:389)
    	at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)
    
    "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fc5be001000 nid=0x4503 runnable [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
    
    "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fc5c0046800 nid=0x4903 in Object.wait() [0x000070000a52f000]
       java.lang.Thread.State: WAITING (on object monitor)
    	at java.lang.Object.wait(Native Method)
    	- waiting on <0x000000076ab08ed0> (a java.lang.ref.ReferenceQueue$Lock)
    	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
    	- locked <0x000000076ab08ed0> (a java.lang.ref.ReferenceQueue$Lock)
    	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
    	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
    
    "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fc5c0802000 nid=0x3303 in Object.wait() [0x000070000a42c000]
       java.lang.Thread.State: WAITING (on object monitor)
    	at java.lang.Object.wait(Native Method)
    	- waiting on <0x000000076ab06bf8> (a java.lang.ref.Reference$Lock)
    	at java.lang.Object.wait(Object.java:502)
    	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    	- locked <0x000000076ab06bf8> (a java.lang.ref.Reference$Lock)
    	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
    
    "VM Thread" os_prio=31 tid=0x00007fc5bf800800 nid=0x3203 runnable 
    
    "GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fc5bd809000 nid=0x2107 runnable 
    
    "GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fc5bd809800 nid=0x2a03 runnable 
    
    "GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fc5bd81b000 nid=0x5403 runnable 
    
    "GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fc5bd81b800 nid=0x2d03 runnable 
    
    "GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007fc5c0000000 nid=0x5303 runnable 
    
    "GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007fc5c0001000 nid=0x5103 runnable 
    
    "GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007fc5c0001800 nid=0x4f03 runnable 
    
    "GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007fc5c0002000 nid=0x4e03 runnable 
    
    "GC task thread#8 (ParallelGC)" os_prio=31 tid=0x00007fc5c0002800 nid=0x4d03 runnable 
    
    "GC task thread#9 (ParallelGC)" os_prio=31 tid=0x00007fc5c0003800 nid=0x3103 runnable 
    
    "VM Periodic Task Thread" os_prio=31 tid=0x00007fc5bda47000 nid=0xa703 waiting on condition 
    
    JNI global references: 15
    
    
    Found one Java-level deadlock:
    =============================
    "Thread-1":
      waiting for ownable synchronizer 0x000000076acfda50, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
      which is held by "Thread-0"
    "Thread-0":
      waiting for ownable synchronizer 0x000000076acfda80, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
      which is held by "Thread-1"
    
    Java stack information for the threads listed above:
    ===================================================
    "Thread-1":
    	at sun.misc.Unsafe.park(Native Method)
    	- parking to wait for  <0x000000076acfda50> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    	at com.example.demo.jikexueyuan.DeadLock2.run(DeadLock2.java:20)
    "Thread-0":
    	at sun.misc.Unsafe.park(Native Method)
    	- parking to wait for  <0x000000076acfda80> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    	at com.example.demo.jikexueyuan.DeadLock2.run(DeadLock2.java:20)
    
    Found 1 deadlock.

也告诉了我们发现一个死锁

ReentrantLock 还有很多其他的功能比如定义公平锁,非公平锁, 条件等待 唤醒 等等. 用法上还是比synchronize要难很多的,而且一不小心用错了 往往就比较悲剧了.

平时开发中如果没有特别的情况,使用synchronize 就足够了,毕竟锁的获取释放等工作交给jvm去做就好了.

然而在面试中很多面试官都喜欢来一点深度.考察考察你

还能怎么办呢? 加油吧

![在这里插入图片描述][20191123230144508.png]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2E4OTcxODA2NzM_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191123222951284.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2E4OTcxODA2NzM=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2E4OTcxODA2NzM_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191123225021168.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2E4OTcxODA2NzM=,size_16,color_FFFFFF,t_70
[20191123230144508.png]: https://img-blog.csdnimg.cn/20191123230144508.png