How to Find Processlist Thread id in gdb !!!!!GDB 使用

分手后的思念是犯贱 2022-03-30 13:38 83阅读 0赞

# https://mysqlentomologist.blogspot.jp/2017/07/ #

## Saturday, July 29, 2017 ##

### How to Find Processlist Thread id in gdb ###

I was involved in a discussion on some complex MySQL-related problem where we had to study backtraces of all threads in **gdb** (produced by the**thread apply all bt** command if you ever forgets this) in a hope to find out why MySQL hangs. In the process the question appeared on how to find the*thread id* for each thread to match it against previous collected outputs of**SHOW PROCESSLIST**. and**SHOW ENGINE INNODB STATUS**.  
  
I assumed I know the answer, as I had to find this out recently enough for[this blog post][] (and before that for the real customer case). The idea is simple. Find a frame where function has a parameter of**THD \*** type (usually named**thd**), like this:

> \#10 0x0000000000cb47fe in do\_command (**thd**=0x7f32512b7000)  
> at /usr/src/debug/percona-server-5.7.18-15/percona-server-5.7.18-15/sql/sql\_parse.cc:960

and check**thread\_id** item of this structure.  
  
In that my blog post it looked as simple as just referring to**thd** of**do\_command**'s frame without even checking much:

> (gdb)**thread 2**  
> \[Switching to thread 2 (Thread 0x7f7f5ce02b00 (LWP 9232))\]  
> \#0 pthread\_cond\_timedwait@@GLIBC\_2.3.2 ()  
> at ../nptl/sysdeps/unix/sysv/linux/x86\_64/pthread\_cond\_timedwait.S:238  
> 238 ../nptl/sysdeps/unix/sysv/linux/x86\_64/pthread\_cond\_timedwait.S: No such file or directory.  
> (gdb)**p do\_command::thd->thread\_id**  
> $9 = 14

I prefer to double check my suggestions before making them, so I immediately tried this with my CentOS 6.9 VM running recent Percona Server 5.7.x by default since that times when I worked at Percona:

> \[root@centos ~\]\#**gdb -p \`pidof mysqld\`**  
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)  
> ...  
> Loaded symbols for /usr/lib64/mysql/plugin/tokudb\_backup.so  
> 0x00007f550ad35383 in poll () from /lib64/libc.so.6  
> Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.209.el6\_9.2.x86\_64 jemalloc-3.6.0-1.el6.x86\_64 keyutils-libs-1.4-5.el6.x86\_64 krb5-libs-1.10.3-65.el6.x86\_64 libaio-0.3.107-10.el6.x86\_64 libcom\_err-1.41.12-23.el6.x86\_64 libgcc-4.4.7-18.el6.x86\_64 libselinux-2.0.94-7.el6.x86\_64 libstdc++-4.4.7-18.el6.x86\_64 nss-softokn-freebl-3.14.3-23.3.el6\_8.x86\_64 numactl-2.0.9-2.el6.x86\_64 openssl-1.0.1e-57.el6.x86\_64 zlib-1.2.3-29.el6.x86\_64  
> (gdb)**thread 1**  
> \[Switching to thread 1 (Thread 0x7f550d2b2820 (LWP 1978))\]\#0 0x00007f550ad35383 in poll () from /lib64/libc.so.6  
> (gdb)**p do\_command::thd->thread\_id**  
> No frame is currently executing in block do\_command(THD\*).  
> (gdb)**thread 2**  
> \[Switching to thread 2 (Thread 0x7f54d837b700 (LWP 2183))\]\#0 0x00007f550ad35383 in poll () from /lib64/libc.so.6  
> (gdb)**p do\_command::thd->thread\_id**  
> Cannot take address of method thread\_id.  
> (gdb)**call do\_command::thd->thread\_id()**  
> Cannot evaluate function -- may be inlined

As you can see I started to check threads one by one and apply that good old trick. Thread 1 had no frame executing**do\_command()**, but I did not gave up and proceeded to the next thread, as I knew I had at least one active connection (I checked the output of**SHOW PROCESSLIST**). There I had a surprise, no way to get**thread\_id** of**thd**. I used tab completion, so I know that**thread\_id** (variable or method) exists, but attempt to call it also failed as you can see.  
  
This is a problem with using**gdb**\-based "tricks" over the code that evolves/changes in time. Last time I used**p do\_command::thd->thread\_id** it was for MariaDB 10.1.x probably, and the item was there. But in MySQL 5.7 (and all forks based on it) there were many code changes, so we should be ready to changes in unexpected places.  
  
I had not added more comments on finding thread id to that discussion, made a note to myself and then, later, decided to check the source code of MySQL 5.7 (I did not have Percona 5.7 one at hand, but they hardly differs in such basic details) to find out what had changed in the**THD** structure so that**thread\_id** is not longer just a variable. I expect to see the structure defined in**sql/sql\_class.h** from the past, but**grep** will help to find this out even if it's no longer the case:

> \[root@centos mysql-server\]\#**grep -n "class THD" sql/\*.h**  
> sql/debug\_sync.h:27:class THD;  
> sql/derror.h:24:class THD;  
> sql/event\_data\_objects.h:40:class THD;  
> ...  
> **sql/sql\_class.h:1412:class THD :public MDL\_context\_owner,**  
> sql/sql\_class.h:4175: raise\_error() or raise\_warning() methods provided by class THD.  
> sql/sql\_cmd.h:25:class THD;  
> ...

I found the following there:

> class THD :public MDL\_context\_owner,  
> public Query\_arena,  
> public Open\_tables\_state  
> \{  
> ...  
> private:  
>  **my\_thread\_id m\_thread\_id;**public:  
> ...  
> /\*\*  
> Assign a value to m\_thread\_id by calling  
> **Global\_THD\_manager**::get\_new\_thread\_id().  
> \*/  
> void set\_new\_thread\_id();  
>  **my\_thread\_id thread\_id() const \{ return m\_thread\_id; \}**...

So, in MySQL 5.7**thread\_id()** is, indeed, a method that was inlined, and essentially it returns private**m\_thread\_id** item. Benefits of C++... I had highlighted**Global\_THD\_manager** singleton also as during my next gdb sessions I had found out that simple global list of threads is also gone and in 5.7 everything is done via that**Global\_THD\_manager**. This is a topic for some other post, though.  
  
At least now I know what to do in**gdb**:

> ...  
> (gdb)**thread 7**\[Switching to thread 7 (Thread 0x7f54d8236700 (LWP 2275))\]\#0 0x00007f550ad35383 in poll () from /lib64/libc.so.6  
> (gdb)**p do\_command::thd->m\_thread\_id**  
> $1 = 86  
> (gdb)**p do\_command::thd->m\_main\_security\_ctx**  
> $3 = \{ **m\_user** = \{m\_ptr = 0x7f5500fdaf90 "myuser", m\_length = 6,  
> m\_charset = 0x1ded640, m\_alloced\_length = 8, m\_is\_alloced = true\},  
> **m\_host** = \{m\_ptr = 0x7f54d98ab090 "localhost", m\_length = 9,  
> m\_charset = 0x1ded640, m\_alloced\_length = 16, m\_is\_alloced = true\},  
> **m\_ip** = \{m\_ptr = 0x7f54f0eb0210 "127.0.0.1", m\_length = 9,  
> m\_charset = 0x1ded640, m\_alloced\_length = 16, m\_is\_alloced = true\},  
> m\_host\_or\_ip = \{m\_ptr = 0x7f54d98ab090 "localhost", m\_length = 9,  
> m\_charset = 0x1ded640, m\_alloced\_length = 0, m\_is\_alloced = false\},  
> m\_external\_user = \{m\_ptr = 0x15167ab "", m\_length = 0,  
> m\_charset = 0x1ded640, m\_alloced\_length = 0, m\_is\_alloced = false\},  
> **m\_priv\_user** = "myuser", '\\000' <repeats 89 times>, m\_priv\_user\_length = 6,  
> m\_proxy\_user = '\\000' <repeats 161 times>, m\_proxy\_user\_length = 0,  
> **m\_priv\_host** = "localhost", '\\000' <repeats 51 times>,  
> m\_priv\_host\_length = 9, m\_master\_access = 1589248, m\_db\_access = 0,  
> m\_password\_expired = false\}  
> ...

So, I know that specific**thread 7**was for a session with Id**86** in the output of**SHOW PROCESSLIST**, and (from**m\_main\_security\_ctx**, also a new name for old things in 5.7) I know it was a session of**myuser** connecting locally.  
  
To summarize, there were notable changes in MySQL 5.7 in THD structure and threads management-related code in general, so make sure to re-check your "old gdb tricks" when you start working with 5.7. Reading the code helps.  
  
Unfortunately (for**gdb** beginners like me) a lot of C++ approaches were introduced, including singletons, iterators based on templates instead of simple double linked lists etc, so one has to work hard to adapt to these. I hope to discuss some of my further findings and new "C++ specific" and "MySQL 5.7 specific" approaches studying MySQL in**gdb** in my upcoming posts.

[this blog post]: http://mysqlentomologist.blogspot.com/2017/04/how-to-find-what-thread-had-executed.html