Linux性能监控及调优资料

深藏阁楼爱情的钟 2023-06-19 13:27 17阅读 0赞

# **Linux的性能监控及调优总结，资料汇总。** #

[Linux Performance Profiling and  Monitoring][Linux Performance Profiling and _Monitoring]

[Linux Performance Tools][]

[Linux Performance  Analysis and Tools][Linux Performance _Analysis and Tools]

[Linux Profiling at Netflix][]

非常全面地介绍Linux系统性能监控，包括CPU、内存、硬盘、网络。

官方网站：[http://www.brendangregg.com/linuxperf.html][http_www.brendangregg.com_linuxperf.html]

[Linux Performance Analysis New Tools and Old Secrets][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 3][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 4][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 5][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 7][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 8][]

# Linux Performance #

This page links to various Linux performance material I've created, including the tools maps on the right. These use a large font size to suit slide decks. You can also print them out for your office wall. They show: [Linux observability tools][], [Linux static performance analysis tools][], [Linux benchmarking tools][], [Linux tuning tools][], and [Linux sar][]. Check the year on the image (bottom right) to see how recently I've updated it.

There is also a hi-res diagram combining observability, static performance tuning, and perf-tools/bcc: [png][], [svg][] (see [discussion][]), but it is not as complete as the other diagrams. For even more diagrams, see my slide decks below.

12-Jul-2019: I have a new book coming out: [BPF Performance Tools: Linux System and Application Observability][BPF Performance Tools_ Linux System and Application Observability]. This summarizes traditional Linux performance tools (iostat(1), perf(1), etc) as well as the new BPF tools.

## Tools ##

*  [perf\_events][perf_events]: perf one-liners, examples, visualizations.
 *  [eBPF tools][]: eBPF tracing tools and examples with bcc.
 *  [perf-tools][]: perf analysis tools using Ftrace (github).
 *  [bcc][]: perf analysis tools using eBPF (github).
 *  [ktap][]: one-liners, examples, and scripts.
 *  [Flame Graphs][]: using [perf][], [SystemTap][], and [ktap][ktap 1].

## Documentation ##

*  [Linux Performance Analysis in 60,000 Milliseconds][Linux Performance Analysis in 60_000 Milliseconds] shows the first ten commands to use in an investigation ([video][], [PDF][]). Written by myself and the performance engineering team at Netflix (2015).
 *  My post [Performance Tuning Linux Instances on EC2][] includes the tunables we're using at Netflix (2015).
 *  A post on [Linux Load Averages: Solving the Mystery][Linux Load Averages_ Solving the Mystery], explaining what they are and why they include the uninterruptible sleep state (2017).
 *  A [gdb Debugging Full Example (Tutorial)][gdb Debugging Full Example _Tutorial], including the use of some perf/debugging tools (2016).
 *  Generating flame graphs on Linux using perf & eBPF:
    
     *  [CPU Flame Graphs][]  
        [Off-CPU Flame Graphs][]  
        [Memory Flame Graphs][]
 *  Posts about eBPF, bcc, and bpftrace (2015-8):
    
     *  [Linux eBPF][] (2015)  
        [bcc: Taming Linux 4.3+ Tracing Superpowers][bcc_ Taming Linux 4.3_ Tracing Superpowers]  
        [tcpconnect and tcpaccept for Linux (bcc)][tcpconnect and tcpaccept for Linux _bcc]  
        [Linux eBPF Stack Trace Hack (bcc)][Linux eBPF Stack Trace Hack _bcc] (2016)  
        [Linux eBPF Off-CPU Flame Graph (bcc)][Linux eBPF Off-CPU Flame Graph _bcc]  
        [Linux Wakeup and Off-Wake Profiling (bcc)][Linux Wakeup and Off-Wake Profiling _bcc]  
        [Linux chain graph prototype (bcc)][Linux chain graph prototype _bcc]  
        [Linux eBPF/bcc uprobes][Linux eBPF_bcc uprobes]  
        [Linux BPF/bcc Road Ahead][Linux BPF_bcc Road Ahead]  
        [Ubuntu Xenial bcc/BPF][Ubuntu Xenial bcc_BPF]  
        [Linux bcc/BPF Tracing Security Capabilities][Linux bcc_BPF Tracing Security Capabilities]  
        [Linux MySQL Slow Query Tracing with bcc/BPF][Linux MySQL Slow Query Tracing with bcc_BPF]  
        [Linux bcc/BPF ext4 Latency Tracing][Linux bcc_BPF ext4 Latency Tracing]  
        [Linux bcc/BPF Run Queue (Scheduler) Latency][Linux bcc_BPF Run Queue _Scheduler_ Latency]  
        [Linux bcc/BPF Node.js USDT Tracing][Linux bcc_BPF Node.js USDT Tracing]  
        [Linux bcc tcptop][]  
        [Linux 4.9's Efficient BPF-based Profiler][Linux 4.9_s Efficient BPF-based Profiler]  
        [DTrace for Linux 2016][]  
        [Linux 4.x Tracing Tools: Using BPF Superpowers][Linux 4.x Tracing Tools_ Using BPF Superpowers]  
        [Linux bcc/BPF tcplife: TCP Lifespans][Linux bcc_BPF tcplife_ TCP Lifespans]  
        [Golang bcc/BPF Function Tracing][Golang bcc_BPF Function Tracing] (2017)  
        [7 BPF tools for performance analysis on Fedora][]  
        [TCP Tracepoints][] (2018)  
        [Linux bcc/eBPF tcpdrop][Linux bcc_eBPF tcpdrop]  
        [bpftrace (DTrace 2.0) for Linux 2018][bpftrace _DTrace 2.0_ for Linux 2018]  
          
          
         
 *  My [lwn.net][] article [Ftrace: The Hidden Light Switch][Ftrace_ The Hidden Light Switch] shows a use case for Linux ftrace (Aug, 2014).
 *  Posts about ftrace-based perf-tools (2014-5):
    
     *  [iosnoop for Linux][], [iosnoop Latency Heat Maps][], [opensnoop for Linux][], [execsnoop for Linux][], [ftrace: The Hidden Light Switch][ftrace_ The Hidden Light Switch], [tcpretrans][]  
        [Page Cache Hit Ratio][], [uprobe: User-Level Dynamic Tracing][uprobe_ User-Level Dynamic Tracing], [Hacking Linux USDT][]
 *  Posts about perf-based perf-tools: [perf Hacktogram][].
 *  Posts about perf\_events (2014-7):
    
     *  [perf CPU Sampling][], [perf Static Tracepoints][], [perf Heat Maps][], [perf Counting][], [perf Kernel Line Tracing][],  
        [perf Off-CPU Time Flame Graphs][], [Linux Profiling at Netflix][Linux Profiling at Netflix 1], [Java Mixed-Mode Flame Graphs][] ([PDF][PDF 1]), [Linux 4.5 perf folded format][],  
        [perf sched for Linux CPU scheduler analysis][]
 *  A page on [Working Set Size Estimation][] for Linux (2018+).
 *  A post on [KPTI/KAISER Meltdown Initial Performance Regressions][KPTI_KAISER Meltdown Initial Performance Regressions] (2018).
 *  In [The PMCs of EC2: Measuring IPC][The PMCs of EC2_ Measuring IPC] I showed the new Performance Monitoring Counter (PMC) support in the AWS EC2 cloud (2017).
 *  [CPU Utilization is Wrong][]: a post explaining the growing problem of memory stall cycles dominating the %CPU metric (2017).
 *  A post about [Linux 4.7 Hist Triggers][] (2016).
 *  The blog post [strace Wow Much Syscall][] discusses strace(1) for production use, and compares it to advanced tracing tools (2014).
 *  [USE Method: Linux Performance Checklist][USE Method_ Linux Performance Checklist]; also see the [USE Method][] page for the description of this methodology.
 *  [Off-CPU Analysis Method][], where I demonstrate this methodology on Linux.
 *  [Systems Performance: Enterprise and the Cloud][Systems Performance_ Enterprise and the Cloud] (Prentice Hall, 2013) uses Linux distributions as the primary example.

## Talks ##

In rough order of recommended viewing or difficulty, intro to more advanced:

### 1. Linux Systems Performance (USENIX LISA 2019) ###

This is my summary of Linux systems performance in 40 minutes, covering six facets: observability, methodologies, benchmarking, profiling, tracing, and tuning. It's intended for everyone as a tour of fundamentals, and some companies have indicated they will use it for new hire training.

A video of the talk is on [usenix.org][] and [youtube][], and the slides are on [slideshare][] or as a [PDF][PDF 2].

For a lot more information on observability tools, profiling, and tracing, see the talks that follow.

### 2. Linux Performance 2018 (PerconaLive 2018) ###

This was a 20 minute keynote summary of recent changes and features in Linux performance in 2018.

A video of the talk is on [youtube][youtube 1], and the slides are on [slideshare][slideshare 1] or as a [PDF.][]

[ ][PDF.]

### 3. Linux Performance Tools (Velocity 2015) ###

At Velocity 2015, I gave a 90 minute tutorial on Linux performance tools, summarizing performance observability, benchmarking, tuning, static performance tuning, and tracing tools. I also covered performance methodology, and included some live demos. This should be useful for everyone working on Linux systems. If you just saw my PerconaLive2016 talk, then some content should be familiar, but with many extras: I focus a lot more on the tools in this talk.

A video of the talk is on youtube ([playlist][]; [part 1][], [part 2][]) and the slides are on [slideshare][slideshare 2] or as a [PDF][PDF 3].

This was similar to my [SCaLE11x][] and [LinuxCon][] talks, however, with 90 minutes I was able to cover more tools and methodologies, making it the most complete tour of the topic I've done. I also posted about it on the [Netflix Tech Blog][].

### 4. How Netflix Tunes EC2 Instances for Performance (AWS re:Invent, 2017) ###

Instead of performance observability, this talk is about tuning. I begin by providing Netflix background, covering instance types and features in the AWS EC2 cloud, and then talk about Linux kernel tunables and observability.

A video of the talk is on [youtube][youtube 2] and the slides are on [slideshare][slideshare 3]:

### 5. Container Performance Analysis (DockerCon, 2017) ###

At DockerCon 2017 in Austin, I gave a talk on Linux container performance analysis, showing how to find bottlenecks in the host vs the container, how to profiler container apps, and dig deeper into the kernel.

A video of the talk is on [youtube][youtube 3] and the slides are on [slideshare][slideshare 4].

### 6. Broken Linux Performance Tools (SCaLE14x, 2016) ###

At the Southern California Linux Expo ([SCaLE 14x][]), I gave a talk on Broken Linux Performance Tools. This was a follow-on to my earlier Linux Performance Tools talk originally at SCaLE11x (and more recently at [Velocity][] as a tutorial). This broken tools talk was a tour of common problems with Linux system tools, metrics, statistics, visualizations, measurement overhead, and benchmarks. It also includes advice on how to cope (the green "What You Can Do" slides).

A video of the talk is on [youtube][youtube 4] and the slides are on [slideshare][slideshare 5] or as a [PDF][PDF 4].

### 7. Using Linux perf at Netflix (Kernel Recipes, 2017) ###

At [Kernel Recipes 2017][] I gave an updated talk on Linux perf at Netflix, focusing on getting CPU profiling and flame graphs to work. This talk includes a crash course on perf\_events, plus gotchas such as fixing stack traces and symbols when profiling Java, Node.js, VMs, and containers.

A video of the talk is on [youtube][youtube 5] and the slides are on [slideshare][slideshare 6]:

There's also an older version of this talk from 2015, which I've [posted][Linux Profiling at Netflix 1] about. To learn more about flame graphs, see my [flame graphs presentation][].

### 8. Give me 15 minutes and I'll change your view of Linux tracing (LISA, 2016) ###

I gave this demo at USENIX/LISA 2016, showing ftrace, perf, and bcc/BPF. A video is on [youtube][youtube 6] (sorry, the sound effects are a bit too loud):.

This was the first part of a longer talk on Linux 4.x Tracing Tools: Using BPF Superpowers. See the full [talk video][] and [talk slides][Linux 4.x Tracing Tools_ Using BPF Superpowers].

### 9. Performance analysis superpowers with Linux eBPF (O'Reilly Velocity, 2017) ###

This talk covers using enhanced BPF (aka eBPF) features added to the Linux 4.x series for performance analysis, observability, and debugging. The front-end used in this talk is bcc (BPF compiler collection), an open source project that provides BPF interfaces and a collection of tools.

A video of the talk is on [youtube][youtube 7], and the slides are on [slideshare][slideshare 7] or as a [PDF][PDF 5].

### 10. Linux Performance Analysis: New Tools and Old Secrets (ftrace) (LISA 2014) ###

At USENIX LISA 2014, I gave a talk on the new ftrace and perf\_events tools I've been developing: the [perf-tools][] collection on github, which mostly uses ftrace: a tracer that has been built into the Linux kernel for many years, but few have discovered (practically a secret).

A video of the talk is on [youtube][youtube 8], and the slides are on [slideshare][slideshare 8] or as a [PDF][PDF 6]. In a [post][] about this talk, I included some more screenshots of these tools in action.

### 11. Performance Checklists for SREs (SREcon, 2016) ###

At [SREcon 2016 Santa Clara][], I gave the closing talk on performance checklists for SREs (Site Reliability Engineers). The later half of this talk included Linux checklists for incident performance response. These may be useful whether you're analyzing Linux performance in a hurry or not.

A video of the talk is on [youtube][youtube 9] and [usenix][], and the slides are on [slideshare][slideshare 9] and as a [PDF][PDF 7]. I included the checklists in a [blog post][].

## Resources ##

Other resources (not by me) I'd recommend for the topic of Linux performance:

*  [Performance analysis & tuning of Red Hat Enterprise Linux - 2015 Red Hat Summit][Performance analysis _ tuning of Red Hat Enterprise Linux - 2015 Red Hat Summit] (video 2hrs): this is a great and in-depth tour of Linux performance tuning that should be largely applicable to all Linux distros.
 *  [Linux Instrumentation][]: slides from a great talk in June 2010 by Ian Munsie, which summarizes the different Linux tracers very well. If you're trying to understand all the tracers and frameworks, this is worth studying (keeping in mind it's from 2010).
 *  [Julia Evans blog][] has many posts about many topics, including performance tools.
 *  [Davidlohr Bueso's][Davidlohr Bueso_s] Linux performance posts.

[Linux Performance Profiling and _Monitoring]: http://www.ilinuxkernel.com/files/Linux.Performance.Profiling.and.Monitoring.April.2015.pdf
[Linux Performance Tools]: http://www.ilinuxkernel.com/files/Linux.Performance.Tools.Oct.2014.pdf
[Linux Performance _Analysis and Tools]: http://www.ilinuxkernel.com/files/Linux.Performance.Analysis.And.Tools.February.2013.pdf
[Linux Profiling at Netflix]: http://www.ilinuxkernel.com/files/Linux.Profiling.at.Netflix.Feb.2015.pdf
[http_www.brendangregg.com_linuxperf.html]: http://www.brendangregg.com/linuxperf.html
[Linux Performance Analysis New Tools and Old Secrets]: http://www.ilinuxkernel.com/files/LinuxPerfAnalysisNewTools.pdf
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191206202850128.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191206204116779.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191206204140385.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20191206204151206.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191206204231440.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20191206204244234.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20191206204259483.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20191206204311878.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/2019120620433283.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hhbjI1MjkzODYxNjE=,size_16,color_FFFFFF,t_70
[Linux observability tools]: http://www.brendangregg.com/Perf/linux_observability_tools.png
[Linux static performance analysis tools]: http://www.brendangregg.com/Perf/linux_static_tools.png
[Linux benchmarking tools]: http://www.brendangregg.com/Perf/linux_benchmarking_tools.png
[Linux tuning tools]: http://www.brendangregg.com/Perf/linux_tuning_tools.png
[Linux sar]: http://www.brendangregg.com/Perf/linux_observability_sar
[png]: http://www.brendangregg.com/Perf/linux_perf_tools_full.png
[svg]: http://www.brendangregg.com/Perf/linux_perf_tools_full.svg
[discussion]: https://www.reddit.com/r/linux/comments/4x4smu/linux_performance_tools_full_version_draft/
[BPF Performance Tools_ Linux System and Application Observability]: http://www.brendangregg.com/bpf-performance-tools-book.html
[perf_events]: http://www.brendangregg.com/perf.html
[eBPF tools]: http://www.brendangregg.com/ebpf.html
[perf-tools]: https://github.com/brendangregg/perf-tools
[bcc]: https://github.com/iovisor/bcc#tools
[ktap]: http://www.brendangregg.com/ktap.html
[Flame Graphs]: http://www.brendangregg.com/flamegraphs.html
[perf]: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#perf
[SystemTap]: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#SystemTap
[ktap 1]: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#ktap
[Linux Performance Analysis in 60_000 Milliseconds]: http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html
[video]: http://www.brendangregg.com/blog/2015-12-03/linux-perf-60s-video.html
[PDF]: http://www.brendangregg.com/Articles/Netflix_Linux_Perf_Analysis_60s.pdf
[Performance Tuning Linux Instances on EC2]: http://www.brendangregg.com/blog/2015-03-03/performance-tuning-linux-instances-on-ec2.html
[Linux Load Averages_ Solving the Mystery]: http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
[gdb Debugging Full Example _Tutorial]: http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html
[CPU Flame Graphs]: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#Linux
[Off-CPU Flame Graphs]: http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html#Linux
[Memory Flame Graphs]: http://www.brendangregg.com/FlameGraphs/memoryflamegraphs.html#Linux
[Linux eBPF]: http://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html
[bcc_ Taming Linux 4.3_ Tracing Superpowers]: http://www.brendangregg.com/blog/2015-09-22/bcc-linux-4.3-tracing.html
[tcpconnect and tcpaccept for Linux _bcc]: http://www.brendangregg.com/blog/2015-10-31/tcpconnect-tcpaccept-bcc.html
[Linux eBPF Stack Trace Hack _bcc]: http://www.brendangregg.com/blog/2016-01-18/ebpf-stack-trace-hack.html
[Linux eBPF Off-CPU Flame Graph _bcc]: http://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html
[Linux Wakeup and Off-Wake Profiling _bcc]: http://www.brendangregg.com/blog/2016-02-01/linux-wakeup-offwake-profiling.html
[Linux chain graph prototype _bcc]: http://www.brendangregg.com/blog/2016-02-05/ebpf-chaingraph-prototype.html
[Linux eBPF_bcc uprobes]: http://www.brendangregg.com/blog/2016-02-08/linux-ebpf-bcc-uprobes.html
[Linux BPF_bcc Road Ahead]: http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-road-ahead-2016.html
[Ubuntu Xenial bcc_BPF]: http://www.brendangregg.com/blog/2016-06-14/ubuntu-xenial-bcc-bpf.html
[Linux bcc_BPF Tracing Security Capabilities]: http://www.brendangregg.com/blog/2016-10-01/linux-bcc-security-capabilities.html
[Linux MySQL Slow Query Tracing with bcc_BPF]: http://www.brendangregg.com/blog/2016-10-04/linux-bcc-mysqld-qslower.html
[Linux bcc_BPF ext4 Latency Tracing]: http://www.brendangregg.com/blog/2016-10-06/linux-bcc-ext4dist-ext4slower.html
[Linux bcc_BPF Run Queue _Scheduler_ Latency]: http://www.brendangregg.com/blog/2016-10-08/linux-bcc-runqlat.html
[Linux bcc_BPF Node.js USDT Tracing]: http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html
[Linux bcc tcptop]: http://www.brendangregg.com/blog/2016-10-15/linux-bcc-tcptop.html
[Linux 4.9_s Efficient BPF-based Profiler]: http://www.brendangregg.com/blog/2016-10-21/linux-efficient-profiler.html
[DTrace for Linux 2016]: http://www.brendangregg.com/blog/2016-10-27/dtrace-for-linux-2016.html
[Linux 4.x Tracing Tools_ Using BPF Superpowers]: http://www.slideshare.net/brendangregg/linux-4x-tracing-tools-using-bpf-superpowers
[Linux bcc_BPF tcplife_ TCP Lifespans]: http://www.brendangregg.com/blog/2016-11-30/linux-bcc-tcplife.html
[Golang bcc_BPF Function Tracing]: http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-tracing.html
[7 BPF tools for performance analysis on Fedora]: https://opensource.com/article/17/11/bccbpf-performance
[TCP Tracepoints]: http://www.brendangregg.com/blog/2018-03-22/tcp-tracepoints.html
[Linux bcc_eBPF tcpdrop]: http://www.brendangregg.com/blog/2018-05-31/linux-tcpdrop.html
[bpftrace _DTrace 2.0_ for Linux 2018]: http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html
[lwn.net]: http://lwn.net/
[Ftrace_ The Hidden Light Switch]: http://lwn.net/Articles/608497/
[iosnoop for Linux]: http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html
[iosnoop Latency Heat Maps]: http://www.brendangregg.com/blog/2014-07-23/linux-iosnoop-latency-heat-maps.html
[opensnoop for Linux]: http://www.brendangregg.com/blog/2014-07-25/opensnoop-for-linux.html
[execsnoop for Linux]: http://www.brendangregg.com/blog/2014-07-28/execsnoop-for-linux.html
[ftrace_ The Hidden Light Switch]: http://www.brendangregg.com/blog/2014-08-30/ftrace-the-hidden-light-switch.html
[tcpretrans]: http://www.brendangregg.com/blog/2014-09-06/linux-ftrace-tcp-retransmit-tracing.html
[Page Cache Hit Ratio]: http://www.brendangregg.com/blog/2014-12-31/linux-page-cache-hit-ratio.html
[uprobe_ User-Level Dynamic Tracing]: http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html
[Hacking Linux USDT]: http://www.brendangregg.com/blog/2015-07-03/hacking-linux-usdt-ftrace.html
[perf Hacktogram]: http://www.brendangregg.com/blog/2014-07-10/perf-hacktogram.html
[perf CPU Sampling]: http://www.brendangregg.com/blog/2014-06-22/perf-cpu-sample.html
[perf Static Tracepoints]: http://www.brendangregg.com/blog/2014-06-29/perf-static-tracepoints.html
[perf Heat Maps]: http://www.brendangregg.com/blog/2014-07-01/perf-heat-maps.html
[perf Counting]: http://www.brendangregg.com/blog/2014-07-03/perf-counting.html
[perf Kernel Line Tracing]: http://www.brendangregg.com/blog/2014-09-11/perf-kernel-line-tracing.html
[perf Off-CPU Time Flame Graphs]: http://www.brendangregg.com/blog/2015-02-26/linux-perf-off-cpu-flame-graph.html
[Linux Profiling at Netflix 1]: http://www.brendangregg.com/blog/2015-02-27/linux-profiling-at-netflix.html
[Java Mixed-Mode Flame Graphs]: http://techblog.netflix.com/2015/07/java-in-flames.html
[PDF 1]: http://www.brendangregg.com/Articles/Netflix_Java_in_Flames.pdf
[Linux 4.5 perf folded format]: http://www.brendangregg.com/blog/2016-04-30/linux-perf-folded.html
[perf sched for Linux CPU scheduler analysis]: http://www.brendangregg.com/blog/2017-03-16/perf-sched.html
[Working Set Size Estimation]: http://www.brendangregg.com/wss.html
[KPTI_KAISER Meltdown Initial Performance Regressions]: http://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html
[The PMCs of EC2_ Measuring IPC]: http://www.brendangregg.com/blog/2017-05-04/the-pmcs-of-ec2.html
[CPU Utilization is Wrong]: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html
[Linux 4.7 Hist Triggers]: http://www.brendangregg.com/blog/2016-06-08/linux-hist-triggers.html
[strace Wow Much Syscall]: http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html
[USE Method_ Linux Performance Checklist]: http://www.brendangregg.com/USEmethod/use-linux.html
[USE Method]: http://www.brendangregg.com/usemethod.html
[Off-CPU Analysis Method]: http://www.brendangregg.com/offcpuanalysis.html
[Systems Performance_ Enterprise and the Cloud]: http://www.brendangregg.com/sysperfbook.html
[usenix.org]: https://www.usenix.org/conference/lisa19/presentation/gregg-linux
[youtube]: https://www.youtube.com/watch?v=fhBHvsi0Ql0&feature=emb_logo
[slideshare]: https://www.slideshare.net/brendangregg/lisa2019-linux-systems-performance
[PDF 2]: http://www.brendangregg.com/Slides/LISA2019_Linux_Systems_Performance.pdf
[youtube 1]: https://youtu.be/sV3XfrfjrPo?t=30m51s
[slideshare 1]: https://www.slideshare.net/brendangregg/linux-performance-2018-perconalive-keynote-95516934
[PDF.]: http://www.brendangregg.com/Slides/Percona2018_Linux_Performance.pdf
[playlist]: https://www.youtube.com/watch?v=FJW8nGV4jxY&list=PLhhdIMVi0o5RNrf8E2dUijvGpqKLB9TCR
[part 1]: https://www.youtube.com/watch?v=FJW8nGV4jxY
[part 2]: https://www.youtube.com/watch?v=zrr2nUln9Kk
[slideshare 2]: http://www.slideshare.net/brendangregg/velocity-2015-linux-perf-tools
[PDF 3]: http://www.brendangregg.com/Slides/Velocity2015_LinuxPerfTools.pdf
[SCaLE11x]: https://www.youtube.com/watch?v=0yyorhl6IjM
[LinuxCon]: http://www.brendangregg.com/blog/2014-08-23/linux-perf-tools-linuxcon-na-2014.html
[Netflix Tech Blog]: http://techblog.netflix.com/2015/08/netflix-at-velocity-2015-linux.html
[youtube 2]: https://www.youtube.com/watch?v=89fYOo1V2pA
[slideshare 3]: https://www.slideshare.net/brendangregg/how-netflix-tunes-ec2-instances-for-performance
[youtube 3]: https://www.youtube.com/watch?v=bK9A5ODIgac
[slideshare 4]: https://www.slideshare.net/brendangregg/container-performance-analysis
[SCaLE 14x]: http://www.socallinuxexpo.org/scale/14x
[Velocity]: http://www.brendangregg.com/linuxperf.html#Velocity2015
[youtube 4]: https://www.youtube.com/watch?v=OPio8V-z03c
[slideshare 5]: http://www.slideshare.net/brendangregg/broken-linux-performance-tools-2016
[PDF 4]: http://www.brendangregg.com/Slides/SCALE2016_Broken_Linux_Performance_Tools.pdf
[Kernel Recipes 2017]: https://kernel-recipes.org/en/2017/talks/perf-in-netflix/
[youtube 5]: https://www.youtube.com/watch?v=UVM3WX8Lq2k
[slideshare 6]: https://www.slideshare.net/brendangregg/kernel-recipes-2017-using-linux-perf-at-netflix
[flame graphs presentation]: http://www.brendangregg.com/flamegraphs.html#Presentation
[youtube 6]: https://www.youtube.com/watch?v=GsMs3n8CB6g
[talk video]: https://www.youtube.com/watch?v=UmOU3I36T2U
[youtube 7]: https://www.youtube.com/watch?v=bj3qdEDbCD4
[slideshare 7]: https://www.slideshare.net/brendangregg/velocity-2017-performance-analysis-superpowers-with-linux-ebpf
[PDF 5]: http://www.brendangregg.com/Slides/Velocity2017_BPF_superpowers.pdf
[youtube 8]: https://www.youtube.com/watch?v=R4IKeMQhM0Y
[slideshare 8]: http://www.slideshare.net/brendangregg/linux-performance-analysis-new-tools-and-old-secrets
[PDF 6]: http://www.brendangregg.com/Slides/LISA2014_LinuxPerfAnalysisNewTools.pdf
[post]: http://www.brendangregg.com/blog/2015-03-17/usenix-lisa-2014-linux-ftrace-perf-tools.markdown
[SREcon 2016 Santa Clara]: https://www.usenix.org/conference/srecon16/program
[youtube 9]: https://www.youtube.com/watch?v=zxCWXNigDpA
[usenix]: https://www.usenix.org/conference/srecon16/program/presentation/gregg
[slideshare 9]: http://www.slideshare.net/brendangregg/srecon-2016-performance-checklists-for-sres
[PDF 7]: http://www.brendangregg.com/Slides/SREcon_2016_perf_checklists.pdf
[blog post]: http://www.brendangregg.com/blog/2016-05-04/srecon2016-perf-checklists-for-sres.html
[Performance analysis _ tuning of Red Hat Enterprise Linux - 2015 Red Hat Summit]: https://www.youtube.com/watch?v=ckarvGJE8Qc
[Linux Instrumentation]: http://www.slideshare.net/DarkStarSword/instrumentation
[Julia Evans blog]: http://jvns.ca/
[Davidlohr Bueso_s]: http://blog.stgolabs.net/search/label/linux