ZFS fsync IOPS performance in FreeBSD

桃扇骨 2022-02-01 19:31 349阅读 0赞

我在上一篇文章讲了一下ZFS的性能优化.

文章提到在Linux (CentOS 6.5 x64)中, ZFS的fsync调用性能不佳的问题, 完全不如ext4, 于是在同一台主机, 我安装了FreeBSD 10 x64. 使用同样的硬件测试一下fsync的性能.

PostgreSQL的安装参考

http://blog.163.com/digoal@126/blog/static/163877040201451181344545/

http://blog.163.com/digoal@126/blog/static/163877040201451282121414/

首先查看块设备, 这里使用12块4 TB的SATA盘. 

  1. # gpart list -a
  2. Geom name: mfid[1-12]

创建zpool, 

  1. # zpool create zp1 mfid1 mfid2 mfid3 mfid4 mfid5 mfid6 mfid7 mfid8 mfid9 mfid10 mfid11 mfid12
  2. # zpool get all zp1
  3. NAME  PROPERTY                       VALUE                          SOURCE
  4. zp1   size                           43.5T                          -
  5. zp1   capacity                       0%                             -
  6. zp1   altroot                        -                              default
  7. zp1   health                         ONLINE                         -
  8. zp1   guid                           8490038421326880416            default
  9. zp1   version                        -                              default
  10. zp1   bootfs                         -                              default
  11. zp1   delegation                     on                             default
  12. zp1   autoreplace                    off                            default
  13. zp1   cachefile                      -                              default
  14. zp1   failmode                       wait                           default
  15. zp1   listsnapshots                  off                            default
  16. zp1   autoexpand                     off                            default
  17. zp1   dedupditto                     0                              default
  18. zp1   dedupratio                     1.00x                          -
  19. zp1   free                           43.5T                          -
  20. zp1   allocated                      285K                           -
  21. zp1   readonly                       off                            -
  22. zp1   comment                        -                              default
  23. zp1   expandsize                     0                              -
  24. zp1   freeing                        0                              default
  25. zp1   feature@async_destroy          enabled                        local
  26. zp1   feature@empty_bpobj            active                         local
  27. zp1   feature@lz4_compress           enabled                        local
  28. zp1   feature@multi_vdev_crash_dump  enabled                        local

创建zfs 

  1. # zfs create -o mountpoint=/data01 -o atime=off zp1/data01
  2. # zfs get all zp1/data01
  3. NAME        PROPERTY              VALUE                  SOURCE
  4. zp1/data01  type                  filesystem             -
  5. zp1/data01  creation              Thu Jun 26 23:52 2014  -
  6. zp1/data01  used                  32K                    -
  7. zp1/data01  available             42.8T                  -
  8. zp1/data01  referenced            32K                    -
  9. zp1/data01  compressratio         1.00x                  -
  10. zp1/data01  mounted               yes                    -
  11. zp1/data01  quota                 none                   default
  12. zp1/data01  reservation           none                   default
  13. zp1/data01  recordsize            128K                   default
  14. zp1/data01  mountpoint            /data01                local
  15. zp1/data01  sharenfs              off                    default
  16. zp1/data01  checksum              on                     default
  17. zp1/data01  compression           off                    default
  18. zp1/data01  atime                 off                    local
  19. zp1/data01  devices               on                     default
  20. zp1/data01  exec                  on                     default
  21. zp1/data01  setuid                on                     default
  22. zp1/data01  readonly              off                    default
  23. zp1/data01  jailed                off                    default
  24. zp1/data01  snapdir               hidden                 default
  25. zp1/data01  aclmode               discard                default
  26. zp1/data01  aclinherit            restricted             default
  27. zp1/data01  canmount              on                     default
  28. zp1/data01  xattr                 off                    temporary
  29. zp1/data01  copies                1                      default
  30. zp1/data01  version               5                      -
  31. zp1/data01  utf8only              off                    -
  32. zp1/data01  normalization         none                   -
  33. zp1/data01  casesensitivity       sensitive              -
  34. zp1/data01  vscan                 off                    default
  35. zp1/data01  nbmand                off                    default
  36. zp1/data01  sharesmb              off                    default
  37. zp1/data01  refquota              none                   default
  38. zp1/data01  refreservation        none                   default
  39. zp1/data01  primarycache          all                    default
  40. zp1/data01  secondarycache        all                    default
  41. zp1/data01  usedbysnapshots       0                      -
  42. zp1/data01  usedbydataset         32K                    -
  43. zp1/data01  usedbychildren        0                      -
  44. zp1/data01  usedbyrefreservation  0                      -
  45. zp1/data01  logbias               latency                default
  46. zp1/data01  dedup                 off                    default
  47. zp1/data01  mlslabel                                     -
  48. zp1/data01  sync                  disabled               local
  49. zp1/data01  refcompressratio      1.00x                  -
  50. zp1/data01  written               32K                    -
  51. zp1/data01  logicalused           16K                    -
  52. zp1/data01  logicalreferenced     16K                    -

测试fsync, 相比Linux有很大的提升, 基本达到了块设备的瓶颈. 

  1. # /opt/pgsql9.3.4/bin/pg_test_fsync -f /data01/1
  2. 5 seconds per test
  3. O_DIRECT supported on this platform for open_datasync and open_sync.
  5. Compare file sync methods using one 8kB write:
  6. (in wal_sync_method preference order, except fdatasync
  7. is Linux's default)
  8.         open_datasync                                 n/a
  9.         fdatasync                                     n/a
  10.         fsync                            6676.001 ops/sec     150 usecs/op
  11.         fsync_writethrough                            n/a
  12.         open_sync                        6087.783 ops/sec     164 usecs/op
  14. Compare file sync methods using two 8kB writes:
  15. (in wal_sync_method preference order, except fdatasync
  16. is Linux's default)
  17.         open_datasync                                 n/a
  18.         fdatasync                                     n/a
  19.         fsync                            4750.841 ops/sec     210 usecs/op
  20.         fsync_writethrough                            n/a
  21.         open_sync                        3065.099 ops/sec     326 usecs/op
  23. Compare open_sync with different write sizes:
  24. (This is designed to compare the cost of writing 16kB
  25. in different write open_sync sizes.)
  26.          1 * 16kB open_sync write        4965.249 ops/sec     201 usecs/op
  27.          2 *  8kB open_sync writes       3039.074 ops/sec     329 usecs/op
  28.          4 *  4kB open_sync writes       1598.735 ops/sec     625 usecs/op
  29.          8 *  2kB open_sync writes       1326.517 ops/sec     754 usecs/op
  30.         16 *  1kB open_sync writes        620.992 ops/sec    1610 usecs/op
  32. Test if fsync on non-write file descriptor is honored:
  33. (If the times are similar, fsync() can sync data written
  34. on a different descriptor.)
  35.         write, fsync, close              5422.742 ops/sec     184 usecs/op
  36.         write, close, fsync              5552.278 ops/sec     180 usecs/op
  38. Non-Sync'ed 8kB writes:
  39.         write                           67460.621 ops/sec      15 usecs/op
  41. # zpool iostat -v 1
  42.                capacity     operations    bandwidth
  43. pool        alloc   free   read  write   read  write
  44. ----------  -----  -----  -----  -----  -----  -----
  45. zp1          747M  43.5T      0  7.31K      0  39.1M
  46.   mfid1     62.8M  3.62T      0    638      0  3.27M
  47.   mfid2     61.9M  3.62T      0    615      0  3.23M
  48.   mfid3     62.8M  3.62T      0    615      0  3.23M
  49.   mfid4     62.0M  3.62T      0    615      0  3.23M
  50.   mfid5     62.9M  3.62T      0    616      0  3.24M
  51.   mfid6     62.0M  3.62T      0    616      0  3.24M
  52.   mfid7     62.9M  3.62T      0    620      0  3.24M
  53.   mfid8     61.6M  3.62T      0    620      0  3.24M
  54.   mfid9     62.2M  3.62T      0    619      0  3.23M
  55.   mfid10    61.8M  3.62T      0    615      0  3.23M
  56.   mfid11    62.2M  3.62T      0    648      0  3.41M
  57.   mfid12    62.1M  3.62T      0    650      0  3.29M
  58. ----------  -----  -----  -----  -----  -----  -----
  59. zroot       2.69G   273G      0      0      0      0
  60.   mfid0p3   2.69G   273G      0      0      0      0
  61. ----------  -----  -----  -----  -----  -----  -----
  63. # iostat -x 1
  64.                         extended device statistics  
  65. device     r/s   w/s    kr/s    kw/s qlen svc_t  %b  
  66. mfid0      0.0   0.0     0.0     0.0    0   0.0   0 
  67. mfid1      0.0 416.6     0.0  7468.5    0   0.1   2 
  68. mfid2      0.0 416.6     0.0  7468.5    0   0.0   2 
  69. mfid3      0.0 429.6     0.0  7480.0    0   0.1   2 
  70. mfid4      0.0 433.6     0.0  7484.0    0   0.1   3 
  71. mfid5      0.0 433.6     0.0  7495.9    0   0.1   2 
  72. mfid6      0.0 421.6     0.0  7484.5    0   0.1   3 
  73. mfid7      0.0 417.6     0.0  7488.5    0   0.1   3 
  74. mfid8      0.0 438.6     0.0  7638.3    0   0.1   2 
  75. mfid9      0.0 437.6     0.0  7510.4    0   0.1   2 
  76. mfid10     0.0 428.6     0.0  7494.4    0   0.1   4 
  77. mfid11     0.0 416.6     0.0  7468.5    0   0.1   2 
  78. mfid12     0.0 416.6     0.0  7468.5    0   0.1   2

disable sync的情形, FreeBSD和Linux下差不多. 

  1. # zfs set sync=disabled zp1/data01
  2. # /opt/pgsql9.3.4/bin/pg_test_fsync -f /data01/1
  3. 5 seconds per test
  4. O_DIRECT supported on this platform for open_datasync and open_sync.
  6. Compare file sync methods using one 8kB write:
  7. (in wal_sync_method preference order, except fdatasync
  8. is Linux's default)
  9.         open_datasync                                 n/a
  10.         fdatasync                                     n/a
  11.         fsync                           115687.300 ops/sec       9 usecs/op
  12.         fsync_writethrough                            n/a
  13.         open_sync                       126789.698 ops/sec       8 usecs/op
  15. Compare file sync methods using two 8kB writes:
  16. (in wal_sync_method preference order, except fdatasync
  17. is Linux's default)
  18.         open_datasync                                 n/a
  19.         fdatasync                                     n/a
  20.         fsync                           65027.801 ops/sec      15 usecs/op
  21.         fsync_writethrough                            n/a
  22.         open_sync                       60239.232 ops/sec      17 usecs/op
  24. Compare open_sync with different write sizes:
  25. (This is designed to compare the cost of writing 16kB
  26. in different write open_sync sizes.)
  27.          1 * 16kB open_sync write       115246.114 ops/sec       9 usecs/op
  28.          2 *  8kB open_sync writes      63999.355 ops/sec      16 usecs/op
  29.          4 *  4kB open_sync writes      33661.426 ops/sec      30 usecs/op
  30.          8 *  2kB open_sync writes      18960.527 ops/sec      53 usecs/op
  31.         16 *  1kB open_sync writes       8251.087 ops/sec     121 usecs/op
  33. Test if fsync on non-write file descriptor is honored:
  34. (If the times are similar, fsync() can sync data written
  35. on a different descriptor.)
  36.         write, fsync, close             47380.701 ops/sec      21 usecs/op
  37.         write, close, fsync             50214.128 ops/sec      20 usecs/op
  39. Non-Sync'ed 8kB writes:
  40.         write                           78263.057 ops/sec      13 usecs/op

[参考]

  1. http://blog.163.com/digoal@126/blog/static/1638770402014526992910/

  2. http://blog.163.com/digoal@126/blog/static/163877040201451181344545/

  3. http://blog.163.com/digoal@126/blog/static/163877040201451282121414/

发表评论

表情：
评论列表 （有 0 条评论，349人围观）

还没有评论，来说两句吧...

相关阅读

    相关 fio测试IOPS

        fio是系统IO测试的工具，覆盖多种不同类型、不同方式的IO测试，并且简单易用。fio在压力测试中，常用于了解不同文件操作的IOPS极限，可以更加全面的了解系统IO处理

    清疚清疚/ 0 赞/ 135 阅读