Mdthbs

This is a very popular question (you can see variations of it on https://stackoverflow.com/q/1198691 , https://serverfault.com/q/219739/203726 and https://askubuntu.com/q/87035/740413 ).

Are there better methods [than dd] to [benchmark disks]?

Yes but they will take longer to run and require knowledge of how to interpret the results - there's no single number that will tell you everything in one go because the following influence the type of test you will run:

• Are you interested in the performance of I/O that is random, sequential or some mix of the two?


• Are you reading from or writing to the disk (or some mixture of the two)?


• Are you concerned about latency, throughput or both?


• Are you trying to understand how different parts of the same hard disk perform (generally speeds a faster closer to the centre of the disk)?


• Are you interested in raw numbers or how a given filesystem will perform when using your disk?


• Are you interested in how a particular size of I/O performs?


• Are you submitting the I/O synchronously or asynchronously?


• How much I/O are you submitting (submit too little the wrong way and all the I/O can be cached so you wind up testing the speed of your RAM rather than the speed of the disk)?


• How compressible is the content of the data you are writing (e.g. zero only data is highly compressible and some filesystems/disks even have a special fast-path for zero only data leading to numbers that are unobtainable with other content)?

And so on.

Assuming you're interested in raw, non-filesystem benchmarks here's a short list of tools with easiest to run at the top and difficult/better/more thorough nearer the bottom:

1. dd (sequential, only shows throughput but configured correctly it can be made to bypass the block cache/wait for I/O to be really completed)


2. hdparm (sequential, only tests reads, only shows throughput, only works with ATA devices, doesn't account for filesystem overhead but configured correctly it can be made to bypass the block cache)


3. GNOME Disk Utility's benchmark (easy to run, graphical but requires a full GNOME install, gives latency and throughput numbers for different types of I/O).


4. 
   fio (can do nearly anything and gives detailed results but requires configuration and an understanding of how to interpret said results). Here's what Linus says about it:
   
   

   
   

   Greg - get Jens' FIO code. It does things right, including writing actual pseudo-random contents, which shows if the disk does some "de-duplication" (aka "optimize for benchmarks):

   
   
   

   [ https://github.com/axboe/fio/ ]

   
   
   

   Anything else is suspect - forget about bonnie or other traditional tools.

   
   

   
   

Source: comment left on Google Plus to Greg Kroah-Hartman by Linus Torvalds.

with the IOPS tool

If you can't be bothered to read all this I'd just recommend the IOPS tool. It will tell you real-world speed depending on block size.

Otherwise - when doing an IO benchmark I would look at the following things:

• blocksize/cache/IOPS/direct vs buffered/async vs sync


• read/write


• threads


• latency



• CPU utilization



• 
  

  Which blocksize will you use: If you want to read/write 1 GB from/to disk this will be quick if you do one I/O operation. But if your application needs to write in 512 byte chunks all over the harddisk in non-sequential pieces (called random I/O although it is not random) this will look differently. Now, databases will do random I/O for the data volume and sequential I/O for the log volume due to their nature. So, first you need to become clear what you want to measure. If you want to copy large video files that's different than if you want to install Linux.

  
  
  

  This blocksize is effecting the count of I/O operations you do. If you do e.g. 8 sequential read (or write, just not mixed) operations the I/O scheduler of the OS will merge them. If it does not, the controller's cache will do the merge. There is practically no difference if you read 8 sequential blocks of 512 bytes or one 4096 bytes chunk. One exception - if you manage to do direct sync IO and wait for the 512 bytes before you request the next 512 bytes. In this case, increasing the block size is like adding cache.

  
  
  

  Also you should be aware that there is sync and async IO: With sync IO you will not issue the next IO request before the current one returns. With async IO you can request e.g. 10 chunks of data and then wait as they arrive. Disctinct database threads will typically use sync IO for log and async IO for data. The IOPS tool takes care of that by measuring all relevant block sizes starting from 512 bytes.

  
  


• 
  

  Will you read or write: Usually reading is faster than writing. But note that caching works quite a different way for reads and writes:

  
  
  
  
  

  • For writes, the data will be handed over to the controller and if it caches, it will acknowledge before the data is on disk unless the cache is full. Using the tool iozone you can draw beautiful graphs of plateaus of cache effects (CPU cache effect and buffer cache effect). The caches becomes less efficient the more has been written.

  
  

  • For reads, read data is held in cache after the first read. The first reads take longest and caching becomes more and more effective during uptime. Noteable caches are the CPU cache, the OS' file system cache, the IO controller's cache and the storage's cache. The IOPS tool only measures reads. This allows it to "read all over the place" and you do not want it to write instead of read.

  
  
  
  



• How many threads will you use: If you use one thread (using dd for disk benchmarks) you will probably get a much worse performance than with several threads. The IOPS tool takes this into account and reads on several threads.




• How important is latency for you: Looking at databases, IO latency becomes enormously important. Any insert/update/delete SQL command will be written into the database journal ("log" in database lingo) on commit before it is acknowledged. This means the complete database may be waiting for this IO operation to be completed. I show here how to measure the average wait time (await) using the iostat tool.




• How important is CPU utilization for you: Your CPU may easily become the bottleneck for your application's performance. In this case you must know how much CPU cycles get burned per byte read/written and optimize into that direction. This can mean to decide for/against PCIe flash memory depending on your measurement results. Again the iostat tool can give you a rough estimation on CPU utilization by your IO operations.

If you have installed PostgreSQL, you can use their excellent pg_test_fsync benchmark. It basically test your write sync performance.

On Ubuntu you find it here: /usr/lib/postgresql/9.5/bin/pg_test_fsync

The great thing about it, is that this tool will show you why enterprise SSD's are worth the extra $.

You can use fio - the Multithreaded IO generation tool. It is packaged by several distributions, e.g. Fedora 25, Debian and OpenCSW.

The fio tool is very flexible, it can be easily used to benchmark various IO
scenarios - including concurrent ones. The package comes with some example
configuration files (cf. e.g. /usr/share/doc/fio/examples). It properly measures things, i.e. it also prints the
standard deviation and quantitative statistics for some figures. Things some
other popular benchmarking tools don't care about.

A simple example (a sequence of simple scenarios: sequential/random X read/write):

$ cat fio.cfg

[global]

size=1g

filename=/dev/sdz



[randwrite]

rw=randwrite



[randread]

wait_for=randwrite

rw=randread

size=256m



[seqread]

wait_for=randread

rw=read



[seqwrite]

wait_for=seqread

rw=write

The call:

# fio -o fio-seagate-usb-xyz.log fio.cfg

$ cat fio-seagate-usb-xyz.log

[..]

randwrite: (groupid=0, jobs=1): err= 0: pid=11858: Sun Apr  2 21:23:30 2017

  write: io=1024.0MB, bw=16499KB/s, iops=4124, runt= 63552msec

clat (usec): min=1, max=148280, avg=240.21, stdev=2216.91

 lat (usec): min=1, max=148280, avg=240.49, stdev=2216.91

clat percentiles (usec):

 |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    7],

 | 30.00th=[   10], 40.00th=[   11], 50.00th=[   11], 60.00th=[   12],

 | 70.00th=[   14], 80.00th=[   16], 90.00th=[   19], 95.00th=[   25],

 | 99.00th=[ 9408], 99.50th=[10432], 99.90th=[21888], 99.95th=[38144],

 | 99.99th=[92672]

bw (KB  /s): min= 7143, max=371874, per=45.77%, avg=15104.53, stdev=32105.17

lat (usec) : 2=0.20%, 4=15.36%, 10=6.58%, 20=69.35%, 50=6.07%

lat (usec) : 100=0.49%, 250=0.07%, 500=0.01%, 750=0.01%

lat (msec) : 4=0.01%, 10=1.20%, 20=0.54%, 50=0.08%, 100=0.03%

lat (msec) : 250=0.01%

  cpu          : usr=1.04%, sys=4.79%, ctx=4977, majf=0, minf=11

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

 issued    : total=r=0/w=262144/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0

 latency   : target=0, window=0, percentile=100.00%, depth=1

randread: (groupid=0, jobs=1): err= 0: pid=11876: Sun Apr  2 21:23:30 2017

  read : io=262144KB, bw=797863B/s, iops=194, runt=336443msec

[..]

bw (KB  /s): min=  312, max= 4513, per=15.19%, avg=591.51, stdev=222.35

[..]

Note that the [global] section has global defaults that can be overriden by
other sections. Each section describes a job, the section name is the job name
and can be freely choosen. By default, different jobs are started in parallel,
thus the above example explicitly serializes the job execution with the
wait_for key. Also, fio uses a block size of 4 KiB - which can be changed, as
well. The example directly uses the raw device for read and write jobs, thus,
make sure that you use the right device. The tool also supports using a
file/directory on existing filesystems.

Other Tools

The hdparm utility provides a very simple read benchmark, e.g.:

# hdparm -t -T /dev/sdz

It's not a replacement for a state-of-the-art benchmarking tool like fio, it
just should be used for a first plausibility check. For example, to check if
the external USB 3 drive is wrongly recognized as USB 2 device (you would see ~
100 MiB/s vs. ~ 30 MiB/s rates then).

As pointed out here here, you can use gnome-disks (if you use Gnome).

Click to the the drive that you want to test and the click on "Additional partition options" (the wheels). Then Benchmark Partition. You'll get average read/write in MB/s and average access times in milliseconds. I found that very comfortable.

It's a little crude, but this works in a pinch:

find <path> -type f -print0 | cpio -0o >/dev/null

You can do some interesting things with this technique, including caching all the /lib and /usr/bin files. You can also use this as part of a benchmarking effort:

find / -xdev -type f -print0 | 

sort -R --from0-file=- | 

timeout "5m" cpio -0o >/dev/null

All filenames on root are found, sorted randomly, and copy them into cache for up to 1 minute. The output from cpio tells you how many blocks were copied. Repeat 3 times to get an average of blocks-per-minute. (Note, the find/sort operation may take a long time -- much longer than the copy. It would be better to cache the find / sort and use split to get a sample of files.)