Understanding IO

The following chapter takes a deeper look at understanding I/O behavior.

iobar [Tools CD] - Display I/O for Disk Devices Graphically

iobar displays two bar-graphs for each disk device. The read and write data rate are displayed in green in the left and right areas. The disk utilization is shown in the middle (red). At the bottom of the bars, input and output rates are displayed numerically, the value can be selected between last (green), average (white) and maximum (red) with the mouse middle button. The display mode can be toggled between logarithmic and linear with the left mouse button. In linear mode, scaling is automatic. All values are in bytes per second.

iobar – sample output

iotop [Tools CD] – Display iostat -x in a top-like Fashion

iotop is a binary that collects I/O statistics for disks, tapes, NFS-mounts, partitions (slices), SVM meta-devices and disk-paths. The display of those statistics can be filtered by device class or using regular expressions. Also the sorting order can be modified.

iotop – sample output

iostat – I/O Wizard

If you are looking at understanding I/O behaviour on a running system, your first stop will be the command iostat. iostat gives fast answers to question:

  • How much I/O in terms of input/output operations/second (IOPS) and throughput (MB/second)?

  • How busy are my I/O subsystems (latency and utilisation)?

In its simplest form, the command iostat -x <interval> (i.e. iostat -x 2) will examine all I/O channels and report statistics. See iostat -xc 2:

As can be seen from the screen capture, the iostat -x <interval> command will report device statistics every <interval> seconds. Every device is reported on a separate line and includes following information:

  • device: device name

  • r/s: device reads per second, i.e. read IOPS.

  • w/s: device writes per second, i.e. write IOPS.

  • kr/s: kilobytes read per second.

  • kw/s: kilobytes write per second.

  • wait: average number of transactions waiting for service (queue length)

  • actv: average number of transactions actively being serviced (removed from the queue but not yet completed) . This is the number of I/O operations accepted, but not yet serviced, by the device.

  • svc_t: average response time of transactions, in milliseconds . The svc_t output reports the overall response time, rather than the service time of a device. The overall time includes the time that transactions are in queue and the time that transactions are being serviced.

  • %w: percent of time there are transactions waiting for service (queue non-empty).

  • %b: percent of time the disk is busy (transactions in progress).

By adding the option -M to iostat, the report outputs megabytes instead of kilobytes.

iostat Usage Scenario – Sequential I/O

Let's study the output of iostat when doing sequential I/O on the system. For that, become super-user and in a terminal window, start the command:

  • dd if=/dev/rdsk/c1d0s0 of=/dev/null bs=128k &

Then start the iostat command with iostat -xM 10 and watch the output. After a minute stop the iostat and dd processes.

As can be seen from the screen capture above, the disk in the test system can sustain a read throughput of just over 25 MB/second, with an average service time below 5 milliseconds.

iostat Usage Scenario – Random I/O

Let's study the output of iostat when doing random I/O on the system. For that, start the command:

  • find / >/dev/null 2>&1 &

Then start the iostat command with iostat -xM 10 and watch the output. After a minute stop the iostat and find processes.

iostat – random I/O

As can be seen from the screen capture above, the same disk in the test system delivers just less than 1 MB/second on random I/O.

Properly sizing an I/O subsystem is not a trivial exercise. One has to take into considerations factors like:

  • Number of I/O operations per second (IOPS)

  • Throughput in Megabytes per second (MB/s)

  • Service times (in milliseconds)

  • I/O pattern (sequential or random)

  • Availability of caching

zpool iostat: iostat for zfs pools

ZFS comes with it's own version of iostat. It's build into the zpool command since the IO is a feature of the pool. The behavior is very similar to iostat. There are however less options:

zpool iostat [-T u | d ] [-v] [pool] ... [interval[count]]

The -T option allows to specify time formats. The version option (-v) shows the IO on a vdev device.

zpool iostat with a 10 second sample time

The verbose option is the option to go in more complex environments:

zpool iostat with verbose option and 10s sampletime

iosnoop [DtraceToolkit] – Print Disk I/O Events

iosnoop is a program that prints disk I/O events as they happen, with useful details such as UID, PID, filename, command, etc. iosnoop is measuring disk events that have made it past system caches.

Let's study the output of iosnoop when doing random I/O on the system. For that, start the command:

  • find / >/dev/null 2>&1 &

Then start the iosnoop command and watch the output. After a minute stop the iosnoop and find processes.

iosnoop – sample output

iopattern [DtraceToolkit] – Print Disk I/O Pattern

iopattern prints details on the I/O access pattern for disks, such as percentage of events that were of a random or sequential nature. By default, totals for all disks are printed.

Let's study the output of iopattern when doing random I/O on the system. For that, start the command:

  • find / >/dev/null 2>&1 &

Then start the iopattern command and watch the output. After a minute stop the iopattern and find processes.

iopattern – sample output

iotop [DtraceToolkit] – Display top Disk I/O Events by Process

iotop prints details on the top I/O events by process.

Let's study the output of iotop when doing random I/O on the system. For that, start the command:

  • find / >/dev/null 2>&1 &

Then start the iotop command and watch the output. After a minute stop the iotop and find processes.

fsstat [Solaris 10+] – Report File System Statistics

fsstat reports kernel file operation activity by the file system type or by the path name, which is converted to a mount point. Please see the man page fsstat(1) for details on all options.

Let's study the output of fsstat when doing random I/O on the system. For that, start the command:

find / >/dev/null 2>&1 &

Then start the fsstat / 1 command and watch the output. After a minute stop the fsstat and find processes.

fsstat – sample output

Thomas Bastian was a coauthor of an earlier version of this document. The earlier version of this page has been published in the "The Developers Edge" in 2009.