Saturday, May 21, 2011

Storage Speed on the PandaBoard

In my experience so far with the PandaBoard, I noticed that the system feels sluggish using the SD card.  I'm using a Class 6 SD card which means it can sustain 6 MB/sec writes.  The USB 2.0 bus is capable of a max 480 Mbps, or 60 MB/sec; in practice, USB tops out at around 30 MB/sec due to CPU overhead.  Still, that's quite a bit faster than the 6 MB/sec the SD card can do.  With that in mind, I wanted to compare the performance of the SD card against a USB 2.0 hard drive.

The bonnie++ tool is great for testing storage performance, so I installed it with a quick yum install bonnie++ and ran it against both the SD card and the hard drive.  I was using the 2.6.35-g6d019da-dirty kernel.

It's generally a good idea to store the output of bonnie++ on a different drive from the one being tested so as not to interfere with the results, so first create a RAM-disk:

[root@fedora-arm ~]# mkdir /bonnie
[root@fedora-arm ~]# mount -t tmpfs none /bonnie

The SD card holds the root filesystem, so I tested it first; note that you have to run bonnie++ as a non-root user.

[jbastian@fedora-arm ~]$ bonnie++ -d /tmp -n 0 -m "SD Card" -q 1>>/bonnie/bonnie.csv 2>/dev/null

It takes a while for bonnie++ to do its thing, so I fired it up and grabbed a bucket of popcorn and a movie.  When it finished, I ran it again on the USB hard drive: just change /tmp to /mnt/usb.  The output is just appended to the bonnie.csv file.

[jbastian@fedora-arm ~]$ bonnie++ -d /mnt/usb -n 0 -m "USB HD" -q 1>>/bonnie/bonnie.csv 2>/dev/null

The output is CSV, but bonnie++ includes a tool to convert it to HTML to make it more readable:

[jbastian@fedora-arm ~]$ bon_csv2html /bonnie/bonnie.csv > bonnie.html

I used tidy -i to reformat the HTML a bit so I could remove the columns for the file creation tests (which I did not run), and the results look like:

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
USB HD2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms

As I expected, the SD card clearly wins with random seek times: 663ms vs 7643ms.  This is one of the major advantages of flash storage over traditional spinning magnetic media since flash does not have to move a head over the disk and then wait for the desired sector to spin under the head.  It's interesting, though, that even with the significantly slower seek times, the hard drive managed to squeeze out 93.9 seeks/sec compared to only 27.4 seeks/sec on the SD card.

The throughput results are more interesting. On sequential output, the hard drive is the winner with 9017 K/sec compared to the SD card's 3125 K/sec.  Writing data is one of the weaknesses of flash storage since it has to re-write an entire block if you need to flip a bit from a 0 to 1. For sequential input, the SD card managed a respectable 17598 K/sec vs the hard drive's 12357 K/sec.

Overall, the SD card performs nicely on large sequential reads, but the USB hard drive seems to outperform it in general usage.  Unfortunately, neither drive came close to the 30 MB/sec I was hoping for.

Just for fun, I also ran zcav, another tool provided by bonnie++ to test the read rates over a hard drive.  On a normal hard drive, the read rates will be faster on the outer edge of the disk since the outer tracks have a longer circumference and thus have more sectors compared to the inner tracks, thus more sectors pass under the head in one revolution of the disk on the outer tracks.

On an SD card, the rates should be constant since there is no spinning disk.

The zcav needs access to the raw device, so I'll run it as root.  Again, store the zcav output in the RAM disk so it doesn't interfere with the testing.  I only tested the first 8 GB of the hard drive since the SD card is only 8 GB in size, and also because testing all 200 GB would take all day!

[root@fedora-arm ~]# zcav -l /bonnie/zcav-sdcard.out /dev/mmcblk0
[root@fedora-arm ~]# zcav -r 8192 -l /bonnie/zcav-usbhd.out -f /dev/sda

The output is just a series of numbers in three columns:
#block offset (GiB), MiB/s, time
0.00 14.59 17.547
0.25 16.10 15.897
0.50 15.09 16.966
...

I think this will be easier to make sense of as a graph, so gnuplot to the rescue:

[jbastian@fedora-arm ~]$ gnuplot
...
gnuplot> set terminal png
Terminal type set to 'png'
Options are 'nocrop font /usr/share/fonts/dejavu/DejaVuSans.ttf 12 size 640,480 '
gnuplot> set output "zcav_graph.png"
gnuplot> set title "ZCAV"
gnuplot> set xlabel "Block Offset (GiB)"
gnuplot> set ylabel "MiB/sec"
gnuplot> set yrange [0:20]
gnuplot> plot "zcav-sdcard.out" using 1:2 title 'SD Card' with lines, \
>             "zcav-usbhd.out" using 1:2 title 'USB HD' with lines
gnuplot> quit

And the graph:

The SD card seems to be reading at about 15 MiB/sec and the USB hard drive at about 10 MiB/sec.  This is a little below the first bonnie++ run showing 17 Mib/sec and 12 MiB/sec respectively, but the SD card is still consistently faster than the USB hard drive at sequential reads.

For comparison, I ran bonnie++ on a desktop system (Core 2 Quad @ 2.66 GHz, 6 GiB RAM, 320 GiB SATA hard drive) running Red Hat Enterprise Linux 6.1 and it had a nice 58 MiB/sec for sequential writes, 72 MiB/sec for sequential reads, and a 1066ms latency for random seeks.  Here's the full set of numbers compared to the PandaBoard:

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
USB HD2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms
Desktop SATA11G578975885111256554263694723244176.91
Latency13998us3126ms411ms25294us862ms1066ms

My conclusion: I will continue using the SD card to hold the root filesystem.  I was hoping to get a performance boost out of a hard drive, but I think the USB bus is the bottleneck for the hard drive.  Hopefully the PandaBoard 2.0 (or whatever it will be called) will have a SATA port on it!

3 comments:

  1. Great analysis, thanks. I've looked at this too since I also find the PandaBoard unbearably sluggish on SD (I use a USB MicroDrive as my rootfs and just use SD for /boot).

    What confuses me about these numbers is that the seek latency for your USB HDD is much higher (but 7.6sec.. for what?) than for the SD card, yet the USB HD can do almost 100 of them per second, and the SD card only 27?

    I wonder what's behind these numbers. I have a strong feeling that the slowness of the PandaBoard SD can't be fully expressed in benchmark numbers (other than the number of hours an apt-get dist-upgrade takes.. :-/). I wish they'd fix it. I bought a 32GB SD card for this thing, but what's the point..

    ReplyDelete
  2. I think the long latency times are due to a bug with the smsc95xx driver, or possibly with the hardware itself. If you look at the numbers from my newer post ("Storage Speed on the PandaBoard Revisited"), you'll see the seek time drops to 1.5 seconds when I'm running 'sudo ping -i 0.001 pandaboard' from another system. Also, the seeks/second increases to 162. One theory that I read was that the chip may be queueing up requests and dealing with them in one burst, possibly to save power.

    ReplyDelete
  3. I saw that post yeah, pretty interesting. I wonder if it also improves SD storage speed, did you notice anything?

    I'd do it myself but the USB HDD of my Pandaboard seems to have died yesterday so I have to find a new storage solution. :-(

    Some reading yesterday did remind me that I have to fix partition alignment on the SDs or the speed will just be horrible.

    ReplyDelete