Saturday, May 21, 2011

Storage Speed on the PandaBoard

In my experience so far with the PandaBoard, I noticed that the system feels sluggish using the SD card.  I'm using a Class 6 SD card which means it can sustain 6 MB/sec writes.  The USB 2.0 bus is capable of a max 480 Mbps, or 60 MB/sec; in practice, USB tops out at around 30 MB/sec due to CPU overhead.  Still, that's quite a bit faster than the 6 MB/sec the SD card can do.  With that in mind, I wanted to compare the performance of the SD card against a USB 2.0 hard drive.

The bonnie++ tool is great for testing storage performance, so I installed it with a quick yum install bonnie++ and ran it against both the SD card and the hard drive.  I was using the 2.6.35-g6d019da-dirty kernel.

It's generally a good idea to store the output of bonnie++ on a different drive from the one being tested so as not to interfere with the results, so first create a RAM-disk:

[root@fedora-arm ~]# mkdir /bonnie
[root@fedora-arm ~]# mount -t tmpfs none /bonnie

The SD card holds the root filesystem, so I tested it first; note that you have to run bonnie++ as a non-root user.

[jbastian@fedora-arm ~]$ bonnie++ -d /tmp -n 0 -m "SD Card" -q 1>>/bonnie/bonnie.csv 2>/dev/null

It takes a while for bonnie++ to do its thing, so I fired it up and grabbed a bucket of popcorn and a movie.  When it finished, I ran it again on the USB hard drive: just change /tmp to /mnt/usb.  The output is just appended to the bonnie.csv file.

[jbastian@fedora-arm ~]$ bonnie++ -d /mnt/usb -n 0 -m "USB HD" -q 1>>/bonnie/bonnie.csv 2>/dev/null

The output is CSV, but bonnie++ includes a tool to convert it to HTML to make it more readable:

[jbastian@fedora-arm ~]$ bon_csv2html /bonnie/bonnie.csv > bonnie.html

I used tidy -i to reformat the HTML a bit so I could remove the columns for the file creation tests (which I did not run), and the results look like:

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
USB HD2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms

As I expected, the SD card clearly wins with random seek times: 663ms vs 7643ms.  This is one of the major advantages of flash storage over traditional spinning magnetic media since flash does not have to move a head over the disk and then wait for the desired sector to spin under the head.  It's interesting, though, that even with the significantly slower seek times, the hard drive managed to squeeze out 93.9 seeks/sec compared to only 27.4 seeks/sec on the SD card.

The throughput results are more interesting. On sequential output, the hard drive is the winner with 9017 K/sec compared to the SD card's 3125 K/sec.  Writing data is one of the weaknesses of flash storage since it has to re-write an entire block if you need to flip a bit from a 0 to 1. For sequential input, the SD card managed a respectable 17598 K/sec vs the hard drive's 12357 K/sec.

Overall, the SD card performs nicely on large sequential reads, but the USB hard drive seems to outperform it in general usage.  Unfortunately, neither drive came close to the 30 MB/sec I was hoping for.

Just for fun, I also ran zcav, another tool provided by bonnie++ to test the read rates over a hard drive.  On a normal hard drive, the read rates will be faster on the outer edge of the disk since the outer tracks have a longer circumference and thus have more sectors compared to the inner tracks, thus more sectors pass under the head in one revolution of the disk on the outer tracks.

On an SD card, the rates should be constant since there is no spinning disk.

The zcav needs access to the raw device, so I'll run it as root.  Again, store the zcav output in the RAM disk so it doesn't interfere with the testing.  I only tested the first 8 GB of the hard drive since the SD card is only 8 GB in size, and also because testing all 200 GB would take all day!

[root@fedora-arm ~]# zcav -l /bonnie/zcav-sdcard.out /dev/mmcblk0
[root@fedora-arm ~]# zcav -r 8192 -l /bonnie/zcav-usbhd.out -f /dev/sda

The output is just a series of numbers in three columns:
#block offset (GiB), MiB/s, time
0.00 14.59 17.547
0.25 16.10 15.897
0.50 15.09 16.966
...

I think this will be easier to make sense of as a graph, so gnuplot to the rescue:

[jbastian@fedora-arm ~]$ gnuplot
...
gnuplot> set terminal png
Terminal type set to 'png'
Options are 'nocrop font /usr/share/fonts/dejavu/DejaVuSans.ttf 12 size 640,480 '
gnuplot> set output "zcav_graph.png"
gnuplot> set title "ZCAV"
gnuplot> set xlabel "Block Offset (GiB)"
gnuplot> set ylabel "MiB/sec"
gnuplot> set yrange [0:20]
gnuplot> plot "zcav-sdcard.out" using 1:2 title 'SD Card' with lines, \
>             "zcav-usbhd.out" using 1:2 title 'USB HD' with lines
gnuplot> quit

And the graph:

The SD card seems to be reading at about 15 MiB/sec and the USB hard drive at about 10 MiB/sec.  This is a little below the first bonnie++ run showing 17 Mib/sec and 12 MiB/sec respectively, but the SD card is still consistently faster than the USB hard drive at sequential reads.

For comparison, I ran bonnie++ on a desktop system (Core 2 Quad @ 2.66 GHz, 6 GiB RAM, 320 GiB SATA hard drive) running Red Hat Enterprise Linux 6.1 and it had a nice 58 MiB/sec for sequential writes, 72 MiB/sec for sequential reads, and a 1066ms latency for random seeks.  Here's the full set of numbers compared to the PandaBoard:

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
USB HD2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms
Desktop SATA11G578975885111256554263694723244176.91
Latency13998us3126ms411ms25294us862ms1066ms

My conclusion: I will continue using the SD card to hold the root filesystem.  I was hoping to get a performance boost out of a hard drive, but I think the USB bus is the bottleneck for the hard drive.  Hopefully the PandaBoard 2.0 (or whatever it will be called) will have a SATA port on it!

Monday, May 9, 2011

Ubuntu on the PandaBoard

Once I was in the mood for trying other operating systems, I thought I'd try Ubuntu for ARM and see how it compared to Fedora and Android.

I downloaded the "Texas Instruments OMAP4 preinstalled netbook image for OMAP4 boards" of Natty Narwhal (11.04) from http://cdimage.ubuntu.com/releases/11.04/release/

Installation was much simpler than Fedora or Android (/dev/mmcblk0 was my SD card); using sudo or running as root:
gunzip -c ubuntu-11.04-preinstalled-netbook-armel+omap4.img.gz | dd bs=4M of=/dev/mmcblk0
sync

I moved the SD card over to the PandaBoard and turned on the power and that was it!  Wow.

It took a long time to boot due to the slow SD card (again, a future post), but eventually it was up and running and I was browsing the web with Firefox.

X-Windows was just using the basic frame buffer driver so the graphics were not very fast, but it was certainly usable for basic web browsing.  I'll have to try the accelerated drivers later.

Some nice touches:

  1. The disk image included the two partitions necessary for the PandaBoard to boot.
  2. The image actually included a kernel (the Fedora images require you to build your own kernel or download one from omappedia.org or pandaboard.org)
  3. During the first boot, it resized the ext3 partition to fill up the SD card; the image had a 2.4 GiB partition for the root filesystem, but it expanded and filled up my 8 GiB SD card.
  4. The NIC has a fixed MAC address without passing a kernel command line argument.  I need to figure out how it did this.
  5. If you need to make changes to the kernel boot arguments, there's a flash-kernel script that updates the boot.scr file for you.  (This reminds of me of running lilo way back when to update the boot process on x86.)

Android on the PandaBoard

For some weekend fun, I thought I'd check out Android on the PandaBoard (a.k.a. "Pandroid", heh) and see how Android compared to my webOS phone (a Palm Pre).

The "installation" of Android was roughly equivalent to the Fedora 13 experience:
  1. Create a small FAT partition and a larger ext3 partition on the SD card
  2. Download Android (I grabbed L27.10.2-P1_pandroid_rls.tar.bz2)
  3. Copy the bootloader/* files to the boot partition (MLO, u-boot.bin, and uImage)
  4. Copy the myfs/* files to the root filesystem partition
  5. Unmount the SD card and move it to the PandaBoard
Booting Android differed a bit from Fedora:
  1. Connect USB2Serial cable and fire up minicom
  2. Attach the power and...  it gives an error:
    booti: cannot find 'boot' partition
    Fastboot entered...
After searching the web for a while, I found a different set of boot arguments in the L27.10.2-P1 release notes to try (as opposed to this set of arguments):
setenv bootargs 'console=ttyO2,115200n8 androidboot.console=ttyO2 mem=456M@0x80000000 mem=512M@0xA0000000 root=/dev/mmcblk0p2 rw rootdelay=2 init=/init vram="32M" omapfb.vram=0:16M,1:16M consoleblank=0'
setenv bootcmd 'mmcinit 0;fatload mmc 0 0x80000000 uImage; bootm 0x80000000'
boot

Trying it again:
  1. Hit the power reset button, then hit a key in minicom to interrupt the autoboot sequence
  2. At the 'PANDA #' prompt, enter (copy and paste, actually) the above setenv commands for bootargs and bootcmd and then boot
It took a while to load (due to the slow SD card, more on this in a future post), but eventually Android popped up on my LCD monitor (connected with an HDMI-A to DVI-D cable).  The touch screen normally used on phone is simulated with a USB mouse and a regular cursor (left button is a finger tap, middle button is the menu, and right button is back), however, the cursor for the mouse was really slow and jumped around to try to keep up with mouse movements.
    A bit more reading and I learned that the Android image does not include accelerated drivers for the PandaBoard's PowerVR SGX540 chipset.  Fortunately, drivers are just another quick download from Texas Instruments.  I grabbed the drivers for L27.10.2-P1 image and copied them to the SD card (using my regular Linux laptop again), booted once more, and the graphics were nice and snappy!

    It seemed to have trouble bringing up the network (I think it needs more drivers), so I couldn't actually do a whole lot except fire up the basic apps that came with Android, e.g., calculator, address book, nothing too exciting, but the core OS worked!

    Friday, May 6, 2011

    PandaBoard Networking with Fedora 13

    With Fedora 13 booting on the PandaBoard, the next step is to bring up the network so I can install more useful software with yum.

    The default configuration for the usb0 NIC is for DHCP:

    [root@fedora-arm ~]# cat /etc/sysconfig/network-scripts/ifcfg-usb0
    DEVICE=usb0
    BOOTPROTO=dhcp
    ONBOOT=no

    But DHCP isn't working for me, it just hangs:

    [root@fedora-arm ~]# ifup usb0

    Determining IP information for usb0...Unknown HZ value! (94) Assume 100.

    I think the Unknown HZ value message is unrelated to the DHCP problem.

    I'm not sure what's going on with DHCP, so I will just try a static IP address for now. First, I need the MAC address of the interface:

    [root@fedora-arm ~]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr 4A:0B:B3:2A:3E:67
              ...

    With the MAC address, I can create a config file for the NIC:

    [root@fedora-arm ~]# cat /etc/sysconfig/network-scripts/ifcfg-usb0
    DEVICE=usb0
    BOOTPROTO=static
    ONBOOT=yes
    HWADDR=4A:0B:B3:2A:3E:67
    TYPE=Ethernet
    IPADDR=192.168.1.111
    NETMASK=255.255.255.0
    GATEWAY=192.168.1.254

    Bring up the interface:

    [root@fedora-arm network-scripts]# ifup usb0
    [root@fedora-arm network-scripts]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr 8A:F4:D3:E9:EB:1A
              inet addr:192.168.1.111 Bcast:192.168.1.255 Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST MTU:1492 Metric:1
              ...

    And ping the router:

    [root@fedora-arm network-scripts]# ping 192.168.1.254
    PING 192.168.1.254 (192.168.1.254) 56(84) bytes of data.
    64 bytes from 192.168.1.254: icmp_seq=1 ttl=255 time=6.80 ms
    64 bytes from 192.168.1.254: icmp_seq=2 ttl=255 time=0.794 ms
    ^C

    Networking is up and running!

    Finally, as a sanity check, I rebooted to make sure networking still comes up okay.

    Wait, what's this?

    [root@fedora-arm network-scripts]# reboot
    ...
    Bringing up loopback interface: [ OK ]
    Bringing up interface usb0: Device usb0 has different MAC address than expected
    , ignoring.
    [FAILED]
    ...

    The MAC address is different? How is that possible?

    Fedora release 13 (Goddard)
    Kernel 2.6.35-g6d019da-dirty on an armv7l (console)

    fedora-arm login: root
    Password:
    Last login: Sat Jan 1 00:00:17 on console
    [root@fedora-arm ~]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr CA:70:56:5B:23:7A
              BROADCAST MULTICAST MTU:1492 Metric:1
              ...

    Wow, that HWaddr is nothing like the address on the previous boot!

    A quick Google search later and I learned the NIC on the PandaBoard does not have a MAC address stored in the EEPROM because it doesn't have an EEPROM!  Thus, it generates a new random MAC address on every boot. Argh.
    http://lwn.net/Articles/435894/
    https://bugs.launchpad.net/ubuntu/+source/linux-ti-omap4/+bug/673504

    The workaround for now is to pass in a MAC address of your choosing on the kernel command line, for example, smsc95xx.macaddr=01:02:03:04:05:06

    So, I powered-down the the PandaBoard and moved the SD card over to my regular Linux workstation to create a new boot.scr file.

    First, I had to yum install uboot-tools to get the mkimage command. Next, I created a boot.cmd file on the first partition of the SD card:

    [root@localhost ~]# mount /dev/mmcblk0p2 /mnt/panda-rootfs
    [root@localhost ~]# mount /dev/mmcblk0p1 /mnt/panda-rootfs/boot
    [root@localhost ~]# cd /mnt/panda-rootfs/boot
    [root@localhost boot]# vi boot.cmd
    [root@localhost boot]# cat boot.cmd
    fatload mmc 0:1 0x80000000 uImage
    setenv bootargs 'ro elevator=noop vram=32M root=/dev/mmcblk0p2 fixrtc console=ttyO2,115200 mem=460M@0x80000000 mem=460M@0xA0000000 smsc95xx.macaddr=CA:BC:15:4D:B4:49'
    bootm 0x80000000

    And then I converted it into the boot script format with the mkimage command:

    [root@localhost boot]# mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n 'Pandaboard boot script' -d boot.cmd boot.scr

    Unmount the SD card and move it back to the Pandaboard and boot it.

    fedora-arm login: root
    Password:
    Last login: Fri May 6 11:03:29 from 192.168.1.76
    [root@fedora-arm ~]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr CA:BC:15:4D:B4:49
              ...

    The MAC address looks good! Let's update the ifcfg-usb0 and set the HWADDR line again.

    [root@fedora-arm ~]# vi /etc/sysconfig/network-scripts/ifcfg-usb0
    [root@fedora-arm ~]# grep HWADDR /etc/sysconfig/network-scripts/ifcfg-usb0
    HWADDR=CA:BC:15:4D:B4:49

    And one more reboot just-to-be-sure:

    ...
    Bringing up loopback interface: [ OK ]
    Bringing up interface usb0: [ OK ]
    ...
    Fedora release 13 (Goddard)
    Kernel 2.6.35-g6d019da-dirty on an armv7l (console)

    fedora-arm login: root
    Password:
    Last login: Fri May 6 11:55:45 on console
    [root@fedora-arm ~]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr CA:BC:15:4D:B4:49
              inet addr:192.168.1.111 Bcast:192.168.1.255 Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST MTU:1492 Metric:1
              ...

    Yay! It works!


    One final note: in addition to the lack of EEPROM to store the MAC address, there also does not seem to be a system clock. Every time I boot, the system date is set to midnight, January 1, 2000. ntpdate and ntpd should help with that. I installed both and set the step-tickers (used by ntpdate) to:

    [root@fedora-arm ~]# cat /etc/ntp/step-tickers
    # List of servers used for initial synchronization.
    0.fedora.pool.ntp.org
    1.fedora.pool.ntp.org
    2.fedora.pool.ntp.org

    Then enable both ntpdate and ntpd with chkconfig:

    [root@fedora-arm ~]# chkconfig ntpdate on
    [root@fedora-arm ~]# chkconfig ntpd on

    And now the date should be correct after a reboot.

    [root@fedora-arm ~]# reboot

    Broadcast message from root@fedora-arm
    (/dev/console) at 12:17 ...
    ...
    Starting sshd: [ OK ]
    ntpdate: Synchronizing with time server: [ OK ]
    Starting ntpd: [ OK ]

    Fedora release 13 (Goddard)
    Kernel 2.6.35-g6d019da-dirty on an armv7l (console)

    fedora-arm login: root
    Password:
    Last login: Fri May 6 12:00:31 on console
    [root@fedora-arm ~]# date
    Fri May 6 12:18:38 CDT 2011

    That looks better!