A beautiful custom wood case for my Pandaboard

My father, Dennis, built a beautiful wooden case to protect my Pandaboard from dust and the elements (and to hide the bright LEDs at night).  Here it is running Fedora 19!

U-Boot 2013.04 (Jun 17 2013 - 12:42:04)

CPU : OMAP4430 ES2.1
Board: OMAP4 Panda
I2C: ready
DRAM: 1 GiB
MMC: OMAP SD/MMC: 0
Using default environment
...

Fedora release 19 (Schrödinger’s Cat)
Kernel 3.9.5-301.fc19.armv7hl on an armv7l (ttyO2)

panda login: jbastian
Password:
[jbastian@panda ~]$ uname -a
Linux panda.localdomain 3.9.5-301.fc19.armv7hl #1 SMP Wed Jun 12 14:56:17 UTC 2013 armv7l armv7l armv7l GNU/Linux


Extract vmlinux with the power of the command line

While working on a problem today, I wanted to search the kernel image for some strings of text.  The kernel image, however, is compressed which means a simple strings-and-grep on the image file won't find anything:

$ file /boot/vmlinuz-3.6.2-4.fc17.x86_64
/boot/vmlinuz-3.6.2-4.fc17.x86_64: Linux kernel x86 boot executable bzImage...
$ strings /boot/vmlinuz-3.6.2-4.fc17.x86_64 | grep microcode
$

Note that bzImage does not necessarily mean the kernel was compressed with bzip2!  It's typically done with gzip.

But a simple gunzip won't work either.  I'll actually use zcat since it doesn't expect a .gz suffix on the filename:

$ zcat /boot/vmlinuz-3.6.2-4.fc17.x86_64
gzip: /boot/vmlinuz-3.6.2-4.fc17.x86_64: not in gzip format

The kernel image is a self-extracting compressed file.  There's a small bit of code at the beginning of the file which extracts the compressed payload in the remainder of the file.  I need to strip that chunk of code from the beginning in order to use gunzip (or zcat).  But how many bytes do I need to strip?

Many file types start with a simple header including a "magic number" (just a well-known fixed number) to help identify and verify the file's type.  If I can find the gzip magic number somewhere in the vmlinuz file, I can strip all the bytes before the magic number.

Files compressed with gzip start with 0x1F8B0800.  A simple test can verify that.  Let's compress a small bit of data and send it to xxd (a hex dumper) and show the first 4 bytes:

$ echo "hello world" | gzip -c | xxd -l4
0000000: 1f8b 0800                                ....

Ok, so now I need to find 0x1F8B0800 somewhere in the vmlinuz file.  Let's start simple with xxd and grep:

$ xxd /boot/vmlinuz-3.6.2-4.fc17.x86_64 | grep "1f8b 0800"
$

Nothing?  Hmm, maybe it's not grouped the same way?  That is, maybe it's "__1f 8b08 00__"?  Fortunately, xxd can change the grouping using the -g option.  Let's use -g1 to just show 1 byte per group, i.e., "1f 8b 08 00".

$ xxd -g1 /boot/vmlinuz-3.6.2-4.fc17.x86_64 | grep "1f 8b 08 00"
$

Still nothing?  Maybe the magic number is split on a line break?  Let's try something shorter, just the first 3 bytes, with a line of context:

$ xxd -g1 /boot/vmlinuz-3.6.2-4.fc17.x86_64 | grep -C1 "1f 8b 08"
00044b0: 78 49 00 48 c7 c1 40 78 49 00 48 c1 e9 03 fd f3  xI.H..@xI.H.....
00044c0: 48 a5 fc 5e 48 8d 83 00 08 49 00 ff e0 1f 8b 08  H..^H....I......
00044d0: 00 00 00 00 00 02 03 ec dd 09 7c 14 e5 1d f8 ff  ..........|.....

Ah-hah, there it is!  The "1f 8b 08" is at the end of the second line and the "00" is at the beginning of the third line.  The byte 0x1f is on the line with offset 0x44c0 and it's 13, or 0xd, bytes into the line, so the magic number begins at offset 0x44cd.  Converting to decimal with the bc command (note: bc expects hex digits in all caps!):

$ echo 'ibase=16; 44CD' | bc
17613

So now I can use dd to extract the compressed image by skipping over the first 17613 bytes of the vmlinuz file:

$ dd bs=1 skip=17613 if=/boot/vmlinuz-3.6.2-4.fc17.x86_64 of=/tmp/vmlinux.gz
4814195+0 records in
4814195+0 records out
4814195 bytes (4.8 MB) copied, 6.28984 s, 765 kB/s

And just to double-check the header:

$ xxd -g1 -l4 /tmp/vmlinux.gz
0000000: 1f 8b 08 00                                      ....

It looks good, so run it through gunzip:

$ gunzip /tmp/vmlinux.gz
gzip: /tmp/vmlinux.gz: decompression OK, trailing garbage ignored

Great!

But, clearly this method is fragile since the magic number is difficult to find with line-breaks and grouping and what-not.  There must be a better way.  What I really need is a simple raw hex dump of the kernel — no offset at the beginning of each line, no byte grouping, no line breaks, no ASCII interpretation at the end of each line — and then I know I can find 1f8b0800 and its offset.

Perusing the xxd man page, I see the -g0 option will remove all grouping, and the -p option will give a plain hex dump (no offsets, no ASCII).  That's a good start:

$ xxd -g0 -p /boot/vmlinuz-3.6.2-4.fc17.x86_64
4d5aea0700c0078cc88ed88ec08ed031e4fbfcbe4000ac20c07409b40ebb
0700cd10ebf231c0cd16cd19eaf0ff00f000000000000000000000000000
b800000044697265637420666c6f70707920626f6f74206973206e6f7420
...
...

I can join each line to make one continuous string by deleting the newline characters with the tr command (and use head to only show the first 100 bytes instead of dumping megabytes of hex into my terminal):

$ xxd -g0 -p /boot/vmlinuz-3.6.2-4.fc17.x86_64 | tr -d '\n' | head -c 100
4d5aea0700c0078cc88ed88ec08ed031e4fbfcbe4000ac20c07409b40ebb0700cd10ebf231c0cd16cd19eaf0ff00f0000000

The grep command has a -b option to print the byte offset of the matching string.  Combine it with the -o option to only print the first match:

$ xxd -g0 -p /boot/vmlinuz-3.6.2-4.fc17.x86_64 | tr -d '\n' | grep -b -o 1f8b0800
35226:1f8b0800

Ah-hah, I have an offset of 35226!  But, remember, 1 byte is printed as 2 hex digits, so it has to be divided by 2.  And I only need the offset (the field before the colon).  I can do both with awk:

$ xxd -g0 -p /boot/vmlinuz-3.6.2-4.fc17.x86_64 | tr -d '\n' | grep -b -o 1f8b0800 | awk -F: '{print $1/2}'
17613

Hooray, 17613 is the same offset I found the hard way above!
I can actually combine all of this into one big command:

$ dd bs=1 skip=$(xxd -g0 -p /boot/vmlinuz-3.6.2-4.fc17.x86_64 | \
                 tr -d '\n' | grep -b -o 1f8b0800 | awk -F: '{print $1/2}') \
     if=/boot/vmlinuz-3.6.2-4.fc17.x86_64 | gzip -d -c > /tmp/vmlinux
4814195+0 records in
4814195+0 records out
4814195 bytes (4.8 MB) copied, 6.00158 s, 802 kB/s

gzip: stdin: decompression OK, trailing garbage ignored

And finally, I can search for my strings:

$ strings /tmp/vmlinux | grep microcode
microcode    : 0x%x
4Atom PSE erratum detected, BIOS microcode update recommended
6perf_event_intel: PEBS enabled due to microcode update
6perf_event_intel: PEBS disabled due to CPU errata, please upgrade
microcode
0mce: [Hardware Error]: PROCESSOR %u:%x TIME %llu SOCKET %u APIC %x
microcode %x
perf_check_microcode

Whew.

Update: Shortly after writing this blog post, I found the grep command can search binary files for hex values with a couple more options: -a to search binary files, and -P to use Perl-compatible regular expressions.  This shortens the command to find the offset to a much simpler:

$ grep -a -b -o -P '\x1f\x8b\x08\x00' /boot/vmlinuz-3.6.2-4.fc17.x86_64 | awk -F: '{print $1}'
17613

And if I use a couple of environment variable, it becomes much easier to read:

$ export VMLINUZ=/boot/vmlinuz-3.6.2-4.fc17.x86_64
$ SKIP=$(grep -aboP '\x1f\x8b\x08\x00' $VMLINUZ | awk -F: '{print $1}')
$ dd bs=1 skip=$SKIP if=$VMLINUZ | gzip -d -c > /tmp/vmlinux
4814195+0 records in
4814195+0 records out
4814195 bytes (4.8 MB) copied, 5.15674 s, 934 kB/s

gzip: stdin: decompression OK, trailing garbage ignored

Static workspaces and Keyboard Shortcuts in Gnome Shell 3.4

I've been a KDE and Xfce user for a long time, but I've been experimenting with Gnome Shell lately out of curiosity.  I tend to use a lot of workspaces (or virtual desktops) to organize my work — email on workspace 2, developing code on workspace 3, web browser on 4, etc. — so I really do not like the dynamic workspaces feature in Gnome Shell because closing the last window on a workspace destroys that workspace and shifts all the others around which confuses me.

But with Fedora 17 and Gnome Shell 3.4, it is possible to disable dynamic workspaces and set a fixed number of workspaces instead.  (It was possible to do this earlier with the Frippery extensions.)  Fire up gnome-tweak-tool and go to the Shell settings to make the change:

I also like to use keyboard shortcuts to switch between workspaces and move windows to various workspaces.  CTRL-F1 to go to workspaces 1, CTRL-F2 for workspace 2, etc.  And CTRL-SHIFT-FN to move a window to workspace N.  Unfortunately, the Keyboard Settings Shortcuts GUI only shows the hotkey settings for desktops 1 through 4:

It is possible to create keyboard shortcuts for desktops 5 and up, but the settings are hidden away in the gsettings database.  So fire up a terminal window and run:

[user@localhost ~]$ gsettings set org.gnome.desktop.wm.keybindings \
    switch-to-workspace-5 "[\"<Control>F5\"]"
[user@localhost ~]$ gsettings set org.gnome.desktop.wm.keybindings \
    move-to-workspace-5 "[\"<Control><Shift>F5\"]"

This sets CTRL-F5 to switch to workspace 5, and CTRL-SHIFT-F5 to move the current window to workspace 5.  Feel free to choose other keys like ALT or SUPER.

To make life easier, I wrote this simple script to change all the hotkeys for desktops 1 through 12:

#!/bin/bash

# Disable dynamic workspaces and set 12 fixed workspaces, and set the
# hot-keys for switching and moving windows to the workspaces.
#
# The Gnome 3.4 Shell keyboard settings GUI only exposes hot-key configuration
# for workspaces 1-4 so you have to use the command line for spaces 5+.
#
# Jeff Bastian, 2012-06-20

echo "Disabling dynamic workspaces"
gsettings set org.gnome.shell.overrides dynamic-workspaces false

echo "Setting 12 fixed workspaces"
gsettings set org.gnome.desktop.wm.preferences num-workspaces 12

for ((x=1 ; x <= 12 ; x++)) ; do
    echo "Setting hotkeys for workspace $x"
    gsettings set org.gnome.desktop.wm.keybindings \
        switch-to-workspace-$x "[\"<Control>F$x\"]"
    gsettings set org.gnome.desktop.wm.keybindings \
        move-to-workspace-$x "[\"<Control><Shift>F$x\"]"
done

echo "Done"
echo "~~~~"
echo "Verify Settings:"
gsettings list-recursively org.gnome.shell.overrides | grep dynamic-workspaces
gsettings list-recursively org.gnome.desktop.wm.preferences | grep num-workspaces
gsettings list-recursively org.gnome.desktop.wm.keybindings | grep to-workspace

A PandaBoard Desktop with Fedora 13

A serial port console, while nice and simple to work on, is a little lacking for eye candy, so today I installed X Windows and Xfce to run a desktop on my PandaBoard.

The installation was the easy part:

yum -y groupinstall Base 'X Window System' 'XFCE' 'GNOME Desktop Environment'

And then it was hurry-up-and-wait while it installed 421 packages.

When it finished, I rebooted it and found a few problems that I had to fix.  The default resolution for the display is a safe 640x480, so I had to add a kernel command line option to make it 1280x1024: omapfb.mode=dvi:1280x1024MR-24@60.  I also added another console=tty0 option so both the serial port and the display could be consoles.  The last console parameter is the primary console and for now I kept the serial port as primary for easier debugging.

[root@fedora-arm ~]# mount /dev/mmcblk0p1 /mnt/boot
[root@fedora-arm ~]# cd /mnt/boot
[root@fedora-arm boot]# vim boot.cmd
[root@fedora-arm boot]# cat boot.cmd
setenv mpurate 800
setenv bootargs 'console=tty0 console=ttyO2,115200n8 ro rootwait root=/dev/sda1 mpurate=${mpurate} init=/sbin/init earlyprintk rd_NO_PLYMOUTH selinux=0 omapfb.mode=dvi:1280x1024MR-24@60'
setenv bootcmd 'mmc rescan 0; mmc init; fatload mmc 0:1 0x80300000 uImage; fatload mmc 0:1 0x81600000 uInitrd; bootm 0x80300000 0x81600000'
boot
[root@fedora-arm boot]# mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n 'Pandaboard Boot Script' -d boot.cmd boot.scr
Image Name: Pandaboard Boot Script
Created: Sat Nov 26 14:22:00 2011
Image Type: ARM Linux Script (uncompressed)
Data Size: 359 Bytes = 0.35 kB = 0.00 MB
Load Address: 00000000
Entry Point: 00000000
Contents:
Image 0: 351 Bytes = 0.34 kB = 0.00 MB
[root@fedora-arm boot]# cd
[root@fedora-arm ~]# umount /mnt/boot

I then updated /etc/inittab to make runlevel 5 the default and rebooted the PandaBoard again.

I noticed ntpdate was failing to set the clock.  I tried running it manually and noticed it couldn't resolve the hostname 0.fedora.pool.ntp.org.  A quick peek at /etc/resolv.conf showed that NetworkManager had re-written the file with nothing but comments because I have /etc/sysconfig/network-scripts/ifcfg-eth0 set up for a static IP address for now.  So I disabled NetworkManager with chkconfig and cleaned up resolv.conf and rebooted once more.

This time it almost worked.  It went through all the boot scripts cleanly, but it got stuck when it should have started X11 and gdm.  Fortunately the serial console still worked, so I logged in and discovered that the /etc/rc.d/rc script was still running and starting up runlevel 5, but it didn't seem to be doing anything.  I then found it was waiting for /etc/rc.d/rc5.d/S99local to finish (aka /etc/rc.d/rc.local).  I looked at this script and remembered the "ugly hack" I had to set up back in April to run agetty on the serial port.  It was running in a never-ending loop which means it will never finish and the init scripts will never start X11.

The Fedora ARM page no longer mentions this ugly hack, so I guess the bug has been fixed and I can use a normal upstart job to manage agetty on the serial port.  I commented out the hack in the rc.local script and created a new /etc/init/serial-ttyO2.conf job:

[root@fedora-arm ~]# cat /etc/rc.d/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# horrible hack to respawn serial console
###while true
###do
###    /sbin/agetty -L 115200 console vt100
###done

[root@fedora-arm ~]# cat /etc/init/serial-ttyO2.conf
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [016]

respawn
pre-start exec /sbin/securetty ttyO2
exec /sbin/agetty /dev/ttyO2 115200 vt100-nav

[root@fedora-arm ~]# initctl start serial-ttyO2
serial-ttyO2 start/running, process 20245

One final reboot and ... Hooray!  It ran through the firstboot setup and then launched X11 and gdm.  I logged in to Xfce and launched Firefox and a Terminal.  I now have a completely silent desktop!

System Clock on the PandaBoard

A quick way to find the physical device backing a filesystem is to use the 'df' command.  (LVM, software-RAID, LUKS, and other layers can make this more complicated.)  I did this on my PandaBoard and was a little surprised at the answer:

[root@fedora-arm ~]# df -Th /
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/root      nfs    7.4G  1.8G  5.3G  25% /

Why is it NFS-mounting a USB flash drive?  And /dev/root doesn't tell me what the device name is.  I was expecting to see something like sda1 or mmcblk0p1.

I thought that was weird, so I edited /etc/fstab and changed the device and filesystem type:

[root@fedora-arm ~]# diff -u /etc/fstab.ORIG /etc/fstab
--- /etc/fstab.ORIG        2011-11-14 10:09:40.000000000 -0600
+++ /etc/fstab  2011-11-14 12:48:58.000000000 -0600
@@ -1,4 +1,4 @@
-/dev/root               /                       nfs     defaults        1 1
+/dev/sda1               /                       ext3    defaults        1 1

And then I rebooted my PandaBoard and quickly learned why it's set to NFS:

                Welcome to Fedora
                Press 'I' to enter interactive startup.
Starting udev: [  OK  ]
Setting hostname fedora-arm:  [  OK  ]
Setting up Logical Volume Management:   No volume groups found
[  OK  ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
fedoraroot: Superblock last mount time (Mon Nov 14 11:19:35 2011,
        now = Sat Jan  1 00:00:01 2000) is in the future.


fedoraroot: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
[FAILED]

There's no battery-backed clock on the PandaBoard so the clock gets reset to 2000-01-01 00:00:00 every time it's power-cycled, and this causes the Fedora boot scripts to get confused and force a filesystem check.

Apparently mounting the root filesystem as NFS is one way to workaround  this problem.

I mentioned this in #fedora-arm on freenode and Chris Tyler suggested modifying the boot scripts to set the clock to 1 minute past the previous write time of the root filesystem.

Here's what I came up with:

--- /etc/rc.d/rc.sysinit.ORIG    2011-03-08 15:51:46.000000000 -0600
+++ /etc/rc.d/rc.sysinit    2011-11-14 12:44:28.441405990 -0600
@@ -577,6 +577,15 @@
 fi
 if [ -z "$fastboot" -a "$READONLY" != "yes" ]; then

+        # set the clock on the PandaBoard to 1 minute after last write
+        # time of root filesystem to avoid "last mount is in the future"
+        echo "Current system clock is $(date)"
+        rootdev=$(egrep '\s/\s' /etc/fstab | awk '{print $1}')
+        lastwrite=$(dumpe2fs -h $rootdev 2>/dev/null | \
+                    grep "Last write time" | sed -e 's/^[^:]*:\s*//')
+        echo "Setting system clock to 1 minute after $lastwrite"
+        date -s "$lastwrite + 1 minute"
+
         STRING=$"Checking filesystems"
         echo $STRING
         fsck -T -t noopts=_netdev -A $fsckoptions

Notice the new messages (red text) during boot:

                Welcome to Fedora
                Press 'I' to enter interactive startup.
Starting udev: [  OK  ]
Setting hostname fedora-arm:  [  OK  ]
Setting up Logical Volume Management:   No volume groups found
[  OK  ]
Current system clock is Sat Jan  1 00:00:01 CST 2000
Setting system clock to 1 minute after Mon Nov 14 12:45:36 2011
Mon Nov 14 12:46:36 CST 2011
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
fedoraroot: clean, 69076/489600 files, 487736/1957605 blocks
[  OK  ]
...

And the root filesystem and device look correct now!

[root@fedora-arm ~]# df -Th /
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3    7.4G  1.8G  5.3G  25% /

Note that I do have ntpd start later in the boot process which sets the actual correct time.

SD Card Speeds

After yesterday's benchmarks revealed a possible problem with the SD card performance, Mark Salter asked me to run a few more tests to see if the DMA API patch set is hurting the SD card (but helping USB storage).

I recompiled the 2.6.40.3-2.02.fc13.armv7l.omap kernel but I disabled the DMA API patch set.  I also tested with smp (the default) and nosmp.  And just for good measure, I re-tested the 2.6.35-g6d019da-dirty kernel because the last time I tested 2.6.35 was in May and I've updated a few packages since then.

It does not appear that the DMA API patch set is to blame for the slower SD card performance.  The random seek times were longer than 76000ms for all variants of 2.6.40.3 compared to just 664ms on 2.6.35.  The sequential reads were fastest with 2.6.35, but today's test was quite a bit faster than May's test; I'm not sure what to make of that.  Likewise, sequential writes were also fastest with 2.6.35.

Thus, some other change between 2.6.35 and 2.6.40 must be responsible for the drop in SD card performance, but what?

Version 1.96		Sequential Output						Sequential Input				Random Seeks
	Size	Per Char		Block		Rewrite		Per Char		Block
		K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU
SD Card 2.6.35 (May 2011)	2G	41	93	3125	5	3371	4	336	99	17598	15	27.4	1
	Latency	1168ms		33487ms		32391ms		44466us		2628ms		663ms
SD Card 2.6.35 (Sept 2011)	2G	39	91	3118	5	3465	4	448	99	19104	16	27.9	1
	Latency	3900ms		42424ms		54958ms		19929us		667ms		664ms
SD Card 2.6.40.3 + DMA patch + SMP	2G	22	65	1862	4	2544	3	339	99	19626	16	15.6	1
	Latency	2793ms		29928ms		39111ms		50446us		665ms		76023ms
SD Card 2.6.40.3 + DMA patch + nosmp	2G	19	58	1669	3	2513	3	343	99	20829	13	17.2	1
	Latency	2858ms		38268ms		42161ms		51148us		667ms		76651ms
SD Card 2.6.40.3 + no patch + SMP	2G	26	76	1669	3	2271	3	411	99	19592	16	15.7	1
	Latency	2855ms		34616ms		58124ms		22279us		666ms		82254ms
SD Card 2.6.40.3 + no patch + nosmp	2G	18	54	1603	3	2395	3	382	99	20894	13	16.5	1
	Latency	5020ms		33958ms		29546ms		51301us		666ms		77587ms

Storage Speed on the PandaBoard Revisited (Again)

Since my last post, I took a short vacation and I've been helping in the Virtual Fedora Activity Days (VFAD) to rebuild packages for Fedora 15 hardfp (hardware floating point).

Research into the slow storage speeds on the PandaBoard (and other Cortex-ARM9 systems) has continued and a patch set developed to hopefully improve the situation.  David Marlin built a 2.6.40.3-2.02.fc13.armv7l.omap kernel with the patch set, so I downloaded a copy and ran bonnie++ again.

The results were mixed for the SD card.  Sequential reads had a nice improvement, but sequential output and random seeks took a big hit.  The random seek times are particularly worrisome with an increase from 663ms to 76023ms!  Something doesn't seem right.

The USB hard drive, though, saw some nice improvements overall. Sequential write latencies dropped from 5.8 seconds down to 0.38 seconds and throughput more than doubled: 9 MiB/s vs 23 MiB/s!  Sequential reads latencies increased a bit, but throughput again more than doubled: 12 MiB/s vs 27 MiB/s!  Random seeks also saw a nice improvement from 7.6s down to 4.9s.

Version 1.96		Sequential Output						Sequential Input				Random Seeks
	Size	Per Char		Block		Rewrite		Per Char		Block
		K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU
SD Card 2.6.35-g6d019da-dirty	2G	41	93	3125	5	3371	4	336	99	17598	15	27.4	1
	Latency	1168ms		33487ms		32391ms		44466us		2628ms		663ms
SD Card 2.6.40.3-2.02.fc13.armv7l.omap	2G	22	65	1862	4	2544	3	339	99	19626	16	15.6	1
	Latency	2793ms		29928ms		39111ms		50446us		665ms		76023ms
USB HD 2.6.35-g6d019da-dirty	2G	43	99	9017	16	4538	6	369	99	12357	9	93.9	6
	Latency	215ms		5849ms		5889ms		52858us		108ms		7643ms
USB HD 2.6.40.3-2.02.fc13.armv7l.omap	2G	39	99	23519	26	11118	15	327	99	27314	18	150.6	12
	Latency	322ms		383ms		341ms		54567us		697ms		4910ms

I also tried a simple test with hdparm as shown on lwn.net:

[root@fedora-arm ~]# uname -r
2.6.40.3-2.02.fc13.armv7l.omap

[root@fedora-arm ~]# hdparm -t /dev/mmcblk0p4

/dev/mmcblk0p4:
 Timing buffered disk reads:   50 MB in  3.10 seconds =  16.11 MB/sec

[root@fedora-arm ~]# hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   66 MB in  3.05 seconds =  21.64 MB/sec

Compared to the older kerrnel:

[root@fedora-arm ~]# uname -r
2.6.35-g6d019da-dirty

[root@fedora-arm ~]# hdparm -t /dev/mmcblk0p4

/dev/mmcblk0p4:
 Timing buffered disk reads:   50 MB in  3.09 seconds =  16.20 MB/sec

[root@fedora-arm ~]# hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   26 MB in  3.04 seconds =   8.56 MB/sec


The SD card's performance is about the same, but the USB hard drive is much faster with the new patch!