Wednesday, September 14, 2011

SD Card Speeds

After yesterday's benchmarks revealed a possible problem with the SD card performance, Mark Salter asked me to run a few more tests to see if the DMA API patch set is hurting the SD card (but helping USB storage).

I recompiled the 2.6.40.3-2.02.fc13.armv7l.omap kernel but I disabled the DMA API patch set.  I also tested with smp (the default) and nosmp.  And just for good measure, I re-tested the 2.6.35-g6d019da-dirty kernel because the last time I tested 2.6.35 was in May and I've updated a few packages since then.

It does not appear that the DMA API patch set is to blame for the slower SD card performance.  The random seek times were longer than 76000ms for all variants of 2.6.40.3 compared to just 664ms on 2.6.35.  The sequential reads were fastest with 2.6.35, but today's test was quite a bit faster than May's test; I'm not sure what to make of that.  Likewise, sequential writes were also fastest with 2.6.35.

Thus, some other change between 2.6.35 and 2.6.40 must be responsible for the drop in SD card performance, but what?

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card 2.6.35 (May 2011)2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
SD Card 2.6.35 (Sept 2011)2G3991311853465444899191041627.91
Latency3900ms42424ms54958ms19929us667ms664ms
SD Card 2.6.40.3 + DMA patch + SMP2G2265186242544333999196261615.61
Latency2793ms29928ms39111ms50446us665ms76023ms
SD Card 2.6.40.3 + DMA patch + nosmp2G1958166932513334399208291317.21
Latency2858ms38268ms42161ms51148us667ms76651ms
SD Card 2.6.40.3 + no patch + SMP2G2676166932271341199195921615.71
Latency2855ms34616ms58124ms22279us666ms82254ms
SD Card 2.6.40.3 + no patch + nosmp2G1854160332395338299208941316.51
Latency5020ms33958ms29546ms51301us666ms77587ms

Tuesday, September 13, 2011

Storage Speed on the PandaBoard Revisited (Again)

Since my last post, I took a short vacation and I've been helping in the Virtual Fedora Activity Days (VFAD) to rebuild packages for Fedora 15 hardfp (hardware floating point).

Research into the slow storage speeds on the PandaBoard (and other Cortex-ARM9 systems) has continued and a patch set developed to hopefully improve the situation.  David Marlin built a 2.6.40.3-2.02.fc13.armv7l.omap kernel with the patch set, so I downloaded a copy and ran bonnie++ again.

The results were mixed for the SD card.  Sequential reads had a nice improvement, but sequential output and random seeks took a big hit.  The random seek times are particularly worrisome with an increase from 663ms to 76023ms!  Something doesn't seem right.

The USB hard drive, though, saw some nice improvements overall. Sequential write latencies dropped from 5.8 seconds down to 0.38 seconds and throughput more than doubled: 9 MiB/s vs 23 MiB/s!  Sequential reads latencies increased a bit, but throughput again more than doubled: 12 MiB/s vs 27 MiB/s!  Random seeks also saw a nice improvement from 7.6s down to 4.9s.


Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card 2.6.35-g6d019da-dirty2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
SD Card 2.6.40.3-2.02.fc13.armv7l.omap2G2265186242544333999196261615.61
Latency2793ms29928ms39111ms50446us665ms76023ms
USB HD 2.6.35-g6d019da-dirty2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms
USB HD 2.6.40.3-2.02.fc13.armv7l.omap2G399923519261111815327992731418150.612
Latency322ms383ms341ms54567us697ms4910ms

I also tried a simple test with hdparm as shown on lwn.net:

[root@fedora-arm ~]# uname -r
2.6.40.3-2.02.fc13.armv7l.omap

[root@fedora-arm ~]# hdparm -t /dev/mmcblk0p4

/dev/mmcblk0p4:
 Timing buffered disk reads:   50 MB in  3.10 seconds =  16.11 MB/sec

[root@fedora-arm ~]# hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   66 MB in  3.05 seconds =  21.64 MB/sec

Compared to the older kerrnel:

[root@fedora-arm ~]# uname -r
2.6.35-g6d019da-dirty

[root@fedora-arm ~]# hdparm -t /dev/mmcblk0p4

/dev/mmcblk0p4:
 Timing buffered disk reads:   50 MB in  3.09 seconds =  16.20 MB/sec

[root@fedora-arm ~]# hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   26 MB in  3.04 seconds =   8.56 MB/sec

The SD card's performance is about the same, but the USB hard drive is much faster with the new patch!