Saturday, November 26, 2011

A PandaBoard Desktop with Fedora 13

A serial port console, while nice and simple to work on, is a little lacking for eye candy, so today I installed X Windows and Xfce to run a desktop on my PandaBoard.

The installation was the easy part:
yum -y groupinstall Base 'X Window System' 'XFCE' 'GNOME Desktop Environment'

And then it was hurry-up-and-wait while it installed 421 packages.

When it finished, I rebooted it and found a few problems that I had to fix.  The default resolution for the display is a safe 640x480, so I had to add a kernel command line option to make it 1280x1024: omapfb.mode=dvi:1280x1024MR-24@60.  I also added another console=tty0 option so both the serial port and the display could be consoles.  The last console parameter is the primary console and for now I kept the serial port as primary for easier debugging.

[root@fedora-arm ~]# mount /dev/mmcblk0p1 /mnt/boot
[root@fedora-arm ~]# cd /mnt/boot
[root@fedora-arm boot]# vim boot.cmd
[root@fedora-arm boot]# cat boot.cmd
setenv mpurate 800
setenv bootargs 'console=tty0 console=ttyO2,115200n8 ro rootwait root=/dev/sda1 mpurate=${mpurate} init=/sbin/init earlyprintk rd_NO_PLYMOUTH selinux=0 omapfb.mode=dvi:1280x1024MR-24@60'
setenv bootcmd 'mmc rescan 0; mmc init; fatload mmc 0:1 0x80300000 uImage; fatload mmc 0:1 0x81600000 uInitrd; bootm 0x80300000 0x81600000'
boot
[root@fedora-arm boot]# mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n 'Pandaboard Boot Script' -d boot.cmd boot.scr
Image Name: Pandaboard Boot Script
Created: Sat Nov 26 14:22:00 2011
Image Type: ARM Linux Script (uncompressed)
Data Size: 359 Bytes = 0.35 kB = 0.00 MB
Load Address: 00000000
Entry Point: 00000000
Contents:
Image 0: 351 Bytes = 0.34 kB = 0.00 MB
[root@fedora-arm boot]# cd
[root@fedora-arm ~]# umount /mnt/boot

I then updated /etc/inittab to make runlevel 5 the default and rebooted the PandaBoard again.

I noticed ntpdate was failing to set the clock.  I tried running it manually and noticed it couldn't resolve the hostname 0.fedora.pool.ntp.org.  A quick peek at /etc/resolv.conf showed that NetworkManager had re-written the file with nothing but comments because I have /etc/sysconfig/network-scripts/ifcfg-eth0 set up for a static IP address for now.  So I disabled NetworkManager with chkconfig and cleaned up resolv.conf and rebooted once more.

This time it almost worked.  It went through all the boot scripts cleanly, but it got stuck when it should have started X11 and gdm.  Fortunately the serial console still worked, so I logged in and discovered that the /etc/rc.d/rc script was still running and starting up runlevel 5, but it didn't seem to be doing anything.  I then found it was waiting for /etc/rc.d/rc5.d/S99local to finish (aka /etc/rc.d/rc.local).  I looked at this script and remembered the "ugly hack" I had to set up back in April to run agetty on the serial port.  It was running in a never-ending loop which means it will never finish and the init scripts will never start X11.

The Fedora ARM page no longer mentions this ugly hack, so I guess the bug has been fixed and I can use a normal upstart job to manage agetty on the serial port.  I commented out the hack in the rc.local script and created a new /etc/init/serial-ttyO2.conf job:

[root@fedora-arm ~]# cat /etc/rc.d/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# horrible hack to respawn serial console
###while true
###do
###    /sbin/agetty -L 115200 console vt100
###done

[root@fedora-arm ~]# cat /etc/init/serial-ttyO2.conf
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [016]

respawn
pre-start exec /sbin/securetty ttyO2
exec /sbin/agetty /dev/ttyO2 115200 vt100-nav


[root@fedora-arm ~]# initctl start serial-ttyO2
serial-ttyO2 start/running, process 20245

One final reboot and ... Hooray!  It ran through the firstboot setup and then launched X11 and gdm.  I logged in to Xfce and launched Firefox and a Terminal.  I now have a completely silent desktop!

Monday, November 14, 2011

System Clock on the PandaBoard

A quick way to find the physical device backing a filesystem is to use the 'df' command.  (LVM, software-RAID, LUKS, and other layers can make this more complicated.)  I did this on my PandaBoard and was a little surprised at the answer:

[root@fedora-arm ~]# df -Th /
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/root      nfs    7.4G  1.8G  5.3G  25% /

Why is it NFS-mounting a USB flash drive?  And /dev/root doesn't tell me what the device name is.  I was expecting to see something like sda1 or mmcblk0p1.

I thought that was weird, so I edited /etc/fstab and changed the device and filesystem type:
[root@fedora-arm ~]# diff -u /etc/fstab.ORIG /etc/fstab
--- /etc/fstab.ORIG        2011-11-14 10:09:40.000000000 -0600
+++ /etc/fstab  2011-11-14 12:48:58.000000000 -0600
@@ -1,4 +1,4 @@
-/dev/root               /                       nfs     defaults        1 1
+/dev/sda1               /                       ext3    defaults        1 1

And then I rebooted my PandaBoard and quickly learned why it's set to NFS:
                Welcome to Fedora
                Press 'I' to enter interactive startup.
Starting udev: [  OK  ]
Setting hostname fedora-arm:  [  OK  ]
Setting up Logical Volume Management:   No volume groups found
[  OK  ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
fedoraroot: Superblock last mount time (Mon Nov 14 11:19:35 2011,
        now = Sat Jan  1 00:00:01 2000) is in the future.


fedoraroot: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
[FAILED]

There's no battery-backed clock on the PandaBoard so the clock gets reset to 2000-01-01 00:00:00 every time it's power-cycled, and this causes the Fedora boot scripts to get confused and force a filesystem check.

Apparently mounting the root filesystem as NFS is one way to workaround  this problem.

I mentioned this in #fedora-arm on freenode and Chris Tyler suggested modifying the boot scripts to set the clock to 1 minute past the previous write time of the root filesystem.

Here's what I came up with:
--- /etc/rc.d/rc.sysinit.ORIG    2011-03-08 15:51:46.000000000 -0600
+++ /etc/rc.d/rc.sysinit    2011-11-14 12:44:28.441405990 -0600
@@ -577,6 +577,15 @@
 fi
 if [ -z "$fastboot" -a "$READONLY" != "yes" ]; then

+        # set the clock on the PandaBoard to 1 minute after last write
+        # time of root filesystem to avoid "last mount is in the future"
+        echo "Current system clock is $(date)"
+        rootdev=$(egrep '\s/\s' /etc/fstab | awk '{print $1}')
+        lastwrite=$(dumpe2fs -h $rootdev 2>/dev/null | \
+                    grep "Last write time" | sed -e 's/^[^:]*:\s*//')
+        echo "Setting system clock to 1 minute after $lastwrite"
+        date -s "$lastwrite + 1 minute"
+
         STRING=$"Checking filesystems"
         echo $STRING
         fsck -T -t noopts=_netdev -A $fsckoptions

Notice the new messages (red text) during boot:
                Welcome to Fedora
                Press 'I' to enter interactive startup.
Starting udev: [  OK  ]
Setting hostname fedora-arm:  [  OK  ]
Setting up Logical Volume Management:   No volume groups found
[  OK  ]
Current system clock is Sat Jan  1 00:00:01 CST 2000
Setting system clock to 1 minute after Mon Nov 14 12:45:36 2011
Mon Nov 14 12:46:36 CST 2011
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
fedoraroot: clean, 69076/489600 files, 487736/1957605 blocks
[  OK  ]
...


And the root filesystem and device look correct now!
[root@fedora-arm ~]# df -Th /
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3    7.4G  1.8G  5.3G  25% /

Note that I do have ntpd start later in the boot process which sets the actual correct time.

Wednesday, September 14, 2011

SD Card Speeds

After yesterday's benchmarks revealed a possible problem with the SD card performance, Mark Salter asked me to run a few more tests to see if the DMA API patch set is hurting the SD card (but helping USB storage).

I recompiled the 2.6.40.3-2.02.fc13.armv7l.omap kernel but I disabled the DMA API patch set.  I also tested with smp (the default) and nosmp.  And just for good measure, I re-tested the 2.6.35-g6d019da-dirty kernel because the last time I tested 2.6.35 was in May and I've updated a few packages since then.

It does not appear that the DMA API patch set is to blame for the slower SD card performance.  The random seek times were longer than 76000ms for all variants of 2.6.40.3 compared to just 664ms on 2.6.35.  The sequential reads were fastest with 2.6.35, but today's test was quite a bit faster than May's test; I'm not sure what to make of that.  Likewise, sequential writes were also fastest with 2.6.35.

Thus, some other change between 2.6.35 and 2.6.40 must be responsible for the drop in SD card performance, but what?

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card 2.6.35 (May 2011)2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
SD Card 2.6.35 (Sept 2011)2G3991311853465444899191041627.91
Latency3900ms42424ms54958ms19929us667ms664ms
SD Card 2.6.40.3 + DMA patch + SMP2G2265186242544333999196261615.61
Latency2793ms29928ms39111ms50446us665ms76023ms
SD Card 2.6.40.3 + DMA patch + nosmp2G1958166932513334399208291317.21
Latency2858ms38268ms42161ms51148us667ms76651ms
SD Card 2.6.40.3 + no patch + SMP2G2676166932271341199195921615.71
Latency2855ms34616ms58124ms22279us666ms82254ms
SD Card 2.6.40.3 + no patch + nosmp2G1854160332395338299208941316.51
Latency5020ms33958ms29546ms51301us666ms77587ms

Tuesday, September 13, 2011

Storage Speed on the PandaBoard Revisited (Again)

Since my last post, I took a short vacation and I've been helping in the Virtual Fedora Activity Days (VFAD) to rebuild packages for Fedora 15 hardfp (hardware floating point).

Research into the slow storage speeds on the PandaBoard (and other Cortex-ARM9 systems) has continued and a patch set developed to hopefully improve the situation.  David Marlin built a 2.6.40.3-2.02.fc13.armv7l.omap kernel with the patch set, so I downloaded a copy and ran bonnie++ again.

The results were mixed for the SD card.  Sequential reads had a nice improvement, but sequential output and random seeks took a big hit.  The random seek times are particularly worrisome with an increase from 663ms to 76023ms!  Something doesn't seem right.

The USB hard drive, though, saw some nice improvements overall. Sequential write latencies dropped from 5.8 seconds down to 0.38 seconds and throughput more than doubled: 9 MiB/s vs 23 MiB/s!  Sequential reads latencies increased a bit, but throughput again more than doubled: 12 MiB/s vs 27 MiB/s!  Random seeks also saw a nice improvement from 7.6s down to 4.9s.


Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card 2.6.35-g6d019da-dirty2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
SD Card 2.6.40.3-2.02.fc13.armv7l.omap2G2265186242544333999196261615.61
Latency2793ms29928ms39111ms50446us665ms76023ms
USB HD 2.6.35-g6d019da-dirty2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms
USB HD 2.6.40.3-2.02.fc13.armv7l.omap2G399923519261111815327992731418150.612
Latency322ms383ms341ms54567us697ms4910ms

I also tried a simple test with hdparm as shown on lwn.net:

[root@fedora-arm ~]# uname -r
2.6.40.3-2.02.fc13.armv7l.omap

[root@fedora-arm ~]# hdparm -t /dev/mmcblk0p4

/dev/mmcblk0p4:
 Timing buffered disk reads:   50 MB in  3.10 seconds =  16.11 MB/sec

[root@fedora-arm ~]# hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   66 MB in  3.05 seconds =  21.64 MB/sec

Compared to the older kerrnel:

[root@fedora-arm ~]# uname -r
2.6.35-g6d019da-dirty

[root@fedora-arm ~]# hdparm -t /dev/mmcblk0p4

/dev/mmcblk0p4:
 Timing buffered disk reads:   50 MB in  3.09 seconds =  16.20 MB/sec

[root@fedora-arm ~]# hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   26 MB in  3.04 seconds =   8.56 MB/sec

The SD card's performance is about the same, but the USB hard drive is much faster with the new patch!

Thursday, July 14, 2011

Persistent MAC Address on the PandaBoard

The PandaBoard doesn't have an EEPROM to store a MAC address for the on-board NIC, and as a result, a new random MAC address is generated on boot.  This can be a problem for DHCP lease pools, firewall rules, and more.

I mentioned in an earlier post that Ubuntu 11.04 is somehow keeping the NIC's MAC address persistent despite the lack of an EEPROM.  I investigated a bit and found it was a simple combination of (1) setting a MAC address in the NetworkManager settings and (2) this driver patch going into the latest upstream kernels: smsc95xx: generate random MAC address once, not every ifup.

Back in Fedora 13, I upgraded to David Marlin's latest 2.6.39-1.01.fc13.armv7l kernel from http://people.redhat.com/dmarlin/packages/FedoraArm/ — which includes the above patch — and booted the PandaBoard.

The first thing I noticed was that the NIC is named eth0 now instead of usb0 (that's a nice touch) and, as expected, it has a new random MAC address:
[root@fedora-arm ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 4E:19:68:14:3C:F9
          BROADCAST MULTICAST  MTU:1492  Metric:1
          ...

I changed the MAC address using the standard tools and it appeared to stick:
[root@fedora-arm ~]# ifconfig eth0 hw ether CA:BC:15:4D:B4:49
[root@fedora-arm ~]# ifconfig eth0
eth0      Link encap:Ethernet HWaddr CA:BC:15:4D:B4:49
          BROADCAST MULTICAST MTU:1492 Metric:1
          ...

I wanted to make this survive a reboot, so I looked at the /etc/sysconfig/network-scripts/ifup-eth script and found this chunk:
# this isn't the same as the MAC in the configuration filename. It is
# available as a configuration option in the config file, forcing the kernel
# to think an ethernet card has a different MAC address than it really has.
if [ -n "${MACADDR}" ]; then
   ip link set dev ${DEVICE} address ${MACADDR}
fi

(The ip command is preferred over the older ifconfig command.)

So, I edited /etc/sysconfig/network-scripts/ifcfg-eth0 and removed the HWADDR variable and set MACADDR instead:
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
MACADDR=CA:BC:15:4D:B4:49
TYPE=Ethernet
IPADDR=192.168.1.2
NETMASK=255.255.255.0
GATEWAY=192.168.1.1

I rebooted the PandaBoard and it worked:
                Welcome to Fedora
                Press 'I' to enter interactive startup.
Starting udev: [  OK  ]
Setting hostname fedora-arm:  [  OK  ]
...
fedora-arm login: root
Password:
Last login: Thu Jul 14 16:48:49 on console
[root@fedora-arm ~]# ifconfig eth0 | grep HWaddr
eth0      Link encap:Ethernet HWaddr CA:BC:15:4D:B4:49

Hooray for persistent MAC addresses!

Saturday, July 9, 2011

PandaBoard Display Resolution

I wanted to return to the persistent-MAC-address problem and compare Fedora vs Ubuntu, so I swapped SD cards to Ubuntu and powered up the PandaBoard.  To my surprise, Ubuntu set the display resolution to 640x480.  This Samsung 2233SW monitor supports 1920x1080, so why is it so low?  I checked the Monitor Preferences and it only listed 640x480 as an option, and I fired up a terminal and 'xrand -q' reported the same.  I know the last time I booted into Ubuntu in May, it was much higher than 640x480.  What changed?

I tried adding a new mode with 'xrandr  --new-mode ...' but that didn't work.

I found the Omappedia Bootargs for Enabling Display page and tried modifying my /boot/boot.script (and running flash-kernel) to add omapfb.mode=dvi:1920x1080MR-24@60 to the kernel command line arguments, but that didn't work either.

A few hours and many unsuccessful attempts later, I looked around the room and saw an older Dell 1707FPc monitor sitting in the corner.  Maybe I was using that monitor in May?  It only supports 1280x1024, so maybe it will work better?  I cleaned up my boot.script and swapped monitors and voila, it came up with 1280x1024!

I guess the PandaBoard doesn't like this Samsung display for some reason.

Thursday, June 2, 2011

Storage Speed on the PandaBoard Revisited

A recent thread on the Fedora arm mailing list had me revisit the storage speed tests I ran recently.  DJ Delorie found that his USB storage speed more than doubled if he pinged the PandaBoard from another host.  The PandaBoard's ethernet is part of the USB smsc95xx chipset, so it makes sense that they're related.

With that in mind, I ran the bonnie++ tests again while pinging the PandaBoard (from the same system running minicom to access the serial console of the PandaBoard). The first row is copied-and-pasted from the last test. The second row holds the new results.

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
USB HD2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms
USB HD with ping2G38991903839970914364992494721162.711
Latency220ms1714ms1319ms62165us16327us1542ms

That's amazing!  Sequential Writes more than doubled: from 9017 K/sec to 19038 K/sec.  And Sequential Reads doubled: from 12357 K/sec to 24947 K/sec!

There seems to be a bug somewhere, maybe in the smsc95xx driver?  If this can be fixed, using a USB hard drive could greatly improve the storage speeds on the PandaBoard.

Saturday, May 21, 2011

Storage Speed on the PandaBoard

In my experience so far with the PandaBoard, I noticed that the system feels sluggish using the SD card.  I'm using a Class 6 SD card which means it can sustain 6 MB/sec writes.  The USB 2.0 bus is capable of a max 480 Mbps, or 60 MB/sec; in practice, USB tops out at around 30 MB/sec due to CPU overhead.  Still, that's quite a bit faster than the 6 MB/sec the SD card can do.  With that in mind, I wanted to compare the performance of the SD card against a USB 2.0 hard drive.

The bonnie++ tool is great for testing storage performance, so I installed it with a quick yum install bonnie++ and ran it against both the SD card and the hard drive.  I was using the 2.6.35-g6d019da-dirty kernel.

It's generally a good idea to store the output of bonnie++ on a different drive from the one being tested so as not to interfere with the results, so first create a RAM-disk:

[root@fedora-arm ~]# mkdir /bonnie
[root@fedora-arm ~]# mount -t tmpfs none /bonnie

The SD card holds the root filesystem, so I tested it first; note that you have to run bonnie++ as a non-root user.

[jbastian@fedora-arm ~]$ bonnie++ -d /tmp -n 0 -m "SD Card" -q 1>>/bonnie/bonnie.csv 2>/dev/null

It takes a while for bonnie++ to do its thing, so I fired it up and grabbed a bucket of popcorn and a movie.  When it finished, I ran it again on the USB hard drive: just change /tmp to /mnt/usb.  The output is just appended to the bonnie.csv file.

[jbastian@fedora-arm ~]$ bonnie++ -d /mnt/usb -n 0 -m "USB HD" -q 1>>/bonnie/bonnie.csv 2>/dev/null

The output is CSV, but bonnie++ includes a tool to convert it to HTML to make it more readable:

[jbastian@fedora-arm ~]$ bon_csv2html /bonnie/bonnie.csv > bonnie.html

I used tidy -i to reformat the HTML a bit so I could remove the columns for the file creation tests (which I did not run), and the results look like:

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
USB HD2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms

As I expected, the SD card clearly wins with random seek times: 663ms vs 7643ms.  This is one of the major advantages of flash storage over traditional spinning magnetic media since flash does not have to move a head over the disk and then wait for the desired sector to spin under the head.  It's interesting, though, that even with the significantly slower seek times, the hard drive managed to squeeze out 93.9 seeks/sec compared to only 27.4 seeks/sec on the SD card.

The throughput results are more interesting. On sequential output, the hard drive is the winner with 9017 K/sec compared to the SD card's 3125 K/sec.  Writing data is one of the weaknesses of flash storage since it has to re-write an entire block if you need to flip a bit from a 0 to 1. For sequential input, the SD card managed a respectable 17598 K/sec vs the hard drive's 12357 K/sec.

Overall, the SD card performs nicely on large sequential reads, but the USB hard drive seems to outperform it in general usage.  Unfortunately, neither drive came close to the 30 MB/sec I was hoping for.

Just for fun, I also ran zcav, another tool provided by bonnie++ to test the read rates over a hard drive.  On a normal hard drive, the read rates will be faster on the outer edge of the disk since the outer tracks have a longer circumference and thus have more sectors compared to the inner tracks, thus more sectors pass under the head in one revolution of the disk on the outer tracks.

On an SD card, the rates should be constant since there is no spinning disk.

The zcav needs access to the raw device, so I'll run it as root.  Again, store the zcav output in the RAM disk so it doesn't interfere with the testing.  I only tested the first 8 GB of the hard drive since the SD card is only 8 GB in size, and also because testing all 200 GB would take all day!

[root@fedora-arm ~]# zcav -l /bonnie/zcav-sdcard.out /dev/mmcblk0
[root@fedora-arm ~]# zcav -r 8192 -l /bonnie/zcav-usbhd.out -f /dev/sda

The output is just a series of numbers in three columns:
#block offset (GiB), MiB/s, time
0.00 14.59 17.547
0.25 16.10 15.897
0.50 15.09 16.966
...

I think this will be easier to make sense of as a graph, so gnuplot to the rescue:

[jbastian@fedora-arm ~]$ gnuplot
...
gnuplot> set terminal png
Terminal type set to 'png'
Options are 'nocrop font /usr/share/fonts/dejavu/DejaVuSans.ttf 12 size 640,480 '
gnuplot> set output "zcav_graph.png"
gnuplot> set title "ZCAV"
gnuplot> set xlabel "Block Offset (GiB)"
gnuplot> set ylabel "MiB/sec"
gnuplot> set yrange [0:20]
gnuplot> plot "zcav-sdcard.out" using 1:2 title 'SD Card' with lines, \
>             "zcav-usbhd.out" using 1:2 title 'USB HD' with lines
gnuplot> quit

And the graph:

The SD card seems to be reading at about 15 MiB/sec and the USB hard drive at about 10 MiB/sec.  This is a little below the first bonnie++ run showing 17 Mib/sec and 12 MiB/sec respectively, but the SD card is still consistently faster than the USB hard drive at sequential reads.

For comparison, I ran bonnie++ on a desktop system (Core 2 Quad @ 2.66 GHz, 6 GiB RAM, 320 GiB SATA hard drive) running Red Hat Enterprise Linux 6.1 and it had a nice 58 MiB/sec for sequential writes, 72 MiB/sec for sequential reads, and a 1066ms latency for random seeks.  Here's the full set of numbers compared to the PandaBoard:

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
USB HD2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms
Desktop SATA11G578975885111256554263694723244176.91
Latency13998us3126ms411ms25294us862ms1066ms

My conclusion: I will continue using the SD card to hold the root filesystem.  I was hoping to get a performance boost out of a hard drive, but I think the USB bus is the bottleneck for the hard drive.  Hopefully the PandaBoard 2.0 (or whatever it will be called) will have a SATA port on it!

Monday, May 9, 2011

Ubuntu on the PandaBoard

Once I was in the mood for trying other operating systems, I thought I'd try Ubuntu for ARM and see how it compared to Fedora and Android.

I downloaded the "Texas Instruments OMAP4 preinstalled netbook image for OMAP4 boards" of Natty Narwhal (11.04) from http://cdimage.ubuntu.com/releases/11.04/release/

Installation was much simpler than Fedora or Android (/dev/mmcblk0 was my SD card); using sudo or running as root:
gunzip -c ubuntu-11.04-preinstalled-netbook-armel+omap4.img.gz | dd bs=4M of=/dev/mmcblk0
sync

I moved the SD card over to the PandaBoard and turned on the power and that was it!  Wow.

It took a long time to boot due to the slow SD card (again, a future post), but eventually it was up and running and I was browsing the web with Firefox.

X-Windows was just using the basic frame buffer driver so the graphics were not very fast, but it was certainly usable for basic web browsing.  I'll have to try the accelerated drivers later.

Some nice touches:

  1. The disk image included the two partitions necessary for the PandaBoard to boot.
  2. The image actually included a kernel (the Fedora images require you to build your own kernel or download one from omappedia.org or pandaboard.org)
  3. During the first boot, it resized the ext3 partition to fill up the SD card; the image had a 2.4 GiB partition for the root filesystem, but it expanded and filled up my 8 GiB SD card.
  4. The NIC has a fixed MAC address without passing a kernel command line argument.  I need to figure out how it did this.
  5. If you need to make changes to the kernel boot arguments, there's a flash-kernel script that updates the boot.scr file for you.  (This reminds of me of running lilo way back when to update the boot process on x86.)

Android on the PandaBoard

For some weekend fun, I thought I'd check out Android on the PandaBoard (a.k.a. "Pandroid", heh) and see how Android compared to my webOS phone (a Palm Pre).

The "installation" of Android was roughly equivalent to the Fedora 13 experience:
  1. Create a small FAT partition and a larger ext3 partition on the SD card
  2. Download Android (I grabbed L27.10.2-P1_pandroid_rls.tar.bz2)
  3. Copy the bootloader/* files to the boot partition (MLO, u-boot.bin, and uImage)
  4. Copy the myfs/* files to the root filesystem partition
  5. Unmount the SD card and move it to the PandaBoard
Booting Android differed a bit from Fedora:
  1. Connect USB2Serial cable and fire up minicom
  2. Attach the power and...  it gives an error:
    booti: cannot find 'boot' partition
    Fastboot entered...
After searching the web for a while, I found a different set of boot arguments in the L27.10.2-P1 release notes to try (as opposed to this set of arguments):
setenv bootargs 'console=ttyO2,115200n8 androidboot.console=ttyO2 mem=456M@0x80000000 mem=512M@0xA0000000 root=/dev/mmcblk0p2 rw rootdelay=2 init=/init vram="32M" omapfb.vram=0:16M,1:16M consoleblank=0'
setenv bootcmd 'mmcinit 0;fatload mmc 0 0x80000000 uImage; bootm 0x80000000'
boot

Trying it again:
  1. Hit the power reset button, then hit a key in minicom to interrupt the autoboot sequence
  2. At the 'PANDA #' prompt, enter (copy and paste, actually) the above setenv commands for bootargs and bootcmd and then boot
It took a while to load (due to the slow SD card, more on this in a future post), but eventually Android popped up on my LCD monitor (connected with an HDMI-A to DVI-D cable).  The touch screen normally used on phone is simulated with a USB mouse and a regular cursor (left button is a finger tap, middle button is the menu, and right button is back), however, the cursor for the mouse was really slow and jumped around to try to keep up with mouse movements.
    A bit more reading and I learned that the Android image does not include accelerated drivers for the PandaBoard's PowerVR SGX540 chipset.  Fortunately, drivers are just another quick download from Texas Instruments.  I grabbed the drivers for L27.10.2-P1 image and copied them to the SD card (using my regular Linux laptop again), booted once more, and the graphics were nice and snappy!

    It seemed to have trouble bringing up the network (I think it needs more drivers), so I couldn't actually do a whole lot except fire up the basic apps that came with Android, e.g., calculator, address book, nothing too exciting, but the core OS worked!

    Friday, May 6, 2011

    PandaBoard Networking with Fedora 13

    With Fedora 13 booting on the PandaBoard, the next step is to bring up the network so I can install more useful software with yum.

    The default configuration for the usb0 NIC is for DHCP:

    [root@fedora-arm ~]# cat /etc/sysconfig/network-scripts/ifcfg-usb0
    DEVICE=usb0
    BOOTPROTO=dhcp
    ONBOOT=no

    But DHCP isn't working for me, it just hangs:

    [root@fedora-arm ~]# ifup usb0

    Determining IP information for usb0...Unknown HZ value! (94) Assume 100.

    I think the Unknown HZ value message is unrelated to the DHCP problem.

    I'm not sure what's going on with DHCP, so I will just try a static IP address for now. First, I need the MAC address of the interface:

    [root@fedora-arm ~]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr 4A:0B:B3:2A:3E:67
              ...

    With the MAC address, I can create a config file for the NIC:

    [root@fedora-arm ~]# cat /etc/sysconfig/network-scripts/ifcfg-usb0
    DEVICE=usb0
    BOOTPROTO=static
    ONBOOT=yes
    HWADDR=4A:0B:B3:2A:3E:67
    TYPE=Ethernet
    IPADDR=192.168.1.111
    NETMASK=255.255.255.0
    GATEWAY=192.168.1.254

    Bring up the interface:

    [root@fedora-arm network-scripts]# ifup usb0
    [root@fedora-arm network-scripts]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr 8A:F4:D3:E9:EB:1A
              inet addr:192.168.1.111 Bcast:192.168.1.255 Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST MTU:1492 Metric:1
              ...

    And ping the router:

    [root@fedora-arm network-scripts]# ping 192.168.1.254
    PING 192.168.1.254 (192.168.1.254) 56(84) bytes of data.
    64 bytes from 192.168.1.254: icmp_seq=1 ttl=255 time=6.80 ms
    64 bytes from 192.168.1.254: icmp_seq=2 ttl=255 time=0.794 ms
    ^C

    Networking is up and running!

    Finally, as a sanity check, I rebooted to make sure networking still comes up okay.

    Wait, what's this?

    [root@fedora-arm network-scripts]# reboot
    ...
    Bringing up loopback interface: [ OK ]
    Bringing up interface usb0: Device usb0 has different MAC address than expected
    , ignoring.
    [FAILED]
    ...

    The MAC address is different? How is that possible?

    Fedora release 13 (Goddard)
    Kernel 2.6.35-g6d019da-dirty on an armv7l (console)

    fedora-arm login: root
    Password:
    Last login: Sat Jan 1 00:00:17 on console
    [root@fedora-arm ~]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr CA:70:56:5B:23:7A
              BROADCAST MULTICAST MTU:1492 Metric:1
              ...

    Wow, that HWaddr is nothing like the address on the previous boot!

    A quick Google search later and I learned the NIC on the PandaBoard does not have a MAC address stored in the EEPROM because it doesn't have an EEPROM!  Thus, it generates a new random MAC address on every boot. Argh.
    http://lwn.net/Articles/435894/
    https://bugs.launchpad.net/ubuntu/+source/linux-ti-omap4/+bug/673504

    The workaround for now is to pass in a MAC address of your choosing on the kernel command line, for example, smsc95xx.macaddr=01:02:03:04:05:06

    So, I powered-down the the PandaBoard and moved the SD card over to my regular Linux workstation to create a new boot.scr file.

    First, I had to yum install uboot-tools to get the mkimage command. Next, I created a boot.cmd file on the first partition of the SD card:

    [root@localhost ~]# mount /dev/mmcblk0p2 /mnt/panda-rootfs
    [root@localhost ~]# mount /dev/mmcblk0p1 /mnt/panda-rootfs/boot
    [root@localhost ~]# cd /mnt/panda-rootfs/boot
    [root@localhost boot]# vi boot.cmd
    [root@localhost boot]# cat boot.cmd
    fatload mmc 0:1 0x80000000 uImage
    setenv bootargs 'ro elevator=noop vram=32M root=/dev/mmcblk0p2 fixrtc console=ttyO2,115200 mem=460M@0x80000000 mem=460M@0xA0000000 smsc95xx.macaddr=CA:BC:15:4D:B4:49'
    bootm 0x80000000

    And then I converted it into the boot script format with the mkimage command:

    [root@localhost boot]# mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n 'Pandaboard boot script' -d boot.cmd boot.scr

    Unmount the SD card and move it back to the Pandaboard and boot it.

    fedora-arm login: root
    Password:
    Last login: Fri May 6 11:03:29 from 192.168.1.76
    [root@fedora-arm ~]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr CA:BC:15:4D:B4:49
              ...

    The MAC address looks good! Let's update the ifcfg-usb0 and set the HWADDR line again.

    [root@fedora-arm ~]# vi /etc/sysconfig/network-scripts/ifcfg-usb0
    [root@fedora-arm ~]# grep HWADDR /etc/sysconfig/network-scripts/ifcfg-usb0
    HWADDR=CA:BC:15:4D:B4:49

    And one more reboot just-to-be-sure:

    ...
    Bringing up loopback interface: [ OK ]
    Bringing up interface usb0: [ OK ]
    ...
    Fedora release 13 (Goddard)
    Kernel 2.6.35-g6d019da-dirty on an armv7l (console)

    fedora-arm login: root
    Password:
    Last login: Fri May 6 11:55:45 on console
    [root@fedora-arm ~]# ifconfig usb0
    usb0      Link encap:Ethernet HWaddr CA:BC:15:4D:B4:49
              inet addr:192.168.1.111 Bcast:192.168.1.255 Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST MTU:1492 Metric:1
              ...

    Yay! It works!


    One final note: in addition to the lack of EEPROM to store the MAC address, there also does not seem to be a system clock. Every time I boot, the system date is set to midnight, January 1, 2000. ntpdate and ntpd should help with that. I installed both and set the step-tickers (used by ntpdate) to:

    [root@fedora-arm ~]# cat /etc/ntp/step-tickers
    # List of servers used for initial synchronization.
    0.fedora.pool.ntp.org
    1.fedora.pool.ntp.org
    2.fedora.pool.ntp.org

    Then enable both ntpdate and ntpd with chkconfig:

    [root@fedora-arm ~]# chkconfig ntpdate on
    [root@fedora-arm ~]# chkconfig ntpd on

    And now the date should be correct after a reboot.

    [root@fedora-arm ~]# reboot

    Broadcast message from root@fedora-arm
    (/dev/console) at 12:17 ...
    ...
    Starting sshd: [ OK ]
    ntpdate: Synchronizing with time server: [ OK ]
    Starting ntpd: [ OK ]

    Fedora release 13 (Goddard)
    Kernel 2.6.35-g6d019da-dirty on an armv7l (console)

    fedora-arm login: root
    Password:
    Last login: Fri May 6 12:00:31 on console
    [root@fedora-arm ~]# date
    Fri May 6 12:18:38 CDT 2011

    That looks better!

    Thursday, April 21, 2011

    My First PandaBoard

    My PandaBoard is unboxed and ready to play work!

    There is no installer (anaconda) for Fedora on ARM systems yet, so installation means downloading an image of the root filesystem and a kernel. The root filesystem is generic for all ARM (v5 and newer) systems so it does not include a kernel.

    Download the root filesystem and kernel onto your host machine with an SD card reader.

    wget http://scotland.proximity.on.ca/fedora-arm/beta/f13/rootfs-f13-beta-2011-03-23.tar.bz2
    wget http://scotland.proximity.on.ca/arm/kernel/pandaboard/2.6.35.3/boot.zip
    wget http://scotland.proximity.on.ca/arm/kernel/pandaboard/2.6.35.3/boot_no_vid.tar.gz
    wget http://scotland.proximity.on.ca/arm/kernel/pandaboard/2.6.35.3/uImage_config

    Links from http://fedoraproject.org/wiki/Architectures/ARM/F13-ARM-Beta2

    Next, the filesystem and the kernel need to be copied to an SD(HC) card. The SD card should have a small FAT32 partition at the beginning to hold the kernel, and the rest of the card can be a regular Linux ext3 filesystem.

    After inserting the SD card in the reader, I can see from the dmesg logs the
    device name:
    $ dmesg
    ...
    [  277.436513] mmc0: new SDHC card at address 0007
    [  277.536720] mmcblk0: mmc0:0007 SD08G 7.43 GiB
    [  277.541804]  mmcblk0: p1
    ...

    This script from omappedia.org will do the trick; many thanks to the folks in #pandaboard on freenode IRC for helping me get the SD card formatted correctly!

    #!/bin/sh

    DRIVE=$1

    dd if=/dev/zero of=$DRIVE bs=1024 count=1024

    SIZE=`fdisk -l $DRIVE | grep Disk | awk '{print $5}'`

    echo DISK SIZE - $SIZE bytes

    CYLINDERS=`echo $SIZE/255/63/512 | bc`

    echo CYLINDERS - $CYLINDERS

    {
    echo ,9,0x0C,*
    echo ,,,-
    } | sfdisk -D -H 255 -S 63 -C $CYLINDERS $DRIVE

    mkfs.vfat -F 32 -n "boot" ${DRIVE}p1
    mke2fs -j -L "rootfs" ${DRIVE}p2

    NOTE: If your system shows the device as /dev/sda1 (or similar), then you will need to remove the 'p' from the last two lines, e.g.,
        mkfs.vfat -F 32 -n "boot" ${DRIVE}1

    Format the device using the above script and then take a look at it with parted:
    $ sudo ./sd-card-format.sh /dev/mmcblk0
    ...
    $ sudo parted /dev/mmcblk0 print
    Model: SD SD08G (sd/mmc)
    Disk /dev/mmcblk0: 7986MB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos

    Number  Start   End     Size    Type     File system  Flags
     1      32.3kB  74.0MB  74.0MB  primary  fat32        boot, lba
     2      74.0MB  7979MB  7904MB  primary  ext3

    That looks better!

    Okay, now I can extract the filesystem onto the SD card.  Mount the second partition onto a /mnt/panda-rootfs directory.
    $ sudo mkdir /mnt/panda-rootfs
    $ sudo mount /dev/mmcblk0p2 /mnt/panda-rootfs
    $ cd /mnt/panda-rootfs && sudo tar xf ~/Download/arm-f13-beta2/rootfs-f13-beta-2011-03-23.tar.bz2

    Finally, the kernel can be extracted into the first partition:
    $ sudo mount /dev/mmcblk0p1 /mnt/panda-rootfs/boot
    $ cd /mnt/panda-rootfs && sudo unzip ~/Download/arm-f13-beta2/kernel/pandaboard/2.6.35.3/boot.zip
    Archive:  ~/Download/arm-f13-beta2/kernel/pandaboard/2.6.35.3/boot.zip
      inflating: boot/boot.scr          
      inflating: boot/MLO               
      inflating: boot/u-boot.bin        
      inflating: boot/uImage

    A quick check to see that both partitions have some used space:
    $ df -Th /mnt/panda-rootfs /mnt/panda-rootfs/boot
    Filesystem    Type    Size  Used Avail Use% Mounted on
    /dev/mmcblk0p2
                  ext3    7.3G  549M  6.4G   8% /mnt/panda-rootfs
    /dev/mmcblk0p1
                  vfat    120M  2.8M  117M   3% /mnt/panda-rootfs/boot

    There's a known bug with launching a getty on the PandaBoard.  For now, edit the /etc/rc.d/rc.local file and uncomment the "horrible hack":
    $ sudo vim /mnt/panda-rootfs/etc/rc.d/rc.local
    # horrible hack to respawn serial console
    while true
    do
                    /sbin/agetty -L 115200 console vt100
    done

    Finally, unmount and remove the SD card and transfer it to the PandaBoard.
    $ cd && sudo umount /mnt/panda-rootfs/boot && sudo umount /mnt/panda-rootfs && sudo sync

    The PandaBoard is set up to use a serial console, so attach a serial cable (or a USB-to-serial cable with a Prolific PL2303 chip) and fire up minicom. I'm using a USB-to-serial cable, so I ran 'minicom -s' to configure it for /dev/ttyUSB0.  With minicom running, I finally attached the power cable to the PandaBoard and watched it boot.

    Texas Instruments X-Loader 1.4.4ss (Mar  1 2011 - 23:55:13)
    Reading boot sector
    Loading u-boot.bin from mmc


    U-Boot 2011.03-rc1 (Feb 09 2011 - 01:46:42)

    CPU  : OMAP4430
    Board: OMAP4 Panda
    I2C:   ready
    DRAM:  1 GiB
    MMC:   OMAP SD/MMC: 0
    Using default environment

    In:    serial
    Out:   serial
    Err:   serial
    Hit any key to stop autoboot:  0
    reading boot.scr

    ...
    ...
    ...

    Fedora release 13 (Goddard)
    Kernel 2.6.35-g6d019da-dirty on an armv7l (console)

    fedora-arm login: root
    Password:
    Last login: Sat Jan  1 00:41:17 on console
    [root@fedora-arm ~]# cat /proc/cpuinfo
    Processor       : ARMv7 Processor rev 2 (v7l)
    processor       : 0
    BogoMIPS        : 1195.29

    processor       : 1
    BogoMIPS        : 1166.88

    Features        : swp half thumb fastmult vfp edsp thumbee neon vfpv3
    CPU implementer : 0x41
    CPU architecture: 7
    CPU variant     : 0x1
    CPU part        : 0xc09
    CPU revision    : 2

    Hardware        : OMAP4430 Panda Board
    Revision        : 0020
    Serial          : 0000000000000000

    It works!

    Next up, networking...

    A Greener Home Server

    I have an old Mac Mini G4 that I've been using as my always-on home server, but it draws a fair amount of power for a system that's idle most of the time (32 watts, up to 85 watts when busy) , so I decided to look into running an ARM system (they typically use less than 5 watts).

    After considering the SheevaPlug, the GuruPlug, the BeagleBoard, the PandaBoard, and even the pink PogoPlug, I decided to order a PandaBoard since it has a dual-core OMAP4 at 1 GHz with 1 GiB of RAM.  It was on backorder for a few days, but it finally arrived!

    Next step: get Fedora up and running on it with the help of