Saturday, November 26, 2011

A PandaBoard Desktop with Fedora 13

A serial port console, while nice and simple to work on, is a little lacking for eye candy, so today I installed X Windows and Xfce to run a desktop on my PandaBoard.

The installation was the easy part:
yum -y groupinstall Base 'X Window System' 'XFCE' 'GNOME Desktop Environment'

And then it was hurry-up-and-wait while it installed 421 packages.

When it finished, I rebooted it and found a few problems that I had to fix.  The default resolution for the display is a safe 640x480, so I had to add a kernel command line option to make it 1280x1024: omapfb.mode=dvi:1280x1024MR-24@60.  I also added another console=tty0 option so both the serial port and the display could be consoles.  The last console parameter is the primary console and for now I kept the serial port as primary for easier debugging.

[root@fedora-arm ~]# mount /dev/mmcblk0p1 /mnt/boot
[root@fedora-arm ~]# cd /mnt/boot
[root@fedora-arm boot]# vim boot.cmd
[root@fedora-arm boot]# cat boot.cmd
setenv mpurate 800
setenv bootargs 'console=tty0 console=ttyO2,115200n8 ro rootwait root=/dev/sda1 mpurate=${mpurate} init=/sbin/init earlyprintk rd_NO_PLYMOUTH selinux=0 omapfb.mode=dvi:1280x1024MR-24@60'
setenv bootcmd 'mmc rescan 0; mmc init; fatload mmc 0:1 0x80300000 uImage; fatload mmc 0:1 0x81600000 uInitrd; bootm 0x80300000 0x81600000'
boot
[root@fedora-arm boot]# mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n 'Pandaboard Boot Script' -d boot.cmd boot.scr
Image Name: Pandaboard Boot Script
Created: Sat Nov 26 14:22:00 2011
Image Type: ARM Linux Script (uncompressed)
Data Size: 359 Bytes = 0.35 kB = 0.00 MB
Load Address: 00000000
Entry Point: 00000000
Contents:
Image 0: 351 Bytes = 0.34 kB = 0.00 MB
[root@fedora-arm boot]# cd
[root@fedora-arm ~]# umount /mnt/boot

I then updated /etc/inittab to make runlevel 5 the default and rebooted the PandaBoard again.

I noticed ntpdate was failing to set the clock.  I tried running it manually and noticed it couldn't resolve the hostname 0.fedora.pool.ntp.org.  A quick peek at /etc/resolv.conf showed that NetworkManager had re-written the file with nothing but comments because I have /etc/sysconfig/network-scripts/ifcfg-eth0 set up for a static IP address for now.  So I disabled NetworkManager with chkconfig and cleaned up resolv.conf and rebooted once more.

This time it almost worked.  It went through all the boot scripts cleanly, but it got stuck when it should have started X11 and gdm.  Fortunately the serial console still worked, so I logged in and discovered that the /etc/rc.d/rc script was still running and starting up runlevel 5, but it didn't seem to be doing anything.  I then found it was waiting for /etc/rc.d/rc5.d/S99local to finish (aka /etc/rc.d/rc.local).  I looked at this script and remembered the "ugly hack" I had to set up back in April to run agetty on the serial port.  It was running in a never-ending loop which means it will never finish and the init scripts will never start X11.

The Fedora ARM page no longer mentions this ugly hack, so I guess the bug has been fixed and I can use a normal upstart job to manage agetty on the serial port.  I commented out the hack in the rc.local script and created a new /etc/init/serial-ttyO2.conf job:

[root@fedora-arm ~]# cat /etc/rc.d/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# horrible hack to respawn serial console
###while true
###do
###    /sbin/agetty -L 115200 console vt100
###done

[root@fedora-arm ~]# cat /etc/init/serial-ttyO2.conf
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [016]

respawn
pre-start exec /sbin/securetty ttyO2
exec /sbin/agetty /dev/ttyO2 115200 vt100-nav


[root@fedora-arm ~]# initctl start serial-ttyO2
serial-ttyO2 start/running, process 20245

One final reboot and ... Hooray!  It ran through the firstboot setup and then launched X11 and gdm.  I logged in to Xfce and launched Firefox and a Terminal.  I now have a completely silent desktop!

Monday, November 14, 2011

System Clock on the PandaBoard

A quick way to find the physical device backing a filesystem is to use the 'df' command.  (LVM, software-RAID, LUKS, and other layers can make this more complicated.)  I did this on my PandaBoard and was a little surprised at the answer:

[root@fedora-arm ~]# df -Th /
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/root      nfs    7.4G  1.8G  5.3G  25% /

Why is it NFS-mounting a USB flash drive?  And /dev/root doesn't tell me what the device name is.  I was expecting to see something like sda1 or mmcblk0p1.

I thought that was weird, so I edited /etc/fstab and changed the device and filesystem type:
[root@fedora-arm ~]# diff -u /etc/fstab.ORIG /etc/fstab
--- /etc/fstab.ORIG        2011-11-14 10:09:40.000000000 -0600
+++ /etc/fstab  2011-11-14 12:48:58.000000000 -0600
@@ -1,4 +1,4 @@
-/dev/root               /                       nfs     defaults        1 1
+/dev/sda1               /                       ext3    defaults        1 1

And then I rebooted my PandaBoard and quickly learned why it's set to NFS:
                Welcome to Fedora
                Press 'I' to enter interactive startup.
Starting udev: [  OK  ]
Setting hostname fedora-arm:  [  OK  ]
Setting up Logical Volume Management:   No volume groups found
[  OK  ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
fedoraroot: Superblock last mount time (Mon Nov 14 11:19:35 2011,
        now = Sat Jan  1 00:00:01 2000) is in the future.


fedoraroot: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
[FAILED]

There's no battery-backed clock on the PandaBoard so the clock gets reset to 2000-01-01 00:00:00 every time it's power-cycled, and this causes the Fedora boot scripts to get confused and force a filesystem check.

Apparently mounting the root filesystem as NFS is one way to workaround  this problem.

I mentioned this in #fedora-arm on freenode and Chris Tyler suggested modifying the boot scripts to set the clock to 1 minute past the previous write time of the root filesystem.

Here's what I came up with:
--- /etc/rc.d/rc.sysinit.ORIG    2011-03-08 15:51:46.000000000 -0600
+++ /etc/rc.d/rc.sysinit    2011-11-14 12:44:28.441405990 -0600
@@ -577,6 +577,15 @@
 fi
 if [ -z "$fastboot" -a "$READONLY" != "yes" ]; then

+        # set the clock on the PandaBoard to 1 minute after last write
+        # time of root filesystem to avoid "last mount is in the future"
+        echo "Current system clock is $(date)"
+        rootdev=$(egrep '\s/\s' /etc/fstab | awk '{print $1}')
+        lastwrite=$(dumpe2fs -h $rootdev 2>/dev/null | \
+                    grep "Last write time" | sed -e 's/^[^:]*:\s*//')
+        echo "Setting system clock to 1 minute after $lastwrite"
+        date -s "$lastwrite + 1 minute"
+
         STRING=$"Checking filesystems"
         echo $STRING
         fsck -T -t noopts=_netdev -A $fsckoptions

Notice the new messages (red text) during boot:
                Welcome to Fedora
                Press 'I' to enter interactive startup.
Starting udev: [  OK  ]
Setting hostname fedora-arm:  [  OK  ]
Setting up Logical Volume Management:   No volume groups found
[  OK  ]
Current system clock is Sat Jan  1 00:00:01 CST 2000
Setting system clock to 1 minute after Mon Nov 14 12:45:36 2011
Mon Nov 14 12:46:36 CST 2011
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
fedoraroot: clean, 69076/489600 files, 487736/1957605 blocks
[  OK  ]
...


And the root filesystem and device look correct now!
[root@fedora-arm ~]# df -Th /
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3    7.4G  1.8G  5.3G  25% /

Note that I do have ntpd start later in the boot process which sets the actual correct time.

Wednesday, September 14, 2011

SD Card Speeds

After yesterday's benchmarks revealed a possible problem with the SD card performance, Mark Salter asked me to run a few more tests to see if the DMA API patch set is hurting the SD card (but helping USB storage).

I recompiled the 2.6.40.3-2.02.fc13.armv7l.omap kernel but I disabled the DMA API patch set.  I also tested with smp (the default) and nosmp.  And just for good measure, I re-tested the 2.6.35-g6d019da-dirty kernel because the last time I tested 2.6.35 was in May and I've updated a few packages since then.

It does not appear that the DMA API patch set is to blame for the slower SD card performance.  The random seek times were longer than 76000ms for all variants of 2.6.40.3 compared to just 664ms on 2.6.35.  The sequential reads were fastest with 2.6.35, but today's test was quite a bit faster than May's test; I'm not sure what to make of that.  Likewise, sequential writes were also fastest with 2.6.35.

Thus, some other change between 2.6.35 and 2.6.40 must be responsible for the drop in SD card performance, but what?

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card 2.6.35 (May 2011)2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
SD Card 2.6.35 (Sept 2011)2G3991311853465444899191041627.91
Latency3900ms42424ms54958ms19929us667ms664ms
SD Card 2.6.40.3 + DMA patch + SMP2G2265186242544333999196261615.61
Latency2793ms29928ms39111ms50446us665ms76023ms
SD Card 2.6.40.3 + DMA patch + nosmp2G1958166932513334399208291317.21
Latency2858ms38268ms42161ms51148us667ms76651ms
SD Card 2.6.40.3 + no patch + SMP2G2676166932271341199195921615.71
Latency2855ms34616ms58124ms22279us666ms82254ms
SD Card 2.6.40.3 + no patch + nosmp2G1854160332395338299208941316.51
Latency5020ms33958ms29546ms51301us666ms77587ms

Tuesday, September 13, 2011

Storage Speed on the PandaBoard Revisited (Again)

Since my last post, I took a short vacation and I've been helping in the Virtual Fedora Activity Days (VFAD) to rebuild packages for Fedora 15 hardfp (hardware floating point).

Research into the slow storage speeds on the PandaBoard (and other Cortex-ARM9 systems) has continued and a patch set developed to hopefully improve the situation.  David Marlin built a 2.6.40.3-2.02.fc13.armv7l.omap kernel with the patch set, so I downloaded a copy and ran bonnie++ again.

The results were mixed for the SD card.  Sequential reads had a nice improvement, but sequential output and random seeks took a big hit.  The random seek times are particularly worrisome with an increase from 663ms to 76023ms!  Something doesn't seem right.

The USB hard drive, though, saw some nice improvements overall. Sequential write latencies dropped from 5.8 seconds down to 0.38 seconds and throughput more than doubled: 9 MiB/s vs 23 MiB/s!  Sequential reads latencies increased a bit, but throughput again more than doubled: 12 MiB/s vs 27 MiB/s!  Random seeks also saw a nice improvement from 7.6s down to 4.9s.


Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
SD Card 2.6.35-g6d019da-dirty2G4193312553371433699175981527.41
Latency1168ms33487ms32391ms44466us2628ms663ms
SD Card 2.6.40.3-2.02.fc13.armv7l.omap2G2265186242544333999196261615.61
Latency2793ms29928ms39111ms50446us665ms76023ms
USB HD 2.6.35-g6d019da-dirty2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms
USB HD 2.6.40.3-2.02.fc13.armv7l.omap2G399923519261111815327992731418150.612
Latency322ms383ms341ms54567us697ms4910ms

I also tried a simple test with hdparm as shown on lwn.net:

[root@fedora-arm ~]# uname -r
2.6.40.3-2.02.fc13.armv7l.omap

[root@fedora-arm ~]# hdparm -t /dev/mmcblk0p4

/dev/mmcblk0p4:
 Timing buffered disk reads:   50 MB in  3.10 seconds =  16.11 MB/sec

[root@fedora-arm ~]# hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   66 MB in  3.05 seconds =  21.64 MB/sec

Compared to the older kerrnel:

[root@fedora-arm ~]# uname -r
2.6.35-g6d019da-dirty

[root@fedora-arm ~]# hdparm -t /dev/mmcblk0p4

/dev/mmcblk0p4:
 Timing buffered disk reads:   50 MB in  3.09 seconds =  16.20 MB/sec

[root@fedora-arm ~]# hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads:   26 MB in  3.04 seconds =   8.56 MB/sec

The SD card's performance is about the same, but the USB hard drive is much faster with the new patch!

Thursday, July 14, 2011

Persistent MAC Address on the PandaBoard

The PandaBoard doesn't have an EEPROM to store a MAC address for the on-board NIC, and as a result, a new random MAC address is generated on boot.  This can be a problem for DHCP lease pools, firewall rules, and more.

I mentioned in an earlier post that Ubuntu 11.04 is somehow keeping the NIC's MAC address persistent despite the lack of an EEPROM.  I investigated a bit and found it was a simple combination of (1) setting a MAC address in the NetworkManager settings and (2) this driver patch going into the latest upstream kernels: smsc95xx: generate random MAC address once, not every ifup.

Back in Fedora 13, I upgraded to David Marlin's latest 2.6.39-1.01.fc13.armv7l kernel from http://people.redhat.com/dmarlin/packages/FedoraArm/ — which includes the above patch — and booted the PandaBoard.

The first thing I noticed was that the NIC is named eth0 now instead of usb0 (that's a nice touch) and, as expected, it has a new random MAC address:
[root@fedora-arm ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 4E:19:68:14:3C:F9
          BROADCAST MULTICAST  MTU:1492  Metric:1
          ...

I changed the MAC address using the standard tools and it appeared to stick:
[root@fedora-arm ~]# ifconfig eth0 hw ether CA:BC:15:4D:B4:49
[root@fedora-arm ~]# ifconfig eth0
eth0      Link encap:Ethernet HWaddr CA:BC:15:4D:B4:49
          BROADCAST MULTICAST MTU:1492 Metric:1
          ...

I wanted to make this survive a reboot, so I looked at the /etc/sysconfig/network-scripts/ifup-eth script and found this chunk:
# this isn't the same as the MAC in the configuration filename. It is
# available as a configuration option in the config file, forcing the kernel
# to think an ethernet card has a different MAC address than it really has.
if [ -n "${MACADDR}" ]; then
   ip link set dev ${DEVICE} address ${MACADDR}
fi

(The ip command is preferred over the older ifconfig command.)

So, I edited /etc/sysconfig/network-scripts/ifcfg-eth0 and removed the HWADDR variable and set MACADDR instead:
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
MACADDR=CA:BC:15:4D:B4:49
TYPE=Ethernet
IPADDR=192.168.1.2
NETMASK=255.255.255.0
GATEWAY=192.168.1.1

I rebooted the PandaBoard and it worked:
                Welcome to Fedora
                Press 'I' to enter interactive startup.
Starting udev: [  OK  ]
Setting hostname fedora-arm:  [  OK  ]
...
fedora-arm login: root
Password:
Last login: Thu Jul 14 16:48:49 on console
[root@fedora-arm ~]# ifconfig eth0 | grep HWaddr
eth0      Link encap:Ethernet HWaddr CA:BC:15:4D:B4:49

Hooray for persistent MAC addresses!

Saturday, July 9, 2011

PandaBoard Display Resolution

I wanted to return to the persistent-MAC-address problem and compare Fedora vs Ubuntu, so I swapped SD cards to Ubuntu and powered up the PandaBoard.  To my surprise, Ubuntu set the display resolution to 640x480.  This Samsung 2233SW monitor supports 1920x1080, so why is it so low?  I checked the Monitor Preferences and it only listed 640x480 as an option, and I fired up a terminal and 'xrand -q' reported the same.  I know the last time I booted into Ubuntu in May, it was much higher than 640x480.  What changed?

I tried adding a new mode with 'xrandr  --new-mode ...' but that didn't work.

I found the Omappedia Bootargs for Enabling Display page and tried modifying my /boot/boot.script (and running flash-kernel) to add omapfb.mode=dvi:1920x1080MR-24@60 to the kernel command line arguments, but that didn't work either.

A few hours and many unsuccessful attempts later, I looked around the room and saw an older Dell 1707FPc monitor sitting in the corner.  Maybe I was using that monitor in May?  It only supports 1280x1024, so maybe it will work better?  I cleaned up my boot.script and swapped monitors and voila, it came up with 1280x1024!

I guess the PandaBoard doesn't like this Samsung display for some reason.

Thursday, June 2, 2011

Storage Speed on the PandaBoard Revisited

A recent thread on the Fedora arm mailing list had me revisit the storage speed tests I ran recently.  DJ Delorie found that his USB storage speed more than doubled if he pinged the PandaBoard from another host.  The PandaBoard's ethernet is part of the USB smsc95xx chipset, so it makes sense that they're related.

With that in mind, I ran the bonnie++ tests again while pinging the PandaBoard (from the same system running minicom to access the serial console of the PandaBoard). The first row is copied-and-pasted from the last test. The second row holds the new results.

Version 1.96Sequential OutputSequential InputRandom
Seeks
SizePer CharBlockRewritePer CharBlock
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU
USB HD2G4399901716453863699912357993.96
Latency215ms5849ms5889ms52858us108ms7643ms
USB HD with ping2G38991903839970914364992494721162.711
Latency220ms1714ms1319ms62165us16327us1542ms

That's amazing!  Sequential Writes more than doubled: from 9017 K/sec to 19038 K/sec.  And Sequential Reads doubled: from 12357 K/sec to 24947 K/sec!

There seems to be a bug somewhere, maybe in the smsc95xx driver?  If this can be fixed, using a USB hard drive could greatly improve the storage speeds on the PandaBoard.