Posts

Earlier this year, I ditched my antiquated Treo 755p for the new hotness in mobile phones—the HTC EVO, an Android phone available from Sprint. Unlike my previous phone, which would last a full week in standby without recharging, the EVO gulps battery like there's no tomorrow. In fact, the limited battery life has been the primary complaint about the phone. Nobody seems to know for certain why the battery life is so poor. Maybe it's the beefy 1Ghz Snapdragon processor, the large LCD screen, or something else entirely.

I was recently traveling in a rural area, and noticed that my EVO's battery would only last halfway through the day. All mobile phones consume more battery in rural areas where they must transmit with greater power to stay in contact with faraway towers. However, the extent to which the EVO's battery life problems were exacerbated by the higher transmitting power got me thinking that perhaps at least some of the battery drain may be due to software which occasionally causes the radio to transmit.

All modern smartphones have various software components which periodically execute in the background, unbeknownst to the user, and sometimes activate the radio to chat with their buddies in the cloud. Some of this chatter may be useful for the user, such as having fresh mail delivered immediately, while other chatter may provide little or no value.

I got curious about what the phone's software was actually doing while I wasn't around, so I captured and analyzed all the network traffic from my phone over a period of four hours. I accomplished this by turning off the 3G radio and packet sniffing the Wi-Fi network. (I didn't have an easy means of sniffing the 3G traffic directly.) I have my phone syncing with my Google account, Google Latitude, a couple of mail servers, and Facebook, and this is reflected in the captured data. I did not touch the phone at all for the duration of the test, so this traffic represents only non-interactive use. Here are the main traffic categories, sorted by decreasing packet count, with local network maintenance traffic (ARP, DHCP, etc.) removed:

In total, about 1.2 MB of data was exchanged in 2277 packets: 186 KB (1270 packets) were transmitted, and 1002 KB (1007 packets) were received.

The Facebook sync seems to be the biggest data hog of these background services. Every two hours, it is downloading information about all my Facebook contacts, as well as fetching profile pictures of most of them (whether or not the profile pictures have changed recently). The sync frequency for my "Facebook for HTC Sense" program can be changed in the settings, so I changed it from every two hours to once a day. After Facebook, the biggest consumers seem to be Latitude and my mail servers. I knew what I was getting into when I configured those, so I can't complain. I didn't even realize that I had Yahoo's "Stocks" program set to sync, so I turned that off altogether. The Sprint Zone sync frequency can be changed in the Sprint Zone application itself, although I haven't found a way to limit the Sprint TV syncing. The next time I'm traveling in a rural area, I should remember to turn off the "background data" and "auto-sync" items in the sync settings.

There are a few possible flaws in using this method to predict battery consumption. The exact relationship between network activity and energy usage is not clear to me. For example, is the transmitter only active for just long enough to send a packet, or does it remain on for a bit longer in the hopes of another packet becoming available in the send queue? What would consume more energy—a burst of 100 kilobytes, or 100 short transmissions of 1 kilobyte each, spread throughout a period of time? Is it more meaningful to consider packet counts, or the total byte count? Are some of these services only transmitting because they know the network connection is Wi-Fi and not 3G?

I decided I needed a clock in my living room, and remembered I had this Matrix Orbital VFD2041 vacuum fluorescent display in my junk box, so I whipped up a few lines of C code to make it show the current time. The VFD connects to my home theater PC via its RS-232 interface and a USB RS-232 dongle.

Here's the source:

• vfdclock.tar.gz

Back in the 1990's, I predicted that all data-oriented utilities — telephone, television, radio, etc. — would eventually become Internet services. Today, the world is rapidly approaching this ideal as people ditch their POTS telephone lines for VoIP and their cable television subscriptions for streaming video. This disruptive technology has created tension in the industry as old-school telecommunications companies cling to their non-Internet services out of fear of their infrastructure turning into “dumb pipes,” while other companies are lining up to provide services over the Internet. Discussion of this disruption usually turns to the subject of bandwidth — the measure of the telecommunications infrastructure's capacity to deliver these services to users.

I became curious about exactly how much bandwidth is used by the multimedia services that I consume, so I decided to graph the bandwidth usage while using these services. I collected the data points from the SNMP network interface counter readings on my router, and compiled them into the graphs below. Each test includes a rough ~10-second lead-in/lead-out, and the data represents 1-second averages unless indicated otherwise.

I was a little surprised to see that Vonage telephone calls use so much bandwidth. I looked at my service settings on the Vonage web site, and my service is indeed configured to use “high quality” audio which uses a nominal bitrate of 90kbps in each direction. This is astonishing, since traditional POTS phone service uses a 64kbps audio channel (8Khz 8-bit sampling). It turns out that by default, Vonage does not use audio compression ^[1], and encodes audio using the raw PCM format. ^[2] I'm probably not getting much benefit from this high quality stream, since I'm using a POTS telephone which limits me to the low-quality 8Khz audio that telephone users have experienced for decades. Some day, I would love to ditch the POTS bottleneck altogether, since I've found that hi-fi audio conferencing is much more pleasant.

There's nothing too surprising about the Pandora bandwidth usage. From the looks of the graph, it's downloading each song in a quick burst at some point before the scheduled playback.

YouTube seems to be very front-heavy, filling a large buffer at the start and topping off the buffer periodically. From the length of the quiet area at the tail of the video, I'd guess that the buffer size in this test was about 78 seconds or 7.5MB.

As expected, watching YouTube video with the HD option uses more bandwidth. The buffer in this case seems to be around 123 seconds or 30MB.

The above test was performed with a Roku HD-XR set-top box streaming a feature-length film from Netflix. The bandwidth usage is extremely bursty, with the Roku box taking large gulps from the stream periodically. Note that the interval between bursts does not reveal the buffer size, since the box is only “topping off” its buffer at these times. However, from the quiet area at the tail of the graph, we can surmise a ballpark estimate of about 350 seconds or 64MB.

Because the burstiness of the first Netflix graph doesn't convey the general bandwidth usage very well, I created this second graph which uses 5-minute averages instead of 1-second averages. This graph represents the same viewing session as the first, but with the bandwidth usage smoothed out. The average bitrate is around 1.5Mbps.

Here's another bursty graph representing the viewing of a TV show on Hulu. I believe the empty intervals within the show correspond to advertisement breaks. Interestingly, it seems that Hulu buffers the advertising in advance, and does not use the network for the duration of the advertisement.

Here's the smoothed version of the Hulu graph with 5-minute averages. The average bitrate is about 694kbps, which is quite a bit more efficient than Netflix streaming. However, I think the quality was a little lower, and Hulu certainly demands a lot from my computer. My older laptop with a Centrino/Pentium-M processor cannot play Hulu video without skipping, although my newer Core 2 Duo laptop handles it fine.

All tests were performed at the Caffeinated Bitstream testing laboratories at an undisclosed location in the Rocky Mountain region.

Footnotes

1. “Vonage utilizes VoIP technology differently from its competition, in that it doesn't compress your voice, so the sound quality is superb.” http://www.vonage.com/how_vonage_works_faq/
2. “Vonage uses the standard PCM audio codec by default because people prefer it.” http://www.interesting-people.org/archives/interesting-people/200407/msg00231.html

After writing the ffremote.py script to help open links in the correct Firefox instance, I decided that it would be really nifty to develop a simple GNOME Panel applet to manage browser instances. (Again, not browser windows, but browser instances—completely separate Firefox processes running with distinct profiles and settings.)

When you click on the applet's icon, a menu is presented which lists the running browser windows and their associated profiles, as well as profiles which are not currently running. If you click on an item representing a running browser window, the window will be raised to the foreground. If you click on an item representing a non-running profile, a new Firefox instance will be launched with the selected profile.

Here are the instructions for using the applet:

1. Install any needed prerequisites, such as python-xlib.
2. Place the script in its final location:
   sudo cp ffselect.py /usr/local/bin/
3. Run the script with the “install” argument with root privileges, to register the script as an applet:
   sudo /usr/local/bin/ffselect.py install
4. Refresh the GNOME Panel:
   killall gnome-panel
5. Right-click the GNOME Panel, select “Add to Panel”, and pick the Firefox Selection Applet.

Here is the applet script; be sure to note the comments at the top:

• ffselect.py - a GNOME Panel applet for managing separate Firefox instances on the desktop.

Update 2010-01-17: This script stopped working after a recent Ubuntu update. Some change exposed a minor bug in the code. This is now corrected.

I'm often running multiple instances of Firefox on my desktop. Not just multiple windows, but multiple instances, with each associated with a different Firefox profile. I do this so I can have one instance running with the default profile for normal web use, and another completely separate instance running in an alternate profile loaded with development tools such as Firebug. This way, I can use different settings for certain development work, and if the development tools go haywire and interfere with the browser, they don't affect the Firefox instance I use for "normal" web browsing. I selected a different theme for the development instance, so I can visually tell them apart on the desktop.

One problem, though, is that when I click links in other applications (mail reader, feed reader, etc.) the link is sometimes opened in the development instance instead of my default instance. This is because the application uses the firefox executable to open the link, which sends the URL to the first instance it happens to see.

To fix this problem, I wrote a small python script which is smart enough to only send the link to the Firefox instance using the default profile. If no instance is using the default profile, a new instance is launched. This script uses Xlib to scan windows in search of the Firefox instance, so it will only work on X11-based platforms (Linux, Solaris, etc.).

To use this script for opening links in non-browser applications, you'll need to install the script as the default program for opening URLs in your desktop environment. In my GNOME environment, I accomplished this by using gconf-editor to set these gconf values:

Here is the script, be sure to note the comments at the top:

• ffremote.py - a python script to open links only in the Firefox instance running the desired profile.

I frequently use the command-line GNU bc calculator to do quick arithmetic while debugging C programs. An ongoing annoyance is bc's insistence that the hexadecimal digits A-F must be entered in uppercase. Other software, such as GDB and printf("%p",...), output lowercase hexadecimal digits, which keeps me from cutting and pasting the values into bc. Manually typing the values can be a big time sink when I need to perform a lot of calculations.

I finally got fed up with this behavior and made a version of bc with the following modifications:

1. Hexadecimal digits may now be entered in lowercase.
2. Input and output defaults to base 16.

I named this alternate version "xc" and use it when I'm performing pointer arithmetic while debugging. Note that since the lowercase letters a-f are now regarded as hexadecimal digits, some bc functionality such as functions and variables may be impacted as a side effect.

Here are links to the baseline bc source code, and the patch with my modifications:

• bc-1.06.tar.gz (via prep.ai.mit.edu)
• bc-lowercase-hex-hack.patch

Here's a cheatsheet for compilation. I needed to install the libreadline5-dev package on my Ubuntu system to compile with readline support, which is needed if you want to use cursor keys to edit your input.

tar xvfpz bc-1.06.tar.gz
patch -p0 < bc-lowercase-hex-hack.patch
cd bc-1.06
./configure --with-readline
make
sudo mv bc/bc /usr/local/bin/xc

Update, April 22nd, 2012: The exact version of bc that works with this patch disappeared from prep.ai.mit.edu, so I'm changing the link to a locally hosted package. (This version is still available at ftp.gnu.org, for now.)

I decided to try using GNOME Terminal with ugly fonts — that is, bitmap fonts which are not anti-aliased. The goal of this experiment was to improve terminal performance when scrolling large amounts of information in a big window. At the end of the day, the performance was not improved in my case. (I have my monitor rotated 90° for a portrait display, and I think my use of rotation may be a bottleneck for my graphics throughput.) However, I decided to keep the ugly fonts anyway, since it seems to allow me to use smaller fonts without sacrificing legibility.

Since there were a few steps involved in forcing the terminal to use ugly fonts, without allowing the ugliness to spread to other applications, I decided to document my findings here.

Finding ugly fonts to use

Some modern systems may not ship with fonts which are sufficiently ugly. Hongli Lai has written about this on his blog, and provides the MiscFixed font for download. I downloaded and installed this font.

Configuring Fontconfig to allow ugly fonts

Surprisingly, the MiscFixed font is not available in GNOME Terminal's font selection dialog box, even after I installed it! It turns out that modern systems (or at least my Ubuntu 8.04), in a noble effort to protect delicate users from the shock of seeing ugly fonts at any cost, have actually configured Fontconfig (the system font manager) to reject bitmap fonts.

Fontconfig can be reconfigured to allow bitmap fonts, for those of us who aren't afraid of cutting ourselves on the sharp edges of these harshly defined glyphs. Most of the Fontconfig configuration is in the form of symbolic links to XML fragments in the /etc/fonts/conf.d directory. I found the link which disallows bitmap fonts (70-no-bitmaps.conf on my system) and replaced it with a link which permits these fonts (../conf.avail/70-yes-bitmaps.conf on my system). Lo and behold, the ugly MiscFixed font appeared in the terminal's font selection dialog box for my enjoyment!

Unfortunately, my elation was short-lived. Other applications inexplicably starting preferring ugly fonts over the beautiful anti-aliased TrueType fonts they previously used. In particular, my web browser began to render most web pages with fonts which were not just ugly, but bitmap-scaled into horrifying contortions. After my eyes stopped bleeding and I completed several rounds of post-trauma counseling, I decided I needed a way to have a special configuration of Fontconfig just for the terminal.

Using an alternate Fontconfig configuration

First, I set up an alternate copy of the Fontconfig configuration by copying /etc/fonts/fonts.conf to fonts-bitmap.conf. I also made a copy of the /etc/fonts/conf.d subdirectory as conf-bitmap.d, configured it to allow bitmap fonts, and changed the new fonts-bitmap.conf to use it.

According to the Fontconfig documentation, I should be able to run applications with the FC_CONFIG_FILE environment variable set to the new configuration file, and have them use the alternate configuration. This didn't work for me — GNOME Terminal used the system configuration regardless. Perhaps my system is too old, or too new, or maybe Pango's use of Fontconfig interferes.

In my quixotic quest for ugly fonts, I wrote a small wrapper library which, when preloaded into an application with LD_PRELOAD, will intercept calls to open(). Requests to open the fonts.conf file will be remapped to open the fonts-bitmap.conf file instead. For convenience, I configured a launcher icon in the GNOME Panel to run the terminal with my wrapper library using the following command line:

sh -c "LD_PRELOAD=~/work/fc-config-override/fc-config-override.so exec gnome-terminal"

Using this method, I can now use ugly fonts in the terminal, while using attractive fonts in all other applications. A link to the wrapper library is below.

Download:

• fc-config-override.tar.gz

The presence of multiple "clipboard" selection buffers in the X Window System has long been a source of irritation with me, and I frequently end up pasting the wrong thing. A few years ago, I hacked the xterm terminal emulator program to always copy into both buffers, and this solved the problem for me. However, I've been playing around with GNOME Terminal lately, so I recently patched its source code to do something similar.

Background

The root of the problem is that there are two different "selection buffers" that are commonly used in the X Window System:

CLIPBOARD

The "CLIPBOARD" selection buffer is used for the cut-and-paste functions that most users are familiar with: Selecting the "cut", "copy" or "paste" menu items from the application's "Edit" menu, or using the corresponding CTRL-X, CTRL-C, or CTRL-V shortcut keys. This selection buffer is the standard means of performing cut-and-paste operations in most modern applications. However, this selection buffer is unfortunately not used when you merely highlight some text in the GNOME Terminal.

PRIMARY

The "PRIMARY" selection buffer receives data when the user highlights text with the mouse. The text in this buffer is pasted when the user presses the middle mouse button in an application's text entry field. This cut-and-paste buffer is a legacy function which new users are generally not told about, in the interests of avoiding mass confusion. Most modern applications support this buffer.

The annoyance is that text highlighted in GNOME Terminal (and thus stored in PRIMARY) can be pasted into another application by clicking the middle mouse button, but cannot be pasted with CTRL-V. Likewise, text copied from the terminal with SHIFT-CTRL-C (or Edit->Copy) might not be available for pasting with the middle mouse button. I frequently get into situations where I have one bit of text in PRIMARY, and another in CLIPBOARD, and I inevitably use the wrong paste function.

A patch for GNOME Terminal 2.22.3

The attached patch modifies the terminal source code to always copy selected text into CLIPBOARD shortly after it has been copied into PRIMARY. Thus, when you highlight a bit of text in the terminal, it can be pasted into another application with either the middle mouse button or CTRL-V.

This patch is for the GNOME Terminal 2.22.3 source code. I'm sure the patch won't work on more recent versions because they've rearranged a few things, but the basic concept should still work. The patch for xterm I wrote a few years ago is also still available.

• gnome-terminal-2.22.3-friendlyclipboard.patch

A friend of mine wrote an interesting blog post advocating the use of Java for rich internet applications, and expressed frustration with its lack of Web 2.0 mind share. This got me to thinking about how a lack of polish often seems to hold Java back, and I wrote the following comment in response. (I'm reposting it here because, yeah, I'm just that desperate for blog material.)

Technically speaking, I think Java could make a killer platform for the sorts of applications you're describing. I quite like Java, and I'm especially looking forward to the improved dynamic language support that Java 7 is supposed to offer.

However, I think Sun has not been very successful at managing the customer experience. Success in this sort of consumer-facing technology requires not only solid engineering, but getting right a lot of the small details that add up to a positive experience. Several things come to mind about Java:

1. As you mentioned, people remember applets from the bad old days.
2. Java has always struggled with trying to provide responsive GUIs, and no amount of switching back and forth between heavyweight and lightweight widgets has helped. I haven't looked at Java GUIs lately (or JavaFX), but like applets, it may be an uphill battle to convince people that things are different now.
3. When I mentioned the use of Java to a customer once, he looked at me in shock and said, "You mean that thing in my system tray that's always bugging me about needing to be updated?" Flash, which has arguably won the applet war, doesn't generate that sort of reaction in people.
4. Oh yeah, speaking of system trays... Java finally introduced support for putting icons in the "system tray" / "notification area" in December 2006 -- about ten years too late. Ten years is not a fast enough response time when competing on feature points related to the user experience.
5. Probably many other minor details...

Steve Jobs would not approve of the conditions that led to the above points. I'm not an Apple fanboy or anything, but let's face it... the dude knows how to sell products. Sun needs to take a page or two from the Apple book, and focus on the spit and shine. Even then, Sun will need to lay down some serious shock and awe to supersede bad memories. I would love to see Sun pull this off, before everyone gives up and just starts writing web applications in x86.

While catching up on a mailing list recently, I was reminded yet again of my ongoing annoyance with the structure of online discussions. Many online forums and comment pages show posts one after another, in the order they were posted. This sort of linear structure (or lack of structure) becomes unwieldy for any but the smallest discussions. The author of a post may have introduced a new and interesting angle to the discussion, but you still have to wade through page after page of postings to see if there are any replies specifically directed to his post. (Here's a random example of a linear discussion forum -- 15 pages of one post after another!)

Tree-threaded discussions offer a more sophisticated approach to online forums. (This format is often referred to simply as "threaded", although linear forums also refer to their groups of postings as "threads".) Wikipedia defines this sort of message layout as "a tree-like view applying logical reply structure before chronological order." Replies are attached directly to the message being replied to, producing a tree-like structure. This is useful because the tree structure mirrors the conceptual organization of the discussion -- as discussions progress, each "branch" of the tree may focus on different details of the subject matter. When discussion goes off on a tangent or becomes off-topic, the damage is confined to a discrete branch of the tree which can be safely ignored. Grouping messages based on reply associations makes much more sense than displaying messages based on what time they were posted -- after all, the content of a post is more closely related to its ancestor and descendants than it is to posts made chronologically before or after. (Here's an example of a tree-threaded discussion layout on Google Groups.)

However, most (or all) of today's tree-threaded discussion systems fail spectacularly at delivering the power of tree threads to the user. They use tree threads as a sorting mechanism, and perhaps indent responses to hint at the underlying tree, but they still show messages one after another in a big list. This can be seen not just on web sites such as the Google Groups example, but also in supposedly thread-capable mail readers such as Thunderbird and Mutt. These programs take a linear stream of posts, use the reply associations to internally construct a beautiful, highly structured tree, and then smash it flat to render a pale shadow of the thread's former structure. Ideally, a discussion system should bring the power of the tree -- and its close conceptual mapping of the contained discussion -- to the user. Such a user interface could provide useful and time-saving features such as:

• Tree navigation. In addition to the usual "next" and "previous", the user could "ascend" the tree to see parent messages, or "descend" to read replies to any given post.
• Tree visualization. If the tree is represented visually, the user could get an overall idea of the discussion's structure, and jump between branches to see what sub-topics have arisen.
• Branch management. If the user judges that a branch has gone off on a tangent, off-topic, or into an uninteresting sub-topic, the user could "junk" the entire branch (the current post and all its descendants) and avoid seeing those unwanted messages, while continuing to peruse the more useful branches. Additionally, such junking could be persistent -- the system would know to automatically junk any additional messages that arrive on that branch.

In all my years of using the Internet, I've only come across one program that gets it right -- a terminal-based USENET client known as Threaded Read News, or trn for short. Trn, which dates back to 1990 (and is a derivative of an even older program), provides access to USENET newsgroup articles in a way that provides the features I mentioned above -- cursor keys can be used to directly navigate a visualization of the tree, and operations (such as junking or "killfiling") can be performed on the entire thread or specific branches of a thread as they are encountered. When I became frustrated enough with trying to keep up with a busy mailing list using the usual clunky tools, I decided it was time to dust off these relics of antiquity. I installed InterNetNews (INN) -- a USENET newsgroup server package -- on my host and directed the mailing list into it using a procmail recipe and INN's provided mailpost utility. The result is that I can now read the mailing list in all of its threaded goodness using trn. Here's a screenshot of trn showing a post -- notice the tree diagram in the upper-right hand corner:

Before you rush out to download the latest version of trn, you should know that these tools from yesteryear are definitely old-school and have an enormously steep learning curve. It's worth it to me because I'm already familiar with trn from using it heavily back in the old days, and also because of the most important advantage of a properly tree-threaded discussion system: speed. It is simply not feasible to consume huge amounts of discussion content in a reasonable amount of time without such a tool. What the world really needs, though, is for these concepts to be implemented in an attractive and easy to use Web 2.0 interface so we can finally rid ourselves of the painful linear and pseudo-tree discussion systems that litter the web today.