|Opening links in the correct Firefox instance|
I'm often running multiple instances of Firefox on my desktop. Not just multiple windows, but multiple instances, with each associated with a different Firefox profile. I do this so I can have one instance running with the default profile for normal web use, and another completely separate instance running in an alternate profile loaded with development tools such as Firebug. This way, I can use different settings for certain development work, and if the development tools go haywire and interfere with the browser, they don't affect the Firefox instance I use for "normal" web browsing. I selected a different theme for the development instance, so I can visually tell them apart on the desktop.
One problem, though, is that when I click links in other applications (mail reader, feed reader, etc.) the link is sometimes opened in the development instance instead of my default instance. This is because the application uses the firefox executable to open the link, which sends the URL to the first instance it happens to see.
To fix this problem, I wrote a small python script which is smart enough to only send the link to the Firefox instance using the default profile. If no instance is using the default profile, a new instance is launched. This script uses Xlib to scan windows in search of the Firefox instance, so it will only work on X11-based platforms (Linux, Solaris, etc.).
To use this script for opening links in non-browser applications, you'll need to install the script as the default program for opening URLs in your desktop environment. In my GNOME environment, I accomplished this by using gconf-editor to set these gconf values:
Here is the script, be sure to note the comments at the top:
- ffremote.py - a python script to open links only in the Firefox instance running the desired profile.
|Lowercase hexadecimal hack for GNU bc|
I frequently use the command-line GNU bc calculator to do quick arithmetic while debugging C programs. An ongoing annoyance is bc's insistence that the hexadecimal digits A-F must be entered in uppercase. Other software, such as GDB and printf("%p",...), output lowercase hexadecimal digits, which keeps me from cutting and pasting the values into bc. Manually typing the values can be a big time sink when I need to perform a lot of calculations.
I finally got fed up with this behavior and made a version of bc with the following modifications:
- Hexadecimal digits may now be entered in lowercase.
- Input and output defaults to base 16.
Here are links to the baseline bc source code, and the patch with my modifications:
tar xvfpz bc-1.06.tar.gz patch -p0 < bc-lowercase-hex-hack.patch cd bc-1.06 ./configure --with-readline make sudo mv bc/bc /usr/local/bin/xc
Update, April 22nd, 2012: The exact version of bc that works with this patch disappeared from prep.ai.mit.edu, so I'm changing the link to a locally hosted package. (This version is still available at ftp.gnu.org, for now.)
|Forcing GNOME Terminal to use ugly fonts|
I decided to try using GNOME Terminal with ugly fonts — that is, bitmap fonts which are not anti-aliased. The goal of this experiment was to improve terminal performance when scrolling large amounts of information in a big window. At the end of the day, the performance was not improved in my case. (I have my monitor rotated 90° for a portrait display, and I think my use of rotation may be a bottleneck for my graphics throughput.) However, I decided to keep the ugly fonts anyway, since it seems to allow me to use smaller fonts without sacrificing legibility.
Since there were a few steps involved in forcing the terminal to use ugly fonts, without allowing the ugliness to spread to other applications, I decided to document my findings here.
Finding ugly fonts to use
Some modern systems may not ship with fonts which are sufficiently ugly. Hongli Lai has written about this on his blog, and provides the MiscFixed font for download. I downloaded and installed this font.
Configuring Fontconfig to allow ugly fonts
Surprisingly, the MiscFixed font is not available in GNOME Terminal's font selection dialog box, even after I installed it! It turns out that modern systems (or at least my Ubuntu 8.04), in a noble effort to protect delicate users from the shock of seeing ugly fonts at any cost, have actually configured Fontconfig (the system font manager) to reject bitmap fonts.
Fontconfig can be reconfigured to allow bitmap fonts, for those of us who aren't afraid of cutting ourselves on the sharp edges of these harshly defined glyphs. Most of the Fontconfig configuration is in the form of symbolic links to XML fragments in the /etc/fonts/conf.d directory. I found the link which disallows bitmap fonts (70-no-bitmaps.conf on my system) and replaced it with a link which permits these fonts (../conf.avail/70-yes-bitmaps.conf on my system). Lo and behold, the ugly MiscFixed font appeared in the terminal's font selection dialog box for my enjoyment!
Unfortunately, my elation was short-lived. Other applications inexplicably starting preferring ugly fonts over the beautiful anti-aliased TrueType fonts they previously used. In particular, my web browser began to render most web pages with fonts which were not just ugly, but bitmap-scaled into horrifying contortions. After my eyes stopped bleeding and I completed several rounds of post-trauma counseling, I decided I needed a way to have a special configuration of Fontconfig just for the terminal.
Using an alternate Fontconfig configuration
First, I set up an alternate copy of the Fontconfig configuration by copying /etc/fonts/fonts.conf to fonts-bitmap.conf. I also made a copy of the /etc/fonts/conf.d subdirectory as conf-bitmap.d, configured it to allow bitmap fonts, and changed the new fonts-bitmap.conf to use it.
According to the Fontconfig documentation, I should be able to run applications with the FC_CONFIG_FILE environment variable set to the new configuration file, and have them use the alternate configuration. This didn't work for me — GNOME Terminal used the system configuration regardless. Perhaps my system is too old, or too new, or maybe Pango's use of Fontconfig interferes.
In my quixotic quest for ugly fonts, I wrote a small wrapper library which, when preloaded into an application with LD_PRELOAD, will intercept calls to open(). Requests to open the fonts.conf file will be remapped to open the fonts-bitmap.conf file instead. For convenience, I configured a launcher icon in the GNOME Panel to run the terminal with my wrapper library using the following command line:
sh -c "LD_PRELOAD=~/work/fc-config-override/fc-config-override.so exec gnome-terminal"
Using this method, I can now use ugly fonts in the terminal, while using attractive fonts in all other applications. A link to the wrapper library is below.
|GNOME Terminal friendly clipboard patch|
The presence of multiple "clipboard" selection buffers in the X Window System has long been a source of irritation with me, and I frequently end up pasting the wrong thing. A few years ago, I hacked the xterm terminal emulator program to always copy into both buffers, and this solved the problem for me. However, I've been playing around with GNOME Terminal lately, so I recently patched its source code to do something similar.
The root of the problem is that there are two different "selection buffers" that are commonly used in the X Window System:
- The "CLIPBOARD" selection buffer is used for the cut-and-paste functions that most users are familiar with: Selecting the "cut", "copy" or "paste" menu items from the application's "Edit" menu, or using the corresponding CTRL-X, CTRL-C, or CTRL-V shortcut keys. This selection buffer is the standard means of performing cut-and-paste operations in most modern applications. However, this selection buffer is unfortunately not used when you merely highlight some text in the GNOME Terminal.
- The "PRIMARY" selection buffer receives data when the user highlights text with the mouse. The text in this buffer is pasted when the user presses the middle mouse button in an application's text entry field. This cut-and-paste buffer is a legacy function which new users are generally not told about, in the interests of avoiding mass confusion. Most modern applications support this buffer.
The annoyance is that text highlighted in GNOME Terminal (and thus stored in PRIMARY) can be pasted into another application by clicking the middle mouse button, but cannot be pasted with CTRL-V. Likewise, text copied from the terminal with SHIFT-CTRL-C (or Edit->Copy) might not be available for pasting with the middle mouse button. I frequently get into situations where I have one bit of text in PRIMARY, and another in CLIPBOARD, and I inevitably use the wrong paste function.
A patch for GNOME Terminal 2.22.3
The attached patch modifies the terminal source code to always copy selected text into CLIPBOARD shortly after it has been copied into PRIMARY. Thus, when you highlight a bit of text in the terminal, it can be pasted into another application with either the middle mouse button or CTRL-V.
This patch is for the GNOME Terminal 2.22.3 source code. I'm sure the patch won't work on more recent versions because they've rearranged a few things, but the basic concept should still work. The patch for xterm I wrote a few years ago is also still available.
|The Java user experience|
A friend of mine wrote an interesting blog post advocating the use of Java for rich internet applications, and expressed frustration with its lack of Web 2.0 mind share. This got me to thinking about how a lack of polish often seems to hold Java back, and I wrote the following comment in response. (I'm reposting it here because, yeah, I'm just that desperate for blog material.)
Technically speaking, I think Java could make a killer platform for the sorts of applications you're describing. I quite like Java, and I'm especially looking forward to the improved dynamic language support that Java 7 is supposed to offer.
However, I think Sun has not been very successful at managing the customer experience. Success in this sort of consumer-facing technology requires not only solid engineering, but getting right a lot of the small details that add up to a positive experience. Several things come to mind about Java:
- As you mentioned, people remember applets from the bad old days.
- Java has always struggled with trying to provide responsive GUIs, and no amount of switching back and forth between heavyweight and lightweight widgets has helped. I haven't looked at Java GUIs lately (or JavaFX), but like applets, it may be an uphill battle to convince people that things are different now.
- When I mentioned the use of Java to a customer once, he looked at me in shock and said, "You mean that thing in my system tray that's always bugging me about needing to be updated?" Flash, which has arguably won the applet war, doesn't generate that sort of reaction in people.
- Oh yeah, speaking of system trays... Java finally introduced support for putting icons in the "system tray" / "notification area" in December 2006 -- about ten years too late. Ten years is not a fast enough response time when competing on feature points related to the user experience.
- Probably many other minor details...
Steve Jobs would not approve of the conditions that led to the above points. I'm not an Apple fanboy or anything, but let's face it... the dude knows how to sell products. Sun needs to take a page or two from the Apple book, and focus on the spit and shine. Even then, Sun will need to lay down some serious shock and awe to supersede bad memories. I would love to see Sun pull this off, before everyone gives up and just starts writing web applications in x86.
|The lost art of threaded discussions|
While catching up on a mailing list recently, I was reminded yet again of my ongoing annoyance with the structure of online discussions. Many online forums and comment pages show posts one after another, in the order they were posted. This sort of linear structure (or lack of structure) becomes unwieldy for any but the smallest discussions. The author of a post may have introduced a new and interesting angle to the discussion, but you still have to wade through page after page of postings to see if there are any replies specifically directed to his post. (Here's a random example of a linear discussion forum -- 15 pages of one post after another!)
Tree-threaded discussions offer a more sophisticated approach to online forums. (This format is often referred to simply as "threaded", although linear forums also refer to their groups of postings as "threads".) Wikipedia defines this sort of message layout as "a tree-like view applying logical reply structure before chronological order." Replies are attached directly to the message being replied to, producing a tree-like structure. This is useful because the tree structure mirrors the conceptual organization of the discussion -- as discussions progress, each "branch" of the tree may focus on different details of the subject matter. When discussion goes off on a tangent or becomes off-topic, the damage is confined to a discrete branch of the tree which can be safely ignored. Grouping messages based on reply associations makes much more sense than displaying messages based on what time they were posted -- after all, the content of a post is more closely related to its ancestor and descendants than it is to posts made chronologically before or after. (Here's an example of a tree-threaded discussion layout on Google Groups.)
However, most (or all) of today's tree-threaded discussion systems fail spectacularly at delivering the power of tree threads to the user. They use tree threads as a sorting mechanism, and perhaps indent responses to hint at the underlying tree, but they still show messages one after another in a big list. This can be seen not just on web sites such as the Google Groups example, but also in supposedly thread-capable mail readers such as Thunderbird and Mutt. These programs take a linear stream of posts, use the reply associations to internally construct a beautiful, highly structured tree, and then smash it flat to render a pale shadow of the thread's former structure. Ideally, a discussion system should bring the power of the tree -- and its close conceptual mapping of the contained discussion -- to the user. Such a user interface could provide useful and time-saving features such as:
- Tree navigation. In addition to the usual "next" and "previous", the user could "ascend" the tree to see parent messages, or "descend" to read replies to any given post.
- Tree visualization. If the tree is represented visually, the user could get an overall idea of the discussion's structure, and jump between branches to see what sub-topics have arisen.
- Branch management. If the user judges that a branch has gone off on a tangent, off-topic, or into an uninteresting sub-topic, the user could "junk" the entire branch (the current post and all its descendants) and avoid seeing those unwanted messages, while continuing to peruse the more useful branches. Additionally, such junking could be persistent -- the system would know to automatically junk any additional messages that arrive on that branch.
In all my years of using the Internet, I've only come across one program that gets it right -- a terminal-based USENET client known as Threaded Read News, or trn for short. Trn, which dates back to 1990 (and is a derivative of an even older program), provides access to USENET newsgroup articles in a way that provides the features I mentioned above -- cursor keys can be used to directly navigate a visualization of the tree, and operations (such as junking or "killfiling") can be performed on the entire thread or specific branches of a thread as they are encountered. When I became frustrated enough with trying to keep up with a busy mailing list using the usual clunky tools, I decided it was time to dust off these relics of antiquity. I installed InterNetNews (INN) -- a USENET newsgroup server package -- on my host and directed the mailing list into it using a procmail recipe and INN's provided mailpost utility. The result is that I can now read the mailing list in all of its threaded goodness using trn. Here's a screenshot of trn showing a post -- notice the tree diagram in the upper-right hand corner:
Before you rush out to download the latest version of trn, you should know that these tools from yesteryear are definitely old-school and have an enormously steep learning curve. It's worth it to me because I'm already familiar with trn from using it heavily back in the old days, and also because of the most important advantage of a properly tree-threaded discussion system: speed. It is simply not feasible to consume huge amounts of discussion content in a reasonable amount of time without such a tool. What the world really needs, though, is for these concepts to be implemented in an attractive and easy to use Web 2.0 interface so we can finally rid ourselves of the painful linear and pseudo-tree discussion systems that litter the web today.
|Software Engineering Fiction: Near-future speculation in <em>Rainbows End</em> and <em>Halting State</em>|
Science fiction writers have always speculated about the future -- sometimes they are quite accurate (Jules Verne predicted live newscasts, space travel, and the Internet in the 19th century) and other times not so much (flying cars haven't taken off yet). Authors typically set their science fiction stories well into the future, so they can get in a few good years of sales before the future becomes the present and their works begin to look naïve and obsolete. Far-future settings also allow an author to solicit a greater suspension of disbelief from the reader -- sure, warp drives and holodecks sound fanciful, but who's to say what wonders may be common in 400 years? However, occasionally authors will accept the difficult challenge of setting their stories in a near-future world similar to our own, where predictions about technological advancement must be grounded in reality and "science as magic" hand waving is not tolerated. It is these stories, when written by authors with expertise in the subject matter and the willingness to conduct research, that provide invaluable inspiration to those of us in the business of turning imagination into reality.
Vernor Vinge's Rainbows End and Charles Stross's Halting State are two recent books that do an excellent job of illuminating possible future paths of software technology. Both books are written by authors who are experienced in the art of software (Vinge is a computer scientist, and Stross is a recovering programmer), and both books express a very specific vision of the next steps in software based on current technology and well understood trends. These books describe a future where ubiquitous computing has become a reality. Computers have mostly disappeared into the woodwork, keyboards and mice have given way to gestures, and users live in an information-rich augmented reality world where electronic eyewear overlays data onto everyday scenes in useful, and sometimes not so useful, ways.
Rainbows End is set in the year 2025, and is told from the perspective of a recently cured Alzheimer's survivor who must come to terms with the changes in the world around him, while unknowingly becoming involved in an international conspiracy. The common operating systems and user interfaces of today, such as Microsoft Windows, are no longer commercially viable (except as legacy interfaces for old people) and have been replaced with a product called Epiphany which ties the new technologies together. Computer hardware is sown into the fabric of clothes, and the display device of choice is a pair of electronically enhanced contact lenses. As far as the story goes, I don't think this is Vinge's best work, and I found the bits about the life-extended elderly being "retrained" for new jobs alongside high school students to be somewhat hokey. However, for idea-obsessed readers, the incredible detail given to technological speculation makes up for the weak story, and I can't resist a book whose most clever chapter has one of the best titles ever: "How-to-Survive-the-Next-Thirty-Minutes.pdf". If you want a taste of Vinge's writing with a better story line, I'd recommend the excellent Marooned in Realtime.
The story of Halting State takes place in the year 2018, and revolves around a software engineer at a company that produces massively multiplayer online games. This character finds himself unwillingly tangled in a complex web of international intrigue that is made possible by the ubiquitous computer technology portrayed in the book. While Halting State describes a similar ubiquitous computing environment as Rainbows End, the hardware speculation is less ambitious -- instead of wearable computers, mobile phones are the principle unit of personal computing, and eyeglasses provide the augmented reality overlay instead of contact lenses. Halting State is unconventionally written in the second person, as a homage to early computer adventure games. While I know at least one person who found this to be distracting, the second person prose became natural and unnoticeable for myself and others after the first chapter or two.
I found a couple of the technological predictions presented in these books to be particularly clever. In the world of Rainbows End, transparent cryptography has encouraged the development of sophisticated economic systems. The authenticity provided by digital signatures and certificates not only protects the movement of money, but allows contracts for specific goods and services to be anonymously subcontracted in real time without the involvement of lawyers. The anonymity allows arbitrary individuals to interact with the economy in various agreed-upon roles -- a concept that software engineers refer to as the "separation of interface and implementation." The efficiency makes possible rich systems of multi-level affiliations, with each level providing value in well-defined ways. These new systems raise issues of their potential for criminal abuse, which are explored in the book. In the following excerpt from Vernor Vinge's Rainbows End, Juan Orozco is subcontracting a task -- which he himself was subcontracted anonymously from a third party -- to Winston Blount, who verifies the authenticity of the payment promise by checking its digital signature which is certified by a trusted third party -- in this case, Bank of America:
"Y-you know, Dean, I may be able to help. No wait -- I don't mean by myself. I have an affiliance you might be interested in."
He seemed to know what affiliance was. Juan explained Big Lizard's deal. "So there could be some real money in this." He showed him the payoff certificates, and wondered how much his recruit would see there.
Blount squinted his eyes, no doubt trying to parse the certificates into a form that Bank of America could validate. After a moment he nodded, without granting Juan numerical enlightenment. "But money isn't everything, especially in my situation."
"Well, um, I bet whoever's behind these certs would have a lot of angles. Maybe you could get a conversion to help-in-kind. I mean, to something you need."
In Halting State, the intersection of refined peer-to-peer networking protocols, massively parallel execution contexts, and the increasing computational power of hand-held devices has led to the development of the Zone, an advanced distributed computing environment which relies on processing power provided by the mobile phones of its users. The phone-based Zone nodes are implemented as virtual machine sandboxes which execute distributed code written in the "Python 3000" language. In the following excerpt from Charles Stross's Halting State, the protagonist explains how the Zone uses cryptography to protect assets in online games:
“Zone games don't run on a central server, they run on distributed-processing nodes using a shared network file system. To stop people meddling with the contents, everything is locked using a cryptographic authorization system. If you ‘own' an item in the game, what it really means is that nobody else can update its location or ownership attributes without you using your digital signature to approve the transaction -- and a co-signature signed by a cluster of servers all spying on each other to make sure they haven't been suborned.”
Both Vinge's and Stross's visions of software's future rely on cryptographic systems which are more ubiquitous and transparent than most of the systems in use today. While secure web transactions have been quite successful and have largely met the goal of transparently verifying the authenticity of the web server, use of certificates by individuals to prove their authenticity to others (whether via the web, email, IM, or other protocols) is extraordinarily rare except among extreme power users and certain high-security corporate environments. Bringing the full power of cryptography to ordinary people, and making it so easy that they don't realize they are using it, is a necessary prerequisite to building the next-generation Internet. Perhaps we need a revival of the cypherpunk movement of the 1990's -- a sort of Cypherpunk 2.0, if you will -- only this time with an emphasis on user interface simplicity that will finally allow cryptographic technology to be accepted by the public and commonly used to provide privacy and authenticity.
The role of writers in injecting a needed dose of imagination and creativity into engineers and entrepreneurs should be taken very seriously. I'm reminded of the story about how the science fiction writer Katherine MacLean, who predicted the role of computers in communication and music in her 1947 novelette Incommunicado, accidentally stumbled into a conference of electrical engineers and found herself quickly surrounded by Bell Telephone researchers who were inspired by her ideas to build the next generation of communications equipment. I hope that the software engineers of today find similar inspiration in the works of visionaries such as Vinge and Stross, and decide that it's time to pull up the shirt sleeves and get to work building the world of tomorrow.
Note: It is my opinion that the use of the above book cover images and excerpts is acceptable under the fair use doctrine in the United States. If you are the author, publisher, or otherwise hold rights to the above materials and believe me to be in error, please contact me with details.
|Western Design Center - Mesa, Arizona|
I'm traveling in Arizona at the moment, and decided to get in a bit of geek tourism by swinging by the Western Design Center (wikipedia) in Mesa, Arizona. The Western Design Center was founded by Bill Mensch, one of the creators of the MOS Technology 6502 series of microprocessors found in many personal computers of the 1980's, including the Commodore 64 and the Apple II. The 6502 series is somewhat nostalgic for me, as I taught myself machine language programming on the 6502 as a kid.
As far as I know, the Western Design Center is the only surviving manufacturer of 65xx microprocessors. I sort of expected to see a fabrication plant at this address instead of a house, though. According to Google Maps, there's even a pool in the back.
|Migrating legacy documents from Word Writer 4 for the Commodore 64|
This advanced system can be used to arrange words and make edits without the need for white-out.
When I was growing up back in the 1980's, I wrote a lot of documents on my Commodore 64 including school assignments, letters, journals, and attempts at fiction. I used several different word processors during this time, but towards the end I was mostly using a product called Word Writer 4 from Timeworks. Now that I'm older, I'd like to preserve these childhood memories by converting them into a more modern, standardized format that can be easily viewed on modern computers. (You may find this shocking, but it turns out that contemporary word processors lack the ability to import from Commodore 64 file formats.)
The Commodore 64 used these 5.25-inch floppy disks that could hold about 170KB each -- about 80 double-space pages. For you kids out there, that's about 1/100,000th the capacity of your iPhone.
The first step is to recover the raw data off of the 5.25-inch floppy disks before they completely rot. Since I still have my trusty Commodore 1541 disk drive, I ordered the XA1541 cable from Vintage Computer Cables which connects the drive to a PC's parallel port. While this lets me connect my antique disk drive to the best computers that the 1990's had to offer, I've found that it's actually hard to find a computer these days that still has a parallel port. Fortunately, I was able to dig up an old Thinkpad 600 (circa 1999) that has a parallel port. Using the cbm4linux (a.k.a. OpenCBM) tools, I was able to copy the data from my old floppy disks to d64 disk image files on my PC. I was then able to use the cbmconvert program with the "-d" option, which reads the specified d64 disk image and dumps each individual file into the current directory.
Each Word Writer 4 document has a filename ending with ".seq" and contains data in a seemingly proprietary format. A small amount of reverse engineering effort reveals that the format is actually fairly close to ASCII, but various character ranges are offset. (No, it's not PETSCII, either!) Some embedded control codes provide markup information about text styles and formatting. I wrote a small C program called ww4convert that reads the Word Writer 4 file on standard input, and provides an HTML version of the document on standard output.
Now I can completely peruse my tome of old documents using my web browser. However, I must provide a word of warning about resurrecting your childhood documents: You may find out that you were really, really horrible at writing as a child, contrary to your memories of writing paragraph after paragraph of elegant prose. If only I could go back in time and knock some grammar and composition sense into the head of my former self.
|Enthalpy of Application|
Warning: The following post may contain weak metaphors and bad analogies. If these are offensive to you or illegal in your jurisdiction, you may want to skip this post.
Like many people who grew up in the forward thinking, happy go-lucky days of the twentieth century when anything seemed possible, I sometimes wonder what happened to all the futuristic luxuries that technology was supposed to have delivered by now. Science fiction promised us flying cars, ray guns, personal robots, and space colonies. I can understand speculative fiction not living up to reality -- after all, nobody has a crystal ball. However, many of us who were involved in software during the early to mid-nineties had some very specific and realistic notions of how computing technology would progress over the coming years, including the following:
- Ubiquitous computing. Computers would fade into the woodwork all around us, and human-computer interaction would take place using natural metaphors instead of clunky terminals. In some ways we're slowly moving in that direction, as powerful computers become embedded into mobile phones and other handheld devices, but we're not there yet.
- Transparent cryptography. Cryptography would be seamlessly integrated into our day-to-day communications. Today, you can use cryptography to communicate with your bank's web site or connect to your office's network in a fairly seamless fashion, but the vision was much broader than this -- all network messaging (email, instant messaging, twitter, etc.) would be encrypted to protect the contents from prying eyes, and digitally signed to provide strong assurance of the authenticity of the sender.
- Natural language processing. By this time we should have been able to verbally interact with our ubiquitous computing infrastructure, asking our computers to "turn off the coffee pot that I accidentally left on," "check on the status of my portfolio," "open the pod bay doors," etc. At present, natural language technology still seems clunky and limited.
- Intelligent agents, micropayments, effective and low-cost telepresence, and many others.
Some of the above ideas exist today in a limited fashion, although none have lived up to the potential that we once envisioned. Some ideas require the overcoming of difficult challenges (such as the ideas related to artificial intelligence: natural language processing and intelligent agents), and others are technically solved to a large degree but have not been refined enough to be successful in consumer applications (transparent cryptography and micropayments, for example). I've been mulling over a theory about why these technologies have either failed to become reality, or haven't lived up to our expectations.In the late twentieth century, technology went through a period of great advancement in capability which outpaced the application of these capabilities as solutions to problems. An entrepreneur who is investigating ideas for a startup company will find a great many tools available, and many unsolved problems. Using existing tools to solve existing problems can be very lucrative, and the entrepreneur will likely chose to deliver these solutions instead of developing new tools. Because there is a finite (although not fixed) amount of innovative energy in the economy, this low-hanging fruit is bound to be picked first. If it's 2004 and you're looking to start a company, which would be more likely to provide a larger and quicker return on investment -- solving the problem of how to richly communicate with friends using established and well-understood web tools (Facebook), or starting an artificial intelligence research and development laboratory that may not bear fruit for years? When the low-hanging fruit becomes scarce, more energy will be spent on the harder problems, and we will be back on track to building the world of tomorrow.
And now, it's time for the bad analogy you've been waiting for. When I think about the temporary diversion of efforts into applications instead of new technologies, I can't seem to shake from my head a graph from high school chemistry class showing enthalpy of vaporization. (Enthalpy can be very roughly defined as the heat content of a thermodynamic system.) When you add energy (heat) to water, its temperature will increase continuously in proportion to the amount of energy added -- until it reaches the boiling point (201°F at any reasonable elevation), at which time the energy is redirected towards the "enthalpy of vaporization" (also called the "heat of evaporation") and the temperature flatlines at the boiling point until vaporization is complete and the energy can once again be used towards increasing the temperature, as seen in this graph:
In a similar fashion, it seems that the technology industry has been in a sort of "enthalpy of application" where energy (effort) has been redirected towards application, and the technology "temperature" has not risen much:
Before anybody accuses me of being hostile towards Web 2.0 companies, let it be clear that I think putting effort into using existing tools to solve problems is just fine and dandy, and this is the natural order of things. (Even if my graphs seem to imply that Facebook is the ramen noodles of technology.) However, it never hurts to try to make sense of these trends. If the current economic downturn begins drying up the low-hanging fruit prematurely, it may be a good idea to begin looking into the grand challenges and long-term endeavors. I'm personally looking forward to working on some of these challenges in the years ahead.