Android

In the past day or so, I've been noticing these "Package file is invalid" errors on my Android devices while trying to upgrade or install certain packages from the Play Store. A bit of searching revealed that many others are having this problem, and various home remedies abound for trying to fix it, such as clearing the Play Store's app cache. Unfortunately, while these remedies may have worked for past problems that led to this error message being displayed, they are useless when trying to fix the issue people are experiencing this weekend.

I decided to do a bit of digging, and I found that Google's web servers are actually sending corrupted packages to the Play Store app. Therefore, no amount of tweaking your device will fix the problem. (Unless such tweaking happens to result in pulling packages from a different web server that doesn't have corrupted files, I suppose.)

UPDATE 2013-08-12: It appears that this problem is isolated to one or more specific servers on Google Play's content distribution network -- if your closest server has corruption, you'll always see this issue unless you move to a different network and a different server is selected. I see the problem here in Colorado, and a brief Twitter survey shows a high concentration of complaints from the U.S. Midwest and Great Lakes region. Suggestions to use a VPN have some merit -- when I VPN into Dallas, I can successfully update/install these problematic packages, because a non-corrupted server is chosen in that case. (Obviously this isn't a great solution.)

UPDATE 2013-08-13: I heard from a Google Play engineer today. It sounds like they're in the process of rolling out a fix, so our package updates and installs should be back to normal very soon!

I've observed this problem on the following devices:

• Galaxy Nexus (Android 4.2)
• Nexus 10 (Android 4.3)

To investigate the problem, I tried downloading the recently released Twitter 4.1.4 package, and compared the downloaded package file (temporarily stored in /data/data/com.android.providers.downloads/cache/downloadfile.apk) to a known good version.

A hex dump of an uncorrupted Twitter 4.1.4 package looks like this around offset 0x0200000:

01fffc0: 6e69 2067 6fcc 8872 6d65 6b2e 0028 2b42  ni go..rmek..(+B
01fffd0: 6972 2069 6e73 616e 206d c4b1 73c4 b16e  ir insan m..s..n
01fffe0: 2079 6f6b 7361 2062 6972 2062 696c 6769   yoksa bir bilgi
01ffff0: 7361 7961 7220 6dc4 b13f 000c 0c42 6f79  sayar m..?...Boy
0200000: 7574 3a20 252e 3166 6b00 0f11 4b6f 6e75  ut: %.1fk...Konu
0200010: 6d75 2064 65c4 9f69 c59f 7469 7200 0303  mu de..i..tir...
0200020: 5369 6c00 2122 2225 3124 7322 2022 2532  Sil.!""%1$s" "%2
0200030: 2473 2220 6c69 7374 6573 696e 6920 6f6c  $s" listesini ol

A hex dump of the corrupted Twitter apk looks like this around offset 0x0200000:

01fffc0: 6e69 2067 6fcc 8872 6d65 6b2e 0028 2b42  ni go..rmek..(+B
01fffd0: 6972 2069 6e73 616e 206d c4b1 73c4 b16e  ir insan m..s..n
01fffe0: 2079 6f6b 7361 2062 6972 2062 696c 6769   yoksa bir bilgi
01ffff0: 504b 0304 1400 0800 0800 e27c 0543 2d70  PK.........|.C-p
0200000: 8d5b c420 0100 986f 0200 1d00 0400 6173  .[. ...o......as
0200010: 7365 7473 2f66 6f6e 7473 2f52 6f62 6f74  sets/fonts/Robot
0200020: 6f2d 4c69 6768 742e 7474 66fe ca00 00ec  o-Light.ttf.....
0200030: 9d07 7c54 55fa f74f 994c 0a21 bd00 8190  ..|TU..O.L.!....

At 16 bytes before the 2-megabyte mark, the corrupted file begins repeating the contents of the beginning of the file, including the ZIP header. It looks like a common programming error when dealing with buffered I/O streams. I first suspected that the Play Store app or the Android framework on my devices had such an error, but then I used tcpdump to examine the actual HTTP traffic as seen from my router:

GET http://r15---sn-qxo7sn7s.c.android.clients.google.com/market/GetBinary/com.twitter.android/420?...
22:01:25.861259 IP 74.125.x.x.80 > 192.168.x.x.39431: Flags [.], seq 2097056:2098516, ack 527, win 245, length 1460
...
0x0230:  2073 cca7 6966 7265 6e69 2067 6fcc 8872  .s..ifreni.go..r
0x0240:  6d65 6b2e 0028 2b42 6972 2069 6e73 616e  mek..(+Bir.insan
0x0250:  206d c4b1 73c4 b16e 2079 6f6b 7361 2062  .m..s..n.yoksa.b
0x0260:  6972 2062 696c 6769 504b 0304 1400 0800  ir.bilgiPK......
0x0270:  0800 e27c 0543 2d70 8d5b c420 0100 986f  ...|.C-p.[.....o
0x0280:  0200 1d00 0400 6173 7365 7473 2f66 6f6e  ......assets/fon
0x0290:  7473 2f52 6f62 6f74 6f2d 4c69 6768 742e  ts/Roboto-Light.
0x02a0:  7474 66fe ca00 00ec 9d07 7c54 55fa f74f  ttf.......|TU..O

Sure enough, the corruption was present in the stream as sent from Google's web server. I assume that the bug is in Google's web server code, or in some intermediate package processing step at the Play Store. Either way, we'll just have to wait for Google to fix the glitch.

I've been researching the idea of using embedded languages in mobile applications as a way of reusing business logic across platforms. I haven't found a lot of information about how much an embedded language will bloat an app's size, so I decided to see for myself. So far, I've written simple "Hello, world" apps for both Lua and Squirrel. Lua is a simple language that has been heavily used in video games for years. Squirrel is a newer language that was inspired by Lua, but uses a more C-like syntax.

These tests are not very scientific, and only demonstrate the bare minimum task of including the language support as a native shared library, and some JNI code to run a script to generate a "Hello, world" message which is returned to the activity.

I'm frankly blown away by the compact size of these language implementations, especially after getting the impression that including Javascript (via Rhino) would cost many hundreds of kilobytes. That wouldn't be a problem for many apps, but for certain small apps, Rhino could end up being much larger than the app itself. In the case of Lua, which is implemented in a mere 20 C source files, you not only get the Lua virtual machine in 46K, but the compiler to boot! Developers can and do use Lua on other platforms such as iOS, and the Lua code has even been compiled to Javascript via emscripten (an LLVM Javascript backend), which adds the potential of reusing code in HTML5 apps.

I haven't played around with writing code in these languages, though, so I'm curious to hear about people's real-world experiences.

While developing Android applications, I'm often juggling lots of Android machines, both real and virtual. Since I often need to connect to these machines over the network with adb connect, I found it useful and educational to write a small home screen widget that always shows the device's IP address. This is a pretty dumb application, but I decided that it would be a good opportunity to learn how to publish apps on the Android Market.

Downloads

• Download Network Information from the Android Market

The main screen lists all network interfaces and enumerates their IPv4/IPv6 addresses.

This home screen widget always shows your current IP address.

An on-screen trackpad and keyboard allow a computer to be remote controlled.

Valence supports mDNS service discovery (aka Bonjour or Avahi) to locate participating VNC servers on the local network.

In my spare cycles recently, I've been tinkering with developing Android code to control home theater components. As someone who has been passionately involved with the consumer electronics industry over the past ten years, my personal home theater system includes quite a collection of disparate devices that defy even the most feature-rich universal remote controls. What the world needs is an open, extensible Android remote control application that can be rapidly updated to support new devices as they are released.

The first module of code I've developed is a remote control for home theater computers, such as the one I use to watch YouTube videos and other web video content. I've built a standalone app with this code for testing purposes. There are plenty of other such apps on the Android Market, but my program, Valence, seeks to provide a simpler experience in the following ways:

• Most computer remote control apps require that the controlled computer be running special software, unique to the app, that relays mouse and keyboard events to the operating system. Not only does this add yet another single-purpose ever-running program to your computer, but the author may not have support for your operating system yet. To solve this problem, Valence uses the industry standard VNC system and its RFB (remote framebuffer) protocol. Many operating systems come with VNC built-in, and many people may already have VNC installed and enabled. Instead of using VNC to see the screen of a remote computer, Valence uses VNC in a strictly one-way fashion—input events are transmitted from the Android handset to the VNC server, but video frames are never sent from the server to the handset.
• Valence supports mDNS service discovery to automatically find VNC servers on your local network, without the need to mess around with IP addresses. Not all VNC servers support discovery, but it sure saves some hassle if yours does. (A few Android phones do have trouble performing such discovery, but most seem to work.)

Today I'm releasing an unpolished "rough draft" of Valence so my friends and colleagues can test its compatibility with their networks. Some known bugs remain in this version, but I'm erring on the side of getting feedback early. Eventually, I'll roll the Valence functionality into my larger remote control framework that provides many more features such as controlling IR devices.

To use Valence, make sure you have VNC server software installed on your computer, and configured to allow connections:

• Mac OS. Macs have built-in VNC software, so no additional software needs to be installed. In "System Preferences," go to the "Sharing" tab and make sure "Remote Management" is selected. Then click "Computer Settings..." and make sure "VNC viewers may control screen with password" is selected, and provide a password.
• Ubuntu Linux. Ubuntu Linux has built-in VNC software, so no additional software needs to be installed. In the System menu, select Preferences then Remote Desktop. Check "Allow other users to view your desktop" and "Allow other users to control your desktop," uncheck "You must confirm each access to this machine," and check "Require the user to enter this password." Assign a password.
• Windows. Windows does not have a built-in VNC server, so you'll need to install one such as TightVNC. Unfortunately, TightVNC seems to be a little jerky with processing mouse movement, so please let me know if you find a better VNC server for Windows that works with recent releases such as Windows 7. On TightVNC's download page, click the "download" link next to "Self-installing package for Windows." Open the downloaded package, click "Run," and proceed through the setup wizard. (Just keep hitting "Next" and "I agree".) Enter a password when prompted in the "Service Configuration" dialog, click "Install," and complete the installation.

Update: June 7, 2011

I've uploaded a new beta package, which is linked below. Changes include the following:

• Bug fix: Valence did not properly store passwords that were less than eight characters. This has been fixed.
• Mouse movement should be much smoother now. In particular, usability with the TightVNC server has been vastly improved. (I'm now setting TCP_NODELAY on the socket to disable Nagle's algorithm.)
• Reorientation (from landscape to portrait and vice-versa) no longer causes the connection to be killed and re-established.
• An "about" page and help documentation are now available by pressing the menu button.
• Bug fix: If the user pressed the back button while Valence was trying to connect, the app would crash. This has been fixed.
• I tweaked the launcher icon slightly.

Update: June 30, 2011

Yet another beta release. Changes include the following:

• Keyboard
  • Added a new button for special keys like "esc", "tab", F1-F10, etc.
  • Better support for the HTC soft keyboard.
  • Better support for international characters.
• Documentation
  • Added a section on security considerations to the online help.
  • The "about" window now contains the build date of the package.
  • The "about" window is now scrollable, so it works in landscape.
• VNC setup
  • In the setup form, alternate port numbers (other than 5900) were not being saved. Fixed.
  • The setup form would reset whenever the device changed orientation. Fixed.
  • Your configured VNC servers can now be edited. Long press on the server, and select "edit."
  • You can now change the displayed name of a VNC server, instead of being forced to see the name that was auto-detected from the server.
  • The setup form is now scrollable, in case it doesn't fit on your screen.
• Trackpad
  • The trackpad code has been completely rewritten to pave the way for new features. This is still a complex piece of logic, and I wouldn't be surprised to see some regressions resulting from the rewrite.
  • The highly anticipated scrolling feature has been added. Swipe up and down with two fingers to scroll. Horizontal scrolling is not yet supported. Different operating systems (and perhaps different applications within an operating system) have different ideas about what a suitable unit of vertical scrolling is, so at present scrolling is a bit too fast on Windows and a bit too slow on Mac.
  • You can now simulate a right mouse button press by tapping with two fingers.
• Miscellaneous bug fixes
  • The TCP socket did not always shut down when the Valence activity finished. Fixed.
  • Valence would crash when the screen was reoriented while an alert was being shown. Fixed.

Update: July 18th, 2011

I've made a beta release of Valence available on the Android Market. There have been no significant changes since the previous release, other than removing a bit of debug logging.

Update: August 27th, 2011

I posted a new release to the Market with an important bug fix, and a few minor changes:

• Users reported a bug that was preventing some handsets from sending right-clicks by tapping with two fingers. This bug has been fixed.
• I updated the touchpad text and the help file to reveal the ability to right-click and scroll by using two fingers.
• I dropped "Beta" from the app's name.

Downloads

• Download Valence Beta from the Android Market

In my previous post, I mentioned my frustration that certain Android phones (including my HTC EVO) cannot receive multicast datagrams. I'd like to get feedback from my friends and colleagues about multicast support on their phones, so I wrote a simple app for testing multicast.

The Multicast Test Tool continually monitors the network for Multicast DNS (mDNS) packets while the app is running in the foreground, and presents the contents of these packets to the user. The app also allows the user to perform simple mDNS queries on the local network. If you run the app and touch the "Query" button, it will query for the default _services._dns-sd._udp.local name, which will solicit mDNS responses from devices on your network that advertise services via mDNS.

Mac and Linux machines will respond to this discovery query. Windows machines will also respond if you are running iTunes and have checked "Share my library on my local network" in the preferences. If you think your network may not have any of these devices which support mDNS service discovery, you can run the attached perl script to transmit a gratuitous mDNS packet.

You can also query specific hosts using the format <hostname>.local. My home network is always buzzing with mDNS traffic, so on my multicast-capable virtual machine I see lots of activity without performing any queries.

If you see any activity at all, it means that your phone supports multicast. If you see no activity, then your phone likely does not support multicast.

Downloads

• multicast_test.apk - This is the Multicast Test Tool app, which you can download and install on your Android phone.
• send_mdns.pl - This perl script transmits a single mDNS packet, which should be detected by the Multicast Test Tool. This is only needed if you don't already have devices on your network (Macs, Linux boxes, etc.) which implement the mDNS service discovery protocol.
• multicast_test_tool.tar.gz - The source code for the Multicast Test Tool. This is only needed if you are curious about the inner workings. (Added 2011-02-02.)

It looks like some (most?) HTC phones running Android, such as my HTC EVO, are not capable of receiving multicast or broadcast datagram packets over the Wi-Fi network. This means that apps which rely on such communication will fail, often with no indication of the problem. From the app's perspective, no obvious error is happening — it can only assume that no other devices on the network are transmitting such datagrams. Multicast communication is becoming increasingly common as a technique for devices to discover each other on a network, and the absence of this capability represents serious breakage that leaves apps crippled. Examples of resources an app might use multicast to discover are:

• iTunes music libraries
• entertainment center components such as Roku and AppleTV
• network shares
• other devices running the same app, to set up multiplayer games

Without a functioning multicast capability, these apps either don't work, or require the user to manually configure the IP addresses of the other devices.

There is speculation that HTC's restriction is an attempt to save power and prolong battery life — it takes energy to process a network's many multicast and broadcast packets, many of which are useless to the device and its applications. However, Android already has a facility for allowing prudent use of multicast when needed. Applications acquire a WifiManager.MulticastLock for the duration of their multicast needs, which causes the Wi-Fi chip to stop filtering multicast packets until the app releases the lock. Thus, multicast processing in the software TCP/IP stack only happens when it's really needed.

This is a bit frustrating, as I'm currently tinkering with some multicast code. Some people have been able to fix the problem by rooting their phones and replacing the /system/bin/wpa_supplicant binary with a stock version. This doesn't seem to help on my EVO. To add to the frustration, it can be difficult to test multicast in the Android emulator. I finally had some success in testing multicast code by running android-x86 in a VirtualBox virtual machine, with the network configured for bridging.

References:

• HTC Desire Android 2.1-update1 does not receive udp broadcasts (Android issue #8407)
• Multicast on Android 2.2 (Stack Overflow)

HTC EVO, shown with the optional extended-life battery.

Earlier this year, I ditched my antiquated Treo 755p for the new hotness in mobile phones—the HTC EVO, an Android phone available from Sprint. Unlike my previous phone, which would last a full week in standby without recharging, the EVO gulps battery like there's no tomorrow. In fact, the limited battery life has been the primary complaint about the phone. Nobody seems to know for certain why the battery life is so poor. Maybe it's the beefy 1Ghz Snapdragon processor, the large LCD screen, or something else entirely.

I was recently traveling in a rural area, and noticed that my EVO's battery would only last halfway through the day. All mobile phones consume more battery in rural areas where they must transmit with greater power to stay in contact with faraway towers. However, the extent to which the EVO's battery life problems were exacerbated by the higher transmitting power got me thinking that perhaps at least some of the battery drain may be due to software which occasionally causes the radio to transmit.

All modern smartphones have various software components which periodically execute in the background, unbeknownst to the user, and sometimes activate the radio to chat with their buddies in the cloud. Some of this chatter may be useful for the user, such as having fresh mail delivered immediately, while other chatter may provide little or no value.

I got curious about what the phone's software was actually doing while I wasn't around, so I captured and analyzed all the network traffic from my phone over a period of four hours. I accomplished this by turning off the 3G radio and packet sniffing the Wi-Fi network. (I didn't have an easy means of sniffing the 3G traffic directly.) I have my phone syncing with my Google account, Google Latitude, a couple of mail servers, and Facebook, and this is reflected in the captured data. I did not touch the phone at all for the duration of the test, so this traffic represents only non-interactive use. Here are the main traffic categories, sorted by decreasing packet count, with local network maintenance traffic (ARP, DHCP, etc.) removed:

Lots of background network activity can be minimized by tweaking the sync settings.

In total, about 1.2 MB of data was exchanged in 2277 packets: 186 KB (1270 packets) were transmitted, and 1002 KB (1007 packets) were received.

The Facebook sync seems to be the biggest data hog of these background services. Every two hours, it is downloading information about all my Facebook contacts, as well as fetching profile pictures of most of them (whether or not the profile pictures have changed recently). The sync frequency for my "Facebook for HTC Sense" program can be changed in the settings, so I changed it from every two hours to once a day. After Facebook, the biggest consumers seem to be Latitude and my mail servers. I knew what I was getting into when I configured those, so I can't complain. I didn't even realize that I had Yahoo's "Stocks" program set to sync, so I turned that off altogether. The Sprint Zone sync frequency can be changed in the Sprint Zone application itself, although I haven't found a way to limit the Sprint TV syncing. The next time I'm traveling in a rural area, I should remember to turn off the "background data" and "auto-sync" items in the sync settings.

There are a few possible flaws in using this method to predict battery consumption. The exact relationship between network activity and energy usage is not clear to me. For example, is the transmitter only active for just long enough to send a packet, or does it remain on for a bit longer in the hopes of another packet becoming available in the send queue? What would consume more energy—a burst of 100 kilobytes, or 100 short transmissions of 1 kilobyte each, spread throughout a period of time? Is it more meaningful to consider packet counts, or the total byte count? Are some of these services only transmitting because they know the network connection is Wi-Fi and not 3G?