Andrew Hsu: The Invention of TOUCH & What’s Next

Bill Gates stood on the stage at the (now-defunct) Comdex show in Las Vegas in 2000 with his schoolboy smile touting the new “tablet PC.”  Penned on the tablet in Bill’s handwriting was “Tablet PC is SUPER COOL!”  Behind the stage a backlit sign read “experience the evolution”.

Microsoft evolution never became a revolution because the company’s disparate and factional divisions failed to work together to vision and implement a turnkey experience.

The revolution happened in 2007 with the launch of the iPhone.

(As with most industries) evolution is often interrupted by black-swan revolutions. Sound (voice communications), touch (pinch and zoom navigation), sight (Heads Up Display [HUD]) all changed the way consumer used the phone and is one of the gating factors in technology adoption.

Knowing what technology will help us evolve and what technology revolutionizes is more of a human insight that a science. Ergonomics help us rearrange the digital furniture; however, changing the way we connect with this communication device is profoundly human. What is beyond touch, what is the next revolution?

A Short History of Touch

Although Gates told reporters off stage in Las Vegas that how excited everyone was in Redmond (Developers were checking the tablet out to play with – “a very good sign,” he said) 6 months later warehouses were still full of the Tablets. Q2 shipments had plummeted 25% with a meager 100,000 total units sold.  Mike Magee, technology writer for the Inquirer wrote despondently that “This is another classic case of IT firms thinking they know what technology people will like, and failing to take off the blinkers.”

Touch appeared back in 1971 over a ten year period began to appear in the form of  infrared technology (such as the Hewlett Packard 150) which show up in various military applications. The IR matrix of beams are used to detect a finger touching the screen.

But the IR technology was expensive and the technology gained more mainstream adoption was “resistive touch”.

It was a simple concept. Resistive touch screens were built using two layers of conductive material (Indium Tin Oxide). The two layers were separated by a small pocket of air. An action was triggered when a stylus, or other object, pressed the top layer into contact with the bottom layer.

The limitation was it was like a pin board. You could tell the device where you were move the point of contact. But it did not have multi-touch functionality essential to pitch and zoom navigation.

Mass-market adoption was not an option:

  1. The screen wore out
  2. Required a stylus pen for accuracy
  3. The air pocket made the screen appear hazy
  4. OEMs had to build a clunky hole in the casing (as the top of the resistive sensor had to be exposed to user’s input)

This is the technology that Bill Gates was holding up at Comdex in 2000*. The unit’s resistive touch stylus was used to input into clunky dialogue boxes to input text and commands. The entire project was “resistive”. The Office team refused to build for the unit adding to the painful UX.

*[A technogeek aside: Microsoft’s Surface touch solution uses Frustrated Total Internal Reflection (FTIR)]

Meeting Andrew Hsu

In 2013, I ran an event on connect screens in New York. I wanted to tell a story about the importance of the screen in the evolution of mobile phone design and adoption. I invited Professor Donnell Walton from Corning Glass, as well as representatives from Microsoft’s Surface team, Google Glass and was looking to find a speaker to explain “touch”.  Maybe I could locate someone from the scuttled Apple Newton team?

I found, much to my surprise (like an anthropologist that finds that we did not evolve directly from monkeys) that the precursor to the 2007 Apple iPhone was a skunk works project headed up by an engineer called Andrew Hsu.

Andrew developed and patented a capacitive touchscreen suitable for mobile devices way back in 1999. He developed a system which computes the location of a user’s fingers based on how they change the capacitance values of an invisible matrix of electrodes.  The capacitive touchscreen did not suffer from the various user experience drawbacks of the resistive touchscreen – it does not wear out, it does not cloudy the underlying display, and it does not require a big hole to be cut into the device casing.  But most importantly, it enables natural finger input.

This capacitive touch is not a mouse click. It is not a data poke with a Stylus. Andrew Hsu’s touch allowed us to communicate in a very human way with pointing and pinching space.

Don Norman is often quoted about touch.

“We’ve lost something really big when we went to the abstraction of a computer with a mouse and a keyboard, it wasn’t real . . . swiping your hand across the page  . . . is more intimate. Think of it not as a swipe, think of it as a caress.”

While mobile success is almost always based on interface and usability, it took seven years for Andrew Hsu to convince the industry to adopt the technology. Revolutions come in simple packages: text messaging, Apple’s mobile application SDK, gesture-based gaming.

We talk about the consumerization of technology; touch was the humanization of technology. In a world where data appeared cerebral and uninviting, we suddenly can interface in this data and content as we do with real object. The physical world became extensible and less scary.

From Click to Pinch & Zoom

In 2006, handset manufacturer LG trialled launched capacitive touch with their designer Prada phone. The LG phone had all the correct ingredients – capacitive touchscreen for intuitive finger input, high resolution display, and one of the first graphics co-processors in a handset. Prada brought style to the table and LG brought the insight that touch that would ultimately inspire the new mobile consumer.

But we had to wait one more year.

When Jobs returned to Apple he shut down the Newton project.  This legacy 1993 technology had poor handwriting recognition and had little traction in the market. But Andrew Hsu’s capacitive touch appealed to Steve Jobs UI sensibilities.

As a post Newtonist, Jobs once said “we are born with five styluses on each hand”.

When he introduced the iPhone, we knew that being able to move large format data on a small screen with a pinch and zoom changed the way the consumer saw their mobile device.  Where Steve Job went further than touch was his insight in designing a full edge-to-edge screen that had the dimensions of a letter size piece of paper.  The screen called out to be touched, worked on and paged through.

Although touch revolutionize the phone and lines weaved around the block for new product releases of Apple new “human” interface, the consumer was still nose-to-screen, bumping into lamp posts while elegantly navigating data a hundred miles away.

“Bump” (the file exchange application recently acquired by Google) and other application including NFC payment extending this love of tactile interface by promote social touch between other phones and public devices such as POS.

Gesture: Moving Beyond the Cool?

While touch is an important sense, sight is essential for navigation. The next revolution is to make data come to live seamlessly in the real world.

When we talk about HUD, we think of the new Google Glass and the opportunity to integrate data into our line of site. In parallel, see the world and the data behind it. Integrated cyborg solutions like Google Glass and future visions of embedded epidermal circuit (seen in Total Recall).

Microsoft had the lead in a new HUD interface using gesture.  XBOX Kinects was the one product that Microsoft was seeing growth in the consumer sector. However, the leviathan was unable to make this a multiscreen strategy fast enough.

Moving gesture elegantly to PCs and window phones never happened. There is a Kinect for Windows but it lacks the software for controlling the interface.

The Leap motion controller is a step forward.  A small multiscreen sensor box not tied to console in the dean but with the ability to tether like a dongle to a wide variety of screens and deliver better sensitivity to Kinect. It has multiple commands down to finger level accuracy.

Andrew Hsu still believes that touch is less ambiguous on the consumer navigation intent. “How can you disambiguate between “accidental” and intentional gestures.  The beauty of touch interaction is that you basically get user intent for “free” – a user typically only touches the device when he/she wants to interact with it.  The cases of accidental activation are much lower and easier to reject.”

Arguably HUD is a solution looking for a problem. Like the inspired Seque cycles, the inventor’s goal was to develop an urban consumer transport vehicle but he failed to get significant adoption. The Segue has now found a home with urban tourist touring groups and airport police. Why? It provided an elevated view with minimal multitasking: Ideal for tourists and law enforcement.

Andrew agrees: “What these technologies really need to address is what sort of “problem” they are trying to solve.  That is, with capacitive touchscreens, there were certainly a number of value propositions that arguably were superior to the previous (resistive) solution that helped transform/enable touch input.  Natural gestures (HUD) is still looking for a compelling value proposition”

Google Glass is a platform without a certain home. Will “super cool” it has not inspired the consumer. We have not seen the “a-ha!” that Jobs brought to the touch. We know new more intuitive human interfaces are coming. But we need a Steve Jobs to take the technology and humanize it for intuitive consumption.

Gary Schwartz is the CEO of Impact Mobile. Having been at the frontlines of the mobile industry for over a decade, Gary is the author of two books, “The Impulse Economy: Understanding Mobile Shoppers” and “Fast Shopper. Slow Store: A Guide to Courting and Capturing the Mobile Consumers,” both of which highlight the current state of the mobile commerce space and chronicle the significant impact that mobile is having on consumers, retailers and brands. Gary is also a chair emeritus for the Interactive Advertising Bureau and the Mobile Entertainment Forum NA and global director of the Location Based Marketing Association.

The 2014 Ad Game Changer: Digital Maps

Gary Schwartz (16 September, 2013)

Of all the widgets and long-forgotten apps on your phone the one with most mobile mindshare is your map app. We have become a mobile society, and in the 2010s, map apps personify our wanderlust. When we open our mobile map, we have intent, direction and purpose. It is vitamin “M”: the ultimate upper and highly addictive.

And map real estate is hot: Apple buys Locationary, Embark and HopStop; Google buys Waze;  Bing is rumored to be in talks with FourSquare; Zillow, the map real estate tycoon, buys EasyStreet, and indoor mapping app company, Aisle411 raises a hefty seed round in the valley. As OEMs beef up their services, we are entering a new phase of map building. Location has always been a data grab. Now, the industry is starting to focus on monetising subway stops, street corners and highways across the world.

The principal challenge is that maps are a new and unique advertising paradigm, and the incumbent search business models, mostly designed for the web’s previous era as a stationary, desktop experience may need to be adjusted.
Galileo to Google

Google Maps, the grand daddy of digital mapping, was born in 2004 as a skunk works project by two Danish brothers in Australia.

First designed as a heavy client app, in 2004 the software came full circle as Lars Rasmussen and his brother were acquired by Google after making a web-based pitch. The same year, Google acquired Keyhole, Inc. and proceeded to use Keyhole’s mark-up language to launch Google Earth in 2005.

During the next five years, Google started to revolutionize digital maps. It is quite possibly the most exciting innovation effort by the company. Not since map mavericks Ptolomy, Copernicus and Galileo has mapping accelerated so profoundly. Within a few short years Google has redefined the way we see the world around us.

Google Maps rolled out road directions in North America in 2006 and their PC-based maps became the pre-GPS automotive assistant. However convenient and customizable, Google maps for the desktop were a print-on-demand version of London’s A-Z pocket maps. In many ways, a Google map printed out before a trip was no different from John Ogilby’s 1675 Britannia detailed strip maps that travels bought to find their way from Norfolk to Newmarket with inns, stables and other points-of-interest as well as clear directions and distances clearly marked. They balanced behind the horse on the coach seat as our laser-printed version would sit on our car dashboard.

But the small screen was the true game changer. The capacitive screen touch invented by Andrew Hsu, combined with the pinch-and-zoom mobile interface developed by Apple made complex map navigation simple, user friendly and, most importantly, mobile.

With multiscreen map adoption, Google Maps expanded. The company launched in Latin America and Asia, and started the subterranean mapping of subways in 2007. In 2008, a view from space; in 2009, a POV from the street and 3D rendering.  And more. Google mapped canals and bike paths, endangered forests and the ocean floors,  the moon and Mars and the ultimate conquest, Macy’s in-store experience.

This was phase one: Build a dominant innovative platform with simple APIs, establish market stickiness and trust by the point-A-to-point-B public.

Now add metadata

Google+ Local launched in 2012, allowing users to post reviews and images into pages hosted by third party sites. This year maps are becoming more customized, providing location-specific information on points-of-interest. While Google has maintained a focus on road navigation with its 2013 acquisition of the crowd sourcing road-warrior Waze software, the operative term on the new Google map is “explore.” Explore photos, recommendations, and restaurants.

Maps plus Google Glass makes the possibility of on-the-go exploration more immersive. Using the Google Mirror API developers can feed real-time GPS info and pre-rendered map images into the eye window of Glass wearers for “dexterous” driving, cycling or walking to the local mall. Glass becomes “a Segway for your head.” And taking maps to the edge of utility: Google Sky (which maps the stars based on your GPS location and vision angle) can be integration with Google Glass to show the outlines of constellations through a transparent filter to view the night sky.

Wow!

And then at the end of this epic journey, Google announces local advertising. Google Maps now allows short sections of advertisements to be placed directly onto the map itself. Local advertising is one of Google’s core business and Google Maps ad purchases are made through the same Google AdWords auction that buyers are already very familiar.

For Google this is simply a terrestrial version of browser-based search. When a consumer enters “Starbucks” in her browser, she finds links to buy “Starbucks Instant Coffee Bundle” on Amazon.com. When a consumer enters Starbucks in Google Maps, she finds local Starbucks to get the real deal (or if Tim Hortons is bidding, an ad for a competitively located Timmy’s coffee store.) Both these use cases involve path to purchase. One is virtual, the other is proximal.

Google hopes Map-based ads will follow the same digital success that Google has had with its search-based ads. Instead of auctioning AdWords at point-of-search, Google auctions ads at point-of-navigation.

Ptolemy what? There has to be more than just that. We’re just not fully there yet.

Don’t forget the Big Apple

Apple recognized the value of maps and knew that they had a Trojan Horse lurking in their mobile operating system in Google Maps. Google’s map app had become the dominant phonetop service with the most unique visitors of any app in-market. When Appl­e launched and preloaded its own proprietary map app in August 2012, Google’s traffic dropped making Facebook the winning app for unique impressions as well as time spend.

(After a few geographical faux pas) Apple started to establish its own relationship with the map consumer. But Google Mappers are loyal. When Google launched its new map app for iOS 6 in December there was a 30 per cent rush of Apple folk upgrading to the new operating system (MoPub.com). Affinity to a map app had influenced these consumers’ mobile behaviour. Quite remarkable.

However, Apple is committed to build a map following. While the company no longer needed to pay licensing to Google, which was good, the key reason for ousting Google Maps was that maps had become a data pillar. By replacing Google, Apple had direct access to a wealth of consumer data and potential advertising revenue.

Yahoo! Maps, Bing Maps, Nokia Maps, and MapQuest all use the NAVTEQ electronic map feed (best known for its automotive navigation services), and like Apple now, they own their own consumer data layer, which is crucial for generating advertising and marketing revenue on maps.

Bing is the major map contender.  In September, the company added 13 million square kilometers (316TB) of aircraft and satellite photography to its service. Microsoft’s large investment in Facebook in 2007 ($240 million) led to the 2013 decision adopt Bing as FaceBook’s mapping and search provider. To do this effectively, and compete with the market-leader, Google, Bing needs to beef up and differentiate its map offerings. Bing has already rolled out “Local Scout” which helps consumers find food and fun across all its screens. Rumors of a FourSquare acquisition (or possibly financing) may be part of this grand strategy.

(Microsoft acquisition of Nokia did not come with their HERE maps assets. Nokia’s HERE maps include road networks, traffic patterns and urban landscapes and as licensed by major properties such as Garmin, Oracle and Amazon.com. Will Nokia take the lead as the premier mapping and location services across different screens? Will they just sell off the asset to Apple after the Microsoft acquisition is complete?)

And the open-source mapping movement is also growing. Washington D.C.-based startup, MapBox, provides more custom navigation and interesting APIs built on top of Open Street Maps. Open is good and allows developers greater flexibility and affordability; Foursquare uses MapBox’s services to display its users’ check-in histories. However MapBox is not preloaded on your Android or Apple phone and while they have an iOS mapping SDK they have no Android footprint. MapBox will certainly play a roll as an embedded technology in sites all over the web; however, it is unlikely that they will be a standalone consumer utility on top of your phone and tablet.

With the proliferation of WiFi networks in retail, vendors such as Cisco drive mobile mapping solutions for shoppers that join the free network. The maps allows for hyper-local, custom mapping that includes restroom as well as promotional information on retail stores.

All of these digital map offerings are entering the mainstream at a time when advertisers are questioning consumer engagement on mobile, and trying to understand how best to follow their consumer in a contextual and relevant manner. Brands and retailers are re-evaluating the way we sell and, more importantly, engage across multiple screens. Their assumptions on path-to-purchase, built during the era of the desktop web, are no longer fully valid and reliable. The classic consumer narrative of home-to-store has changed and retailers and brand can no longer simply hire a Director of Shopper Insights and hope for the best.

Advertising and marketing is about providing a consistent message at aisle, and checkout, wherever the shopper finds the retailer. If the advertisers wants to get back in the game, possibly the most exciting place to be right now is on the map. When the consumer and shopper opens their map app when they have intent to meet someone, go somewhere or buy something. All this drives commerce. Maps provide unadulterated path-to-purchase.

Narrative: Going beyond advertising

So what is the new advertising paradigm for maps? Maps have layered functionality: terrain, roads, satellite, traffic, public transport and images. Then there is the exploration layer: recommendations and general points-of-interest. And now Google has provided an additional local advertising layer.

However, the adding an advertising layer may not prove to be effective in a map environment. There is more value to the exploration layer. Yes, maps help us move in a utilitarian fashion from Point-A to Point-B and that is why Waze and other transit acquisitions have been so important.  But maps also have a very non-utilitarian function.

Maps help the consumer simply “explore” and is what will ultimately connect map users to brands and content owners. Maps tell stories because precisely they have a beginning and an end, and are defined by intent and clear purpose.

Instagram, Twitter, and Facebook already situate the user’s photos and comments at a latitude and longitude: a country, a city, a bar. However these social graphs are not map applications and location is an important but secondary metatag.

The opportunity is to build a new bespoke map layer for brands and content owners. Think map first.

Startups such as Findery and CityMaps have map based UGC (user-generated content) engines. Where is the content input engine for brands? How can brands visualize content and actively map this data across all their customers’ screens?

One company called Mapiary, based out of Singapore, is developing the tools to allow brands and retailers to layer rich navigation onto the map. How can Unilever’s Becel margarine be more relevant to power walkers globally? Or how can Heineken weave narrative into a city pub crawl? Diageo, can map a DJ tour for Smirnoff. The NYTimes can map their 36-Hour travel series in a rich contextual manner. This is new digital cartouche and as important as the underlying map.

Where is the new vision of brand advertising? After all the innovation that Lars Rasmussen (Google Maps) and John Hanke (Google Earth) brought to maps we surely need to go beyond paid search models and allow owned content to become a rich and valuable layer in the 2014 map.

BNN Interview on the “Battle to Monetize Maps” discussing the positioning of Apple, Google & Microsoft. http://t.co/M1Pa0Qcijd 

Gary Schwartz is the CEO of Impact Mobile. Having been at the frontlines of the mobile industry for over a decade, Gary is the author of two books, “The Impulse Economy: Understanding Mobile Shoppers” and “Fast Shopper. Slow Store: A Guide to Courting and Capturing the Mobile Consumers,” both of which highlight the current state of the mobile commerce space and chronicle the significant impact that mobile is having on consumers, retailers and brands. Gary is also a chair emeritus for the Interactive Advertising Bureau and the Mobile Entertainment Forum NA and global director of the Location Based Marketing Association.