Replaced by TR #355
This is the original version as distributed widely in early 1994. The earliest version was distributed in the Media Lab in 1993. A postscript version added images in June 1995.
Unpublished paper meant for the popular press
The Cyborgs are Coming
or
The Real Personal Computers
by Thad Starner [cyborg@media.mit.edu]
(submitted to Wired)
People look at me strangely when I walk down the street
these days. However, I'm not particularly surprised; I have a box
strapped to my waist with wires reaching out to my hand and up to my
eye. I often hold silent conversations with myself, electronically
taking notes on the world around me. Occasionally one of my
observations triggers electronic memories and gives me new insights.
No wonder people look at me strangely. You see, I'm one of the
world's first cyborgs.
We are on the edge of the next stage of human development: the
combination of man and machine into an organism more powerful than
either. Almost every user of a computer will be affected in some way:
students, secretaries, lawyers, doctors, scientists, stock brokers,
and CEO's, just to name a few. While the technology necessary for
this merger may currently look strange, the hardware is obtainable
from today's off-the-shelf components. For $3000 you can strap on
prototype technology that makes present PDA's (personal digital
assistants) pale in comparison. In mass production that price could
fall to around $1000. Currently, the hardware consists of a small,
light graphics display called the Private Eye(R) (the current version is
720x280) that fits over an eye in a pair of sunglasses, a one-handed
chording keyboard (which functions like a full 101-key keyboard), and
a small DOS-based computer which fits on the waist (in my case, the PC
includes 85M of hard drive, 2M of RAM, and several ports including a
PCMCIA). For a little more money, a cellular phone and modem can be
added (for those net addicts). While almost plebeian in design, the
combination of these inventions points to a very powerful paradigm in
human-computer interactions. Furthermore, three multi-billion dollar
product and service areas will be developed by such technology.
The Vision
Science fiction has foretold the merging of man and machine
for many years. Cyborgs with minds partly consisting of silicon
are almost commonplace in today's fiction. Usually, these characters are
portrayed as the dark side of humanity, dependent on prosthetic neural
circuitry to continue life. However, some writers have seen these
devices as voluntary additions to the human host, augmenting but not
supplanting the intelligence already there. This field of
"Intelligence Amplification" (to borrow a term from Vernor Vinge) is
the topic of this article. While such a term brings visions of direct
brain interfaces, nothing so grandiose (and difficult) will be discussed
here. Instead, a simpler interface will be described which has
similarly powerful properties of persistence and consistency.
In recent years, computers have gotten smaller, lighter, and
more powerful while consuming less power. Fueling this trend is a
great base of users who rationalize a need for computing power while
traveling (or at least while roaming within an organization). Often
these notebook machines are used for such mundane tasks as "To Do"
lists, appointments, and business contacts. However, without such
functions the user can be paralyzed. In fact, the pen-based computer
community is trying to fill the need of the users who find notebook
computers awkward for doing these tasks (unpacking, powering it on,
booting, finding a place to type, etc.). However, even these machines
are still inconvenient in the real world, for a variety of reasons
that will be examined later. An ideal interface would be with the
user all the time, listening to the user's real world interactions and
updating appropriate files automatically. Going even farther, the
computer should monitor the virtual world and notify the user when
appropriate (important e-mail, the value of gold dropping $100.00,
etc.). While this goal is extremely hard for many reasons, first
level approximations to such man-machine relationships can be made
using relatively simple hardware and software.
The Private Eye
Made several years ago, the Private Eye is one of the most
unrecognized revolutions in display technology. This small, 1 oz.
display uses a single row of 280 LED's and a scanning mirror to
display a screen of 720x280 pixels to the user's eye. More modern
versions have resolutions of 1024x768 pixels. The image is crisp, and
the focus can be put anywhere from 10 inches to infinity. Since the
display is worn close to the eye (for example, in a pair of
sunglasses), the projected image is equivalent to a large screen
display. Due to the "sharing" effect of the human visual system, the
user can see both the real world and the virtual at the same time
(some variations on this theme use two Private Eyes with half-silvered
mirrors so that stereoscopic cues can be used and both eyes can see
the virtual and the real at the same time). Furthermore, since there
is no large glass or plastic surface to scratch or bend (the actual
display surface is ~ 1" x 1"), the Private Eye is more robust than
most other portable displays. The unit is designed to withstand a
three foot drop, and, in my experience, can handle even rougher
treatment. In fact, the Private Eye would not be too difficult to
ruggedize for use in the military. Additionally, since the display is
kept near the user's head where humans are much more careful with
respect to impacts, the Private Eye is much less likely to be exposed
to damage than the LCD screens in the stereotypical PDA.
While this graphics display is not as powerful as the direct
brain connect interfaces described in fiction, the visual system can
process an enormous amount of information and is thus a great primary
interface for receiving data. In addition, the overlay of graphics on
to the real world allows virtual annotation of real world objects.
A Revolutionary User Interface: The Keyboard?
So far, keyboards on notebook and pocket computers are either too
large for convenience or too small to use. This is a direct result
of assuming the standard QWERTY keyboard is good for portable computing.
Manufacturers are afraid that it would take users too long to learn a
new way of typing. However, beside me is a one-handed chording
keyboard which anyone can be taught to use in 5 minutes. It is
certainly much easier to learn than the QWERTY interfaces (the letters
actually go in order-"abcdef..."-but are arranged so that speed is not
particularly limited). In an hour, a beginner can be touch typing.
In a weekend a speed of 10 words/minute can be obtained. Shortly, 35+
words/minute can be achieved (my personal rate is around 50
words/minute with a macro package). This is the Twiddler keyboard
from HandyKey (addresses are included at the end of this article). It
even includes a tilt activated mouse. The Twiddler is but one of many
one-handed designs out there. Some designs allow instant access to
both hands if necessary (the Twiddler straps on to one hand). This
feature may be very desirable in medical fields. In any case, these
devices allow the use of full-featured keyboards anywhere (including
walking down the street in the rain). When finished, they can be
stuck in a pocket or left on the belt for easy, instant access. Not
only are these keyboards convenient, they do not require much CPU
power (unlike handwriting), always correctly recognize a user's input,
and can take an amazing amount of abuse (I have kicked mine into a
door, stepped on it, and gotten it wet, etc.).
Putting It Together
When the Private Eye and a one-handed keyboard are combined in
a computer interface, the result overcomes the limitations in
screen-size, access, and user input imposed by many of today's PDA's.
The user can continually see both the real world and the virtual in
his everyday work. The virtual world can be accessed even while the
user is walking down the street, attending a cocktail party at a
conference, attending a patient, giving a PhD defense, or taking a
quick lunch before going back to Wall Street for more trading. The
interface is persistent and reliable (due to the simplicity and
packaging of its parts). Just this beginning system has many
possibilities, and I would like to dwell on these applications as well
as marketability of the current system before moving on.
Simple Applications
By adding basic communications through radio or cellular
technology to the base system, many applications present themselves.
As a computer professional, I find the ability to log into my computer
system anywhere, anytime a serious boon. Even if communications are
not possible for some reason, the ability to edit text and read mail
locally is a major asset. Computer system administrators could find
such technology invaluable for detecting and fixing employer problems
without having to be physically present. While such interactions are
possible on a notebook computer, the added portability and the
persistence of this interface allow for better access. Students can
take notes in class without having to glance down at their screen.
Lawyers can be in communication with their office databases and
support staff while cross-examining a witness. Repairmen can make
inquiries and orders to the home office without interfering with their
work. Health care providers can query databases for precedents or
consult a remote physician while examining their patients. Brokers
could transfer commodities, offer bids, or consult without shouting to
be heard (and, maybe someday, even trade while away from the floor).
Racing enthusiasts could bet and monitor their winnings without being
at the track. These are but a few of the many applications that are
possible. In fact, every day I wear my interface people find more
uses for it.
The Billion Dollar Hardware Business
While this beginning system may seen clumsy, it is quite
usable. At first, only a certain breed of technophile or
time-critical information consumer would be interested in looking odd
to gain the power and convenience such an interface would allow (with
the current system, I am often mistaken for a telephone repairman).
However, there are a growing number of individuals out there that
qualify. In fact, marketing organizations have identified a new
niche: the computer professional. These individuals use a computer
every day and find it essential to their work. Furthermore, according
to some estimates, 50% of these individuals earn over $100,000 a year.
Even if the system was just sold as an expensive toy, a market may
exist. More seriously, due to the unsuitability of handwriting
interfaces for many tasks and the familiar DOS feel to my initial
system, it may usurp the large market the pen manufacturers predict
for their systems. Customization may leverage the concept into the
service industry (inventory control, quality management, railway
conductors, telemarketers, etc.) When improvements such as speech
recognition, smaller designs, etc. come along the market will expand
to a broader band of users, much like the notebook market has
(notebook systems are presently outselling desktop models). Here is a
chance for entirely new computer product lines, with upgrade paths
every two years. Along with these lines come the necessary support
hardware, such as digital modems and radio gear.
The Billion Dollar Communications Business
The Internet, television cable, and cellular telephone all
started as very small systems. Today these communication mediums are
almost institutions. Providing cheap and reliable wireless digital
communication technology will become an incredible source of revenue.
Even with just the notebook computer paradigm, many foresee a
tremendous growth. With the addition of wearable computing (not to
mention intelligence amplification), these figures can only improve
exponentially. In fact, technology for both long range and very short
range communication will be in high demand, since, after commuting to
work, the user's wearable computer should automatically hook into the
office work environment at higher bandwidths to help the user with
normal chores. While this may not supplant the need wired interfaces
to a powerful desktop system, the wearable can still help it's user
operate these more powerful machines on a personal level, if only to
separate more casual work (e-mail, weather updates, phone calls on the
wearable) from concentration intensive work (CAD, accounting,
visualization on the desktop).
Another communications issue is the interfacing of the
different parts of a wearable together. While the present interface
is wired, it is easy to examine a low power communication system to
wirelessly combine the keyboard, display, and computer. In fact,
there has been some discussion of using the body itself as the
communications carrier. According to some initial experiments by
La Monte Yarroll, speeds as high as 1 Mbaud may be possible by driving
a 5V signal across the skin. More conservative methods may include
infared or low power radio frequency.
The Billion Dollar Software Business
While the initial systems may be DOS or Mac based, the new
interface paradigm of persistency allows radical changes in software
design. The new software should make the user interface simple and
consistent in most situations. An improved level of user competency
may arise from the increased use of the persistent interface (this is
happening anyways as our children are growing up in a computer
literate world), so these interfaces may become more complex than
ever. A particular change of software design will be in determining
when to interrupt the user for especially urgent incoming information
or clarification of user input. The goal is to improve the user's
productivity, not overwhelm his sensors. The research/software
product field of Intelligent Agents may go far in addressing this
issue. In fact, an artificial agent will be presented later as a tool
for the author's wearable computer. This field will exist whether or
not this particular hardware platform is created. With an increasing
amount of information being generated, intelligent tools will be
necessary in the coming world.
Furthermore, until communications transponders become
ubiquitous, software will be needed to make the transition from
connected to disconnected use transparent. Much theoretical and
practical work has gone into such systems already, but the commercial
implementations lag behind.
The Cyborgs Are Coming: The New Computing
One of the simplest, yet most poignant, applications of this
"wearable" technology is augmented memory. Today, many computer users
already utilize the excellent memories of their computers for storing
phone numbers, addresses, and "to do" lists. However, many of these
users are then helpless away from their terminals (or have to lug out
their notebooks or, at best, their palmtops, each time they want to
check something). With durable wearable technology, these users can
check and update their schedules wherever they may be and without
interrupting whoever they may be talking to at the time (especially
useful for storing e-mail addresses at conferences). In fact,
reminders, meeting agendas, grocery lists, and lecture notes could be
automatically or semi-automatically overlaid on to the real world as
appropriate. These applications just scratch the surface of what is
possible.
Wearable computing allows a symbiotic relationship between
computer and human which combines some of the strongest advantages of
both: the creativity and intuition of a human with the precise storage
and searching capacity of the computer. Suppose, that a reader of
this article has the interface as previously described. As the reader
scans the text (supposing, for the moment, that this article is in
paper form and not on-line), he types in notes, unanswered questions,
and comments in one window of his word processor. The reader's
Remembrance Agent (RA) (an intelligent, adaptable piece of software
that specializes to a user's needs) listens to the input and
immediately conducts a search through the user's directories (local
and/or remote) for files with similar contents. In another window of
the word processor, the RA reports appropriate lines from files
found in its search. These lines are ranked according to some measure
of "usefulness" that is either directly programmed into the RA or
learned over time. In this manner, the reader can quickly be reminded of
similar pieces of information obtained in the past. Through these
small memory assists, the reader can compare two people's views,
confirm statistics, or generate entirely new ideas synthesized from
the foundations laid by others. Furthermore, the Remembrance Agent
can suggest files for the storage of this article and the reader's
notes on it, possibly improving the reader's organizational skills.
An initial implementation of this software has been completed, and a
more sophisticated and powerful version is underway.
The implications of such a system are tremendous. Imagine
college students having immediate access to their education for the
past 20 years, reporters and police detectives who can interactively
and possibly automatically search for clues and leads, stock brokers
whose systems automatically listen to news feeds for information that
might affect prices, scientists with automatic access to a common
storehouse of information which may spur new contacts and discoveries,
CEO's with up-to-the-minute reports on their own and competitor's
companies, lawyers whose Remembrance Agents discover a precedent based
on a new twist in a court room trial, and doctors whose description of
a patient's symptoms finds a match with a rare case reported on the
other side of the world. The list goes on and on.
Makers of PDA's have been suggesting similar possibilities for
several years now. Many have recently toned down their claims. They
have been duped by the concept of handwriting recognition and toy
scenarios. Some have underestimated the problems of text retrieval,
user interface, or intelligent agent design. What makes this scenario
different? As the next section will show, the intimate, fluid
relationship of man and machine and the large size of the information
databases may change the situation.
Having a wearable computer makes note paper obsolete. A
searchable, organized environment where nothing is lost is very
attractive to the users of note paper. With constant access to a
computer screen and keyboard, the user can store all of his notes for
the day (especially useful for students); take along a textbook,
newspaper, or novel to read on the subway; play a video game; catch up
on e-mail or netnews; debug programming; or compose his next piece of
poetry wherever and whenever he wants. This is a very strong force
for keeping everything on-line. Note that this particular interface
reinforces this behavior much more than handwriting based PDA's where
the awkwardness of unpacking, using two hands, and recognition errors
limit the utility of the machine. Thus, the wearable computer can
expect much more input from the user than more traditional machines.
With this greater input directly from the user, especially over the
period of years, a Remembrance Agent has a much greater likelihood of
being useful. The Remembrance Agent could easily remind the user of
something he typed several years ago (and subsequently forgot) which
has pertinence to a present problem (even with low recall rates in
unpersonalized text retrieval studies, automated recall is better than
human recall when a database gets large or when the information is
obtained over time). Furthermore, through this intimate, interactive
relationship with the user, the Remembrance Agent can more easily
learn the user's preferences. Another advantage is that, if the
interface deals exclusively with plain text, both the hardware and the
software can be upgraded many times without disturbing the knowledge
gained in the past. However, neither may ever need to be upgraded for
the functionality described. This would allows a revolutionary
concept in the computer world: a life-long relationship between a user
and a particular machine interface. As the machine and user adapt to
each other over the years, a new, integrated being might emerge
combining the best features of both. Imagine a policeman who never
forgets a face (adding a digitizing camera and simple face recognition
software), an architect who never forgets a structure, or a history
teacher who remembers everything he has ever read or been taught.
Augmented Reality
Overlaying text on the real world in the augmented memory
applications above can be thought of as a particular subfield in the
realm of Augmented Reality. Augmented Reality refers to taking the
virtual computer environment and combining it with the real. Wearable
computing offers a simple, cost effective way to begin experimentation
in this field. Using Private Eyes to overlay a mono or binocular
image on the real world opens many possibilities. With the addition
of a tracking system, the user could have a virtual desk overlaid in
three dimensions on his real desk. Graphical user interfaces could
add physical position to the descriptors of certain files. For
example, a user could leave files at different locations in the
office. These could act as reminders for certain actions the user has
to perform. In addition, such a wearable with tracking might enable
remote conference participants to be overlaid on the real world.
Repairmen might get visual instructions overlaid on the devices they
are supposed to fix. Architects and interior designers could have
blueprints overlaid on a physical structure as they walk through it (a
longer distance tracker like the Global Positioning System could be
used). Construction engineers could visualize changes to a structure
in the field. Doctors could visualize the inside of their patients
before (or while) they operate. Note that these complicated graphics
might not need to be rendered on the wearable. Instead, a base
computer might be used to calculate the graphics necessary for the
application and then transfer the information to the wearable for
display. Several research efforts are already underway on these
topics. However, the registration and tracking tasks necessary in
some of these applications are difficult and may not be overcome in
the near future.
Knowledge Transfer
One of the serious issues facing engineering companies today
is the fast turn around of their employees. Often, by the time the
employee is trained, he is looking for another job. However, if the
employee used a Remembrance Agent to help keep notes on his training
and work, his replacement can learn a great deal by simply
copying the RA's files. In this way, the replacement can have access to
a mini-expert for his new job even when the original employee has
left.
Intelligence Amplification Through Collectives
Through the coupling of users with wearable interfaces, large
intelligent collectives might form. The first implementation might be
similar to an Internet irc channel, where several like-minded users
congregate to talk. Such a channel might be used for real-time
two-way communication from a conference attendee to remote participants who
could not make it in person (possibly with images). A "help"
channel might also be useful where users listen and answer questions
during spare minutes for the common good (I repeatedly use such an
interface for just this purpose at MIT, tapping into hundreds of other
users). In this way, the power of a large group can be harnessed
without much organization and without interruption of regular work.
Another way to harness the power of a group is to allow access
to members' Remembrance Agents. Thus, if I know that Chris is an
expert on digital signal processing, I can just ask his Remembrance
Agent about convolution without having to trouble Chris directly.
So far, the collectives described have been loosely coupled
and not personal. However, a tight collaboration can be formed
between two people by dedicating a portion of each person's screen to
the other's work. For example, let us imagine such a system between
George and Chris, two computer scientists. Each time George looks at
a file, the name of the file and the few lines around George's cursor
appear automatically on Chris's screen. While Chris may not pay
attention to these small disruptions (which are similar to what his
Remembrance Agent may do), he has a constant idea of George's
context. Next time George and Chris actively talk, Chris can be
easily brought up-to-date on George's work. Furthermore, if something
George types catches Chris's eye, then Chris can actively give advice
(for example, Chris knows the location of a particular file or command
which George seems to be searching for). Note that this system can
also be asynchronous and filtered by an agent to avoid sending too
many updates (keystroke by keystroke would be too disruptive) and to
avoid displaying information when the receiving party is asleep.
Simple extensions of this example can be applied to many fields.
The Here and Now
Unfortunately, the traditional computer companies have been
ignoring this potential market, and the pen-based companies still hang
on to the myth that handwriting recognition is the correct interface
for PDA's. However, there are several research companies,
universities, and independent inventors who have discovered wearable
interfaces and have started prototyping the necessary hardware to
become a "cyborg." Below are some of the companies and individuals
that I have found instrumental in creating my current system and
probably can be tapped to make copies. A wearable web page is being
developed to provide more information on vendors.
Doug Platt (showed up at the Media Lab with a working prototype when
mine was still in pieces - my present unit was custom
made by him and then revamped by me- has several ideas
on chording keyboards as well as the unified technology,)
dplatt@cellar.org
Select Tech
(215) 277 4264
1657 The Fairway, Suite 151, Jenkintown, PA 19046
HandyKey Corp. (the one-handed keyboard/mouse)
(516) 474-4405
141 Mt. Sinai Avenue
Mt. Sinai, NY 11766
handykey@mcimail.com
Private Eye (display)
Reflection Technology Inc.
230 Second Ave.
Waltham, MA 02154
617-890-5905 FAX 617-890-5918
However, the marketers of the Private Eye are now
Phoenix Group
Plainview, NY (516) 349-1919
Park Engineering (the main base unit...their general version
has a limited speech recognition board built in)
Spokane, Washington
(unfortunately this address has changed)
As for my personal system, I am slowly evolving a software and
hardware environment I need for everyday use. I am also working on a
study of the long term effects of using this particular design
(physiological, psychological, and productivity). Hopefully, as more
wearable users appear (there are about 4 presently), I will be able to
do studies on collaborative work as well.
Acknowledgements and Disclaimer
When I first began gathering equipment to experiment with
wearable computing, I thought I was the first. Of course, this was
wrong. Many people have worked on these systems both before and after
my personal revelation. While I try to keep track of everyone who has
influenced my opinions or given me facts to work with, I know that I
can't possibly name them all. In particular, however, I would like to
thank the following people for equipment, support, ideas, and
criticism during this trek into a new frontier: Russ Hoffman (who
probably started me thinking in this direction with his "silly"
science project back in '86); Devon McCullough; Doug Platt; Steve
Feiner; Pattie Maes; Henry Lieberman; Olin Shivers; Steve Roberts;
Henry Fuchs; Mike Hawley; the folks at HandyKey Corp., Reflection
Technology, and Park Engineering; the participants in
sci.virtual-worlds and comp.sys.pen; and my co-workers at BBN and MIT
who had to suffer through my enthusiastic outbursts and strange
experiments these last few years.
My opinions are my own.
Sidebar: Why handwriting-based PDA's won't do it
Personal Digital Assistants are supposed to be just what their
names imply, personal and assisting. The PDA manufacturers would have
you believe that you can (or will be able to) take these machines with
you wherever you go, keeping notes, updating schedules, etc. However,
today's machines have fundamentally bad interfaces for the
following reasons:
(1) Small screens. While the rest of the computer world has
been migrating to larger and larger displays so that the user has
enough room to use GUI's, the screens on PDA's have been getting
smaller and smaller. Unfortunately, today's PDA's emphasize
portability, which forces the smaller sized screens. Also, the
handwriting interface most of the PDA's proclaim requires enough room
for the user to write. This provides a fundamental limit on the
physical size of the screen.
(2) Awkward. All the PDA's on the current market require
unzipping, unvelcro-ing, or otherwise unpackaging the PDA when you
want to use it and then repackaging it when you are finished (while
the Newton and the GRID Palmtop are small enough to be attached to
the body, you still have to unvelcro the Newton from your pants or
take out the pen for the Palmtop). Furthermore, almost all the PDA's
require both hands for use (one to steady the tablet, the
other to write). This is very inconvenient whenever simple one-line
notes are required. Also, the user has to be careful to not
damage the large LCD screen (for instance, don't put it in your
back pocket).
(3) Handwriting is a bad interface. The pen-based
manufacturers claim that pen computing provides an intuitive
interface with no training to operate. However,
handwriting is NOT intuitive. We spend several years in school
learning how to form our letters properly (some of us never learned).
The pen manufacturers claim that this is still a lowest common
denominator that is taught in the schools, and we can assume users
will know how to write. However, in today's elementary schools,
children are also being taught how to type. In fact, some claim
that by the time today's first graders graduate, they will have
typed 40,000 lines of code! Handwriting is not the wave of the
future, it is the wave of the past.
Assume then that handwriting recognition is a temporary
measure (which many manufacturers claim, since speech recognition is
now foreseeable). However, today's handwriting recognition simply does
not work well. To get any useful work out of a handwriting system
requires both user and computer training. So much for the walk-up
interface! Pen manufacturers claim that this will improve with time,
and indeed it will. Many research efforts in the area are now
beginning to bear fruit. However, good handwriting recognition
(writing a cursive paragraph with only one or two recognition
mistakes) still requires most of the processing power of today's top
workstations. With this amount of power, adequate speech recognition
can be run just as easily! Why write when you can just talk?
Even if one ignores the previous two objections to
handwriting recognition, there is still a more basic problem.
Handwriting is just too slow. Even assuming perfect, immediate
recognition of handwriting, typing is faster for transferring
information from a user to a computer. Of course, speech recognition
is still faster than either handwriting or typing (in general).
However, even assuming cheap, fast speech recognition, there will be
times when speech is not convenient (privacy or when already talking
with others). Even in a speech recognition future, keyboarding will
still be useful by allowing another, possibly parallel, mode of
communication between human and computer.