A Wager on the Turing Test: The Rules
An explanation of rules behind the Turing Test, used to determine the winner of a long bet between Ray Kurzweil and Mitch Kapor over whether artificial intelligence will be achieved by 2029.
Published April 9, 2002 on KurzweilAI.net. Click here
to see why Ray Kurzweil thinks he will win. Click here
to read why Mitch Kapor thinks he'll win. Finally, see Ray's response.
Background on the "Long Now Turing Test Wager."
Ray Kurzweil maintains that a computer (i.e., a machine intelligence)
will pass the Turing test by 2029. Mitchell Kapor believes this
will not happen.
This wager is intended to be the inaugural long term bet to be
administered by the Long Now Foundation. The proceeds of the wager
are to be donated to a charitable organization designated by the
winner.
This document provides a brief description of the Turing Test and
a set of high level rules for administering the wager. These rules
contemplate setting up a "Turing Test Committee" which will create
the detailed rules and procedures to implement the resolution of
the wager. A primary objective of the Turing Test Committee will
be to set up rules and procedures that avoid and deter cheating.
Brief Description of the Turing test. In
a 1950 paper ("Computing Machinery and Intelligence," Mind
59 (1950): 433- 460, reprinted in E. Feigenbaum and J. Feldman,
eds., Computers and Thought, New York: McGraw-Hill,
1963), Alan Turing describes his concept of the Turing Test, in
which one or more human judges interview computers and human foils
using terminals (so that the judges won't be prejudiced against
the computers for lacking a human appearance). The nature of the
dialog between the human judges and the candidates (i.e., the computers
and the human foils) is similar to an online chat using instant
messaging. The computers as well as the human foils try to convince
the human judges of their humanness. If the human judges are unable
to reliably unmask the computers (as imposter humans) then the computer
is considered to have demonstrated human-level intelligence1.
Turing was very specifically nonspecific about many aspects of
how to administer the test. He did not specify many key details,
such as the duration of the interrogation and the sophistication
of the human judge and foils. The purpose of the rules described
below is to provide a set of procedures for administering the test
some decades hence.
The Procedure for the Turing Test Wager: The Turing Test General
Rules
These Turing Test General Rules may be modified by agreement of
Ray Kurzweil and Mitchell Kapor, or, if either Ray Kurzweil and
/ or Mitchell Kapor is not available, then by the Turing Test Committee
(described below). However, any such change to these Turing Test
General Rules shall only be made if (i) these rules are determined
to have an inconsistency, or (ii) these rules are determined to
be inconsistent with Alan Turing's intent of determining human-level
intelligence in a machine, or (iii) these rules are determined to
be unfair, or (iv) these rules are determined to be infeasible to
implement.
I. Definitions.
A Human is a biological human person as that term is understood
in the year 2001 whose intelligence has not been enhanced through
the use of machine (i.e., nonbiological) intelligence, whether used
externally (e.g., the use of an external computer) or internally
(e.g., neural implants). A Human may not be genetically enhanced
(through the use of genetic engineering) beyond the level of human
beings in the year 2001.
A Computer is any form of nonbiological intelligence (hardware
and software) and may include any form of technology, but may not
include a biological Human (enhanced or otherwise) nor biological
neurons (however, nonbiological emulations of biological neurons
are allowed).
The Turing Test Committee will consist of three Humans, to be selected
as described below.
The Turing Test Judges will be three Humans selected by the Turing
Test Committee.
The Turing Test Human Foils will be three Humans selected by the
Turing Test Committee.
The Turing Test Participants will be the three Turing Test Human
Foils and one Computer.
II. The Procedure
The Turing Test Committee will be appointed as follows.
- One member will be Ray Kurzweil or his designee, or, if not
available, a person appointed by the Long Now Foundation. In the
event that the Long Now Foundation appoints this person, it shall
use its best efforts to appoint a Human person that best represents
the views of Ray Kurzweil (as expressed in the attached essay
"Why I Think I Will Win The Long Now Turing Test Wager.")
- A second member will be Mitchell Kapor or his designee, or,
if not available, a person appointed by the Long Now Foundation.
In the event that the Long Now Foundation appoints this person,
it shall use its best efforts to appoint a Human person that best
represents the views of Mitchell Kapor (as expressed in the attached
essay "Why I Think I Will Win The Long Now Turing Test Wager.")
- A third member will be appointed by the above two members, or
if the above two members are unable to agree, then by the Long
Now Foundation, who in its judgment, is qualified to represent
a "middle ground" position.
Ray Kurzweil, or his designee, or another member of the Turing
Test Committee, or the Long Now Foundation may, from time to time
call for a Turing Test Session to be conducted and will select or
provide one Computer for this purpose. For those Turing Test Sessions
called for by Ray Kurzweil or his designee or another member of
the Turing Test committee (other than the final one in 2029), the
person calling for the Turing Test Session to be conducted must
provide (or raise) the funds necessary for the Turing Test Session
to be conducted. In any event, the Long Now Foundation is not obligated
to conduct more than two such Turing Test Sessions prior to the
final one (in 2029) if it determines that conducting such additional
Turing Test Sessions would be an excessive administrative burden.
The Turing Test Committee will provide the detailed rules and procedures
to implement each such Turing Test Session using its best efforts
to reflect the rules and procedures described in this document.
The primary goal of the Turing Test Committee will
be to devise rules and procedures which avoid and deter cheating
to the maximum extent possible. These detailed rules and procedures
will include (i) specifications of the equipment to be used, (ii)
detailed procedures to be followed, (iii) specific instructions
to be given to all participants including the Turing Test Judges,
the Turing Test Human Foils and the Computer, (iv) verification
procedures to assure the integrity of the proceedings, and (v) any
other details needed to implement the Turing Test Session. Beyond
the Turing Test General Rules described in this document, the Turing
Test Committee will be guided to the best of its ability by the
original description of the Turing Test by Alan Turing in his 1950
paper. The Turing Test Committee will also determine procedures
to resolve any deadlocks that may occur in its own deliberations.
Each Turing Test Session will consist of at least three Turing
Test Trials.
For each such Turing Test Trial, a set of Turing Test Interviews
will take place, followed by voting by the Turing Test Judges as
described below.
Using its best judgment, the Turing Test Committee will appoint
three Humans to be the Turing Test Judges.
Using its best judgment, the Turing Test Committee will appoint
three Humans to be the Turing Test Human Foils. The Turing Test
Human Foils should not be known (either personally or by reputation)
to the Turing Test Judges.
During the Turing Test Interviews (for each Turing Test Trial),
each of the three Turing Test Judges will conduct online interviews
of each of the four Turing Test Candidates (i.e., the Computer and
the three Turing Test Human Foils) for two hours each for a total
of eight hours of interviews conducted by each of the three Turing
Test Judges (for a total of 24 hours of interviews).
The Turing Test Interviews will consist of online text messages
sent back and forth as in a online "instant messaging" chat, as
that concept is understood in the year 2001.
The Human Foils are instructed to try to respond in as human a
way as possible during the Turing Test Interviews.
The Computer is also intended to respond in as human a way as possible
during the Turing Test Interviews.
Neither the Turing Test Human Foils nor the Computer are required
to tell the truth about their histories or other matters. All of
the candidates are allowed to respond with fictional histories.
At the end of the interviews, each of the three Turing Test Judges
will indicate his or her verdict with regard to each of the four
Turing Test Candidates indicating whether or not said candidate
is human or machine. The Computer will be deemed to have passed
the "Turing Test Human Determination Test" if the Computer has fooled
two or more of the three Human Judges into thinking that it is a
human.
In addition, each of the three Turing Test Judges will rank the
four Candidates with a rank from 1 (least human) to 4 (most human).
The computer will be deemed to have passed the "Turing Test Rank
Order Test" if the median rank of the Computer is equal to or greater
than the median rank of two or more of the three Turing Test Human
Foils.
The Computer will be deemed to have passed the Turing Test if it
passes both the Turing Test Human Determination Test and the Turing
Test Rank Order Test.
If a Computer passes the Turing Test, as described above, prior
to the end of the year 2029, then Ray Kurzweil wins the wager. Otherwise
Mitchell Kapor wins the wager.
1 Turing's initial description of his test was as a
parlor game in which judges try to determine the gender of male
and female human contestants. He then suggests the applicability
of this type of game to its present purpose of determining when
the level of intelligence of a machine is indistinguishable from
that of a human.
| 
