Computing Machinery and Intelligence

How to read the original

1. Read The Imitation Game and Critique of the New Problem together; the initial game and later machine substitutions are more specific than the popular slogan.
2. Separate the discussion of digital computers from the objections about thought and behavior.
3. Read Learning Machines as a proposal about development and training, not a report of a modern language-model experiment.

Replace a vague question with an observable procedure

Turing proposes discussing machine intelligence through an imitation game and examines objections to the proposal. The initial game involves a human interrogator distinguishing participants under constrained communication, followed by questions about substituting a machine. Treating the paper as a universal one-sentence test discards those details and the subsequent discussion of learning. Behavioral discrimination is an operational question; it does not by itself settle consciousness, moral status, or competence on every task.

The original worked protocol below is a modern teaching exercise, not a claim that Turing specified this experiment. Suppose judges see paired text conversations, one from a human and one from a machine, and must identify the machine. Randomize which transcript is labeled X. Give both participants the same prompt and length budget, remove author metadata, and define how unanswered questions count. Before collecting results, state the judge population, conversation duration, permitted topics, and scoring rule.

Trace the measurement rather than naming a winner

Imagine twenty independent judges each evaluate one independently sampled pair. Twelve identify the machine correctly and eight do not. The observed discrimination accuracy is 12/20=0.6. Under a simple chance model of independent fair guesses, the expected number correct is ten. Neither 0.6 accuracy nor the fact that it is near one half proves indistinguishability. The sample is small, and the experiment needs an uncertainty analysis and a predeclared meaning of practically indistinguishable.

For a rough sense of variation, the standard deviation of the correct-count under fair independent guessing is sqrt(200.50.5), about 2.24. A count of twelve is less than one such standard deviation above ten. That arithmetic describes one null model. It does not establish that judges guessed, that the machine understood the topic, or that its factual answers were reliable.

If one judge rates all twenty pairs, the observations may share that judge’s strategy and biases. If the same human transcript is reused, content also becomes clustered. Counting every rating as independent can exaggerate the amount of evidence. Experimental units and assignment matter as much as the headline success percentage.

Separate discrimination from task performance

A machine can imitate conversational hesitation or errors without solving a problem correctly. A human can be misclassified because of unfamiliar language, accessibility needs, or an unusual communication style. A judge may exploit timestamps or formatting rather than reasoning about content. These possibilities suggest different controls and separate outcome measures, not a conclusion that all behavioral evaluation is worthless.

Add a task with externally checkable answers, such as debugging a small program or explaining a supplied proof. Score correctness independently of author identification. A system might be hard to distinguish yet frequently wrong, or easy to distinguish yet highly useful. Those are different empirical findings and should remain different columns in the report.

What the inference can support

The tested system, prompts, judges, and interface define the scope of the result. Generalizing to new subjects or longer interactions requires further evidence. A failed test can reveal a limitation of the evaluated behavior without proving that no future system could succeed. A successful test does not license claims about unobserved internal experience.

Turing’s methodological contribution is productive partly because it makes critics specify what they would observe. The same discipline applies to modern benchmarks: define the procedure, expose confounds, preserve uncertainty, and distinguish the measured capability from the philosophical or practical conclusion one hopes to draw.

Reconstruction exercise

A demonstration reports that a machine fooled judges in 40 of 50 conversations. All conversations used one topic, all were chosen by the system’s creator after seeing the results, and judges saw different response-time displays for humans and machines. Identify three reasons the headline rate does not establish broad indistinguishability. Redesign the experiment with a predeclared sample, randomized presentation, and at least two separately scored outcomes. Explain why even a well-controlled result should not be described as proof of consciousness.

Worked solution

Post-selection can exclude failures and inflate the reported rate. One topic limits the domain of generalization. Response-time displays introduce a cue unrelated to the intended textual behavior and may systematically help or mislead judges. A revised protocol should define the tested system version, draw prompts from a declared pool before responses are scored, randomize labels, standardize the displayed interface, and record all trials including failures. Score author discrimination separately from task correctness using an independent rubric or externally checkable answers. Recruit or describe the judge population and account for repeated ratings by the same person. Report sample sizes and uncertainty, rather than treating one percentage as universal. The observations concern behavior under a procedure; consciousness is not an outcome directly measured by those labels, so the experiment cannot alone prove its presence or absence.

Test the limits and transfer the idea

1. Probability and experimental design · allow 45 minutes

Quantify a discrimination result without overclaiming

Use an original paired-transcript experiment inspired by, but not identical to, the paper's imitation game. Twenty independently sampled judges each see a distinct human/machine pair with randomized labels. Twelve identify the machine correctly. Under the declared null model each independently guesses with probability 1/2.

Calculate the exact one-sided probability of twelve or more correct identifications under that null. Explain why a large value fails to establish practical indistinguishability. Propose a predeclared equivalence margin and describe the interval evidence needed for that claim. Then identify how reusing one transcript or one judge would affect the independence assumption. Keep discrimination separate from task correctness and consciousness.

Get a starting hint

Under the null, K is binomial. Failure to reject a point null is not evidence that the true accuracy lies inside a narrow interval around one half.

Compare your reasoning with the worked solution

Under the stated null, P(K>=12)=sum from k=12 to 20 of binomial(20,k)/2^20=263950/1048576, approximately 0.251722. This probability concerns obtaining at least the observed correct count under fair independent guessing; it is not the probability that the null hypothesis is true. With this small experiment, the result does not provide strong evidence against that null.

Practical indistinguishability is a different contract. For example, predeclare that discrimination accuracy must lie within [0.45,0.55] for the tested population and procedure. Then require an appropriately constructed confidence interval or equivalence test to place the parameter inside that interval at a declared error level. The observed estimate 0.60 is outside that margin, and this sample is much too imprecise to establish such a narrow equivalence claim. Choosing a wider margin after seeing results would change the question opportunistically.

One judge rating many pairs may reuse a strategy; one transcript shown repeatedly may carry a unique cue. Those ratings can be dependent, so a simple binomial model with one independent trial per rating is not automatically justified. Sample and analyze at the appropriate judge and item levels. Score externally checkable task performance separately: a response can look human while being wrong. Even a controlled equivalence result describes observable behavior in the declared test, not all topics, all users, or an internal state such as consciousness. Turing's operational proposal motivates specifying the procedure, not collapsing these separate inferences.

Check your reasoning

• Calculates the exact tail probability and interprets it correctly.
• Distinguishes null rejection from an explicitly bounded equivalence claim.
• Addresses judge/item dependence and limits the behavioral inference.

2. Counterexample and evaluation · allow 40 minutes

Test a learning machine on an unseen rule

The paper's Learning Machines discussion asks how education can change a system's behavior. Consider an original toy learner for integer inputs. Training examples are (1,1),(2,4),(3,9). Hypothesis A predicts f(x)=x². Hypothesis B predicts g(x)=x²+(x-1)(x-2)(x-3). Both are implemented exactly.

Verify agreement on training data and disagreement at x=4 and x=5. Explain why selecting only training accuracy cannot distinguish their generalization. Design a reproducible evaluation protocol specifying the intended task, a held-out test, model selection, and the role of feedback. Then explain why no finite training set alone identifies an arbitrary unrestricted function and why this does not make learning useless.

Get a starting hint

The added product vanishes at every training input. Outside that set it expresses an alternative inductive assumption.

Compare your reasoning with the worked solution

For x=1,2,3, one factor of the product is zero, so both hypotheses return 1,4,9. At x=4 the product is 3×2×1=6: f(4)=16 and g(4)=22. At x=5 it is 4×3×2=24: f(5)=25 and g(5)=49. Zero training error therefore does not select the intended rule.

Declare the task explicitly, for example computing the square on integers from 1 to 100, with correctness checked against a trusted specification. Freeze the training examples and candidate-selection procedure before testing. Use separate validation data to select models or teaching feedback, then evaluate once on an untouched test sample with recorded seed and system version. Report the sampled input distribution and failures; a test restricted to 1,2,3 measures retention of demonstrations, not new-input behavior. If feedback is given on test answers and the system changes, those examples have become part of training and require a fresh evaluation.

For any finite set of distinct training inputs a1,...,an, adding a nonzero multiple of the product (x-a1)...(x-an) to a polynomial hypothesis preserves all training outputs while changing some unseen ones. More generally, unrestricted functions can be defined arbitrarily outside a finite sample. Learning becomes useful through inductive restrictions, task structure, additional evidence, and evaluation under a meaningful distribution. This original construction illustrates an experimental issue; it is not a claim that Turing proposed these polynomials or solved modern generalization theory in the paper.

Check your reasoning

• Verifies both training agreement and the two held-out disagreements.
• Separates training, model selection, test feedback, and final evaluation.
• Gives the general finite-sample counterexample without concluding learning is impossible.

Check your understanding

A system is frequently judged human but gives wrong answers to factual questions. Which conclusion is justified?

• It has proved general intelligence.
• Its factual errors are irrelevant to every evaluation.
• Discrimination performance and factual reliability must be reported separately.

Explanation

The two measurements answer different questions.

Why is using one judge for fifty trials different from fifty independent judges?

• Repeated judgments can share a strategy or bias, creating dependence.
• One judge cannot read text.
• The denominator must always be one.

Explanation

Repeated observations do not automatically supply independent evidence.

Original paper and sources

Alan M. Turing. Computing Machinery and Intelligence. Mind 59(236), October 1950, pp. 433-460. Linked resource is a university-hosted transcription of the primary text.

Read the original paper →

Read the primary paper: University of Maryland, Baltimore County

• Computing Machinery and Intelligence: University of Maryland, Baltimore County