Asynchronous I/O moves the waiting boundary

Prerequisites: Devices, buffers, blocking calls, and queues.

Learning objectives

• Distinguish asynchronous submission from completion.
• Explain buffer lifetime and partial-transfer obligations.
• Use queue depth and service time to reason about throughput.

Returning early does not mean finishing early

A blocking I/O call keeps the calling thread waiting until its documented completion condition occurs. An asynchronous interface separates submission from completion, allowing the caller to do other work while a request is outstanding. The device still needs time to perform the operation. Asynchrony changes how that waiting is represented and overlapped.

Consider a request to read a block into a buffer. Submission identifies the operation, destination, and a way to recognize completion. The program must preserve the buffer and associated request state until the interface permits reuse. A successful submission says that the request was accepted under that API’s rules. It does not necessarily say that the bytes are already available or that the eventual operation will succeed.

Follow the lifecycle completely

A useful request trace has four stages: prepare, submit, complete, and retire. Preparation establishes a valid buffer and request identifier. Submission transfers specified responsibilities to the operating system or device. Completion reports status and the amount transferred. Retirement consumes the result and releases or reuses resources when safe.

Direct memory access allows a device to transfer data without the CPU copying every byte through ordinary instructions. It still requires setup, address and permission management, and coordination before software consumes the result. Interrupts can notify completion; polling repeatedly checks for it. Polling spends processor time but may reduce notification overhead or latency in an appropriate high-rate setting. Neither technique is universally superior.

Queue depth buys overlap and costs memory

Suppose a device takes an average of two milliseconds per request and can overlap enough independent work. At steady state, Little’s law relates average outstanding requests L, completion rate λ, and average residence time W by L=λW, using consistent units and stable averages. Sustaining 1000 completions per second with W=0.002 seconds requires an average of two outstanding requests.

This relation does not prove that increasing queue depth creates arbitrary throughput. A device has bandwidth, processing, and scheduling limits. Past saturation, extra queued work mainly increases residence time. If arrivals exceed sustainable completion rate indefinitely, an unbounded queue grows without bound.

Backpressure limits outstanding work or asks producers to slow down. Cancellation also needs a contract: requesting cancellation may race with completion and may not immediately permit buffer reuse. Partial transfers and errors belong in the normal state machine. An asynchronous program is correct only when every accepted request reaches a handled terminal state and its resources remain valid throughout the interval in which another component can still access them.

Primary reading

• I/O Devices (Arpaci-Dusseau and Arpaci-Dusseau, OSTEP).

Practice problem

An I/O service completes an average of 2000 requests per second, with mean residence time four milliseconds. Compute average outstanding requests under steady-state assumptions. Explain what happens to latency if the service is saturated and a client doubles queued requests without increasing service capacity. Describe the buffer-reuse rule for a request whose cancellation is only requested, not confirmed.

Worked solution

Little’s law gives L=λW=2000×0.004=8 requests on average. If service capacity is already saturated, placing more work in its queue does not create more device bandwidth. The additional requests wait longer, increasing residence time and often tail latency. Exact changes depend on service-time variability and scheduling, so doubling the queue is not a universal doubling formula for latency. A cancellation request is itself an action that may race with normal completion. Unless the API explicitly guarantees otherwise, the program must retain the buffer until a completion or cancellation result establishes that no device or kernel operation can access it. Reusing it earlier can corrupt another operation or expose incorrect data.

Check your understanding

What does successful asynchronous submission necessarily establish?

• The bytes are already consumed by the application.
• Only the acceptance guarantees defined by the submission API.
• The request can no longer fail.
• The buffer may always be freed immediately.

Explanation

Completion and resource-release guarantees are separate parts of the lifecycle.

At fixed saturated service capacity, increasing outstanding work usually does what?

• Creates unlimited throughput.
• Adds waiting time once additional overlap no longer helps.
• Eliminates all partial transfers.
• Makes device latency exactly zero.

Explanation

A larger queue cannot exceed the underlying service capacity.

Primary reading

• I/O Devices: Arpaci-Dusseau and Arpaci-Dusseau, OSTEP

Open the study and save your work →