Chessboards arise frequently in computer vision theory and practice because their highly structured geometry is well-suited for algorithmic detection and processing. The appearance of chessboards in computer vision can be divided into three main areas: camera calibration, feature extraction, and real-world board state recognition (handling occlusions). This article provides a unified discussion of the role that chessboards play in the canonical methods from these areas, including references to the seminal literature, examples, and pointers to software implementations.
Chessboard camera calibration
A classical problem in computer vision is three-dimensional (3D) reconstruction, where one seeks to infer 3D structure about a scene from two-dimensional (2D) images of it.D. Forsyth and J. Ponce. Computer Vision: A Modern Approach. Prentice Hall. (2002). . Practical cameras are complex devices, and photogrammetry is needed to model the relationship between image sensor measurements and the 3D world. In the standard pinhole camera model, one models the relationship between world coordinates and image (pixel) coordinates via the perspective transformation
where is the projective space of dimension .
In this setting, camera calibration is the process of estimating the parameters of the matrix of the perspective model. Camera calibration is an important step in the computer vision pipeline because many subsequent algorithms require knowledge of camera parameters as input.R. Szeliski. Computer Vision: Algorithms and Applications. Springer Science and Business Media. (2010). . Chessboards are often used during camera calibration because they are simple to construct, and their planar grid structure defines many natural interest points in an image. The following two methods are classic calibration techniques that often employ chessboards.
Direct linear transformation
Direct linear transformation (DLT) calibration uses correspondences between world points and camera image points to estimate camera parameters. In particular, DLT calibration exploits the fact that the perspective pinhole camera model defines a set of similarity relations that can be solved via the direct linear transformation algorithm.O. Faugeras. Three-dimensional Computer Vision. MIT Press. (1993). . To employ this approach, one requires accurate coordinates of a non-degenerate set of points in 3D space. A common way to achieve this is to construct a camera calibration rig (example below) built from three mutually perpendicular chessboards. Since the corners of each square are equidistant, it is straightforward to compute the 3D coordinates of each corner given the width of each square. The advantage of DLT calibration is its simplicity; arbitrary cameras can be calibrated by solving a single homogeneous linear system. However, the practical use of DLT calibration is limited by the necessity of a 3D calibration rig and the fact that extremely accurate 3D coordinates are required to avoid numerical instability.
Multiplane calibration
Multiplane calibration is a variant of camera auto-calibration that allows one to compute the parameters of a camera from two or more views of a planar surface. The seminal work in multiplane calibration is due to Zhang.Z. Zhang. "A flexible new technique for camera calibration." IEEE Transactions on Pattern Analysis and Machine Intelligence. vol. 22(11), pp. 1330-1334 (2000). Zhang's method calibrates cameras by solving a particular homogeneous linear system that captures the homographic relationships between multiple perspective views of the same plane. This multiview approach is popular because, in practice, it is more natural to capture multiple views of a single planar surface - like a chessboard - than to construct a precise 3D calibration rig, as required by DLT calibration. The following figures demonstrate a practical application of multiplane camera calibration from multiple views of a chessboard.J. Bouguet, "Camera calibration toolbox for MATLAB". http://www.vision.caltech.edu/bouguetj/calib_doc/. (2013).
Chessboard feature extraction
The second context in which chessboards arise in computer vision is to demonstrate several canonical feature extraction algorithms. In feature extraction, one seeks to identify image interest points, which summarize the semantic content of an image and, hence, offer a reduced dimensionality representation of one's data. Chessboards - in particular - are often used to demonstrate feature extraction algorithms because their regular geometry naturally exhibits local image features like edges, lines, and corners. The following sections demonstrate the application of common feature extraction algorithms to a chessboard image.
Corners
Corners are a natural local image feature exploited in many computer vision systems. Loosely speaking, one can define a corner as the intersection of two edges. A variety of corner detection algorithms exist that formalize this notion into concrete algorithms. Corners are a useful image feature because they are necessarily distinct from their neighboring pixels. The Harris corner detector is a standard algorithm for corner detection in computer vision.C. Harris and M. Stephens. "A combined corner and edge detector." Proceedings of the 4th Alvey Vision Conference. pp. 147-151 (1988). The algorithm works by analyzing the eigenvalues of the 2D discrete structure tensor matrix at each image pixel and flagging a pixel as a corner when the eigenvalues of its structure tensor are sufficiently large. Intuitively, the eigenvalues of the structure tensor matrix associated with a given pixel describe the gradient strength in a neighborhood of that pixel. As such, a structure tensor matrix with large eigenvalues corresponds to an image neighborhood with large gradients in orthogonal directions - i.e., a corner.
A chessboard contains natural corners at the boundaries between board squares, so one would expect corner detection algorithms to successfully detect them in practice. Indeed, the following figure demonstrates Harris corner detection applied to a perspective-transformed chessboard image. Clearly, the Harris detector is able to accurately detect the corners of the board.
Lines
Lines are another natural local image feature exploited in many computer vision systems. Geometrically, the set of all lines in a 2D image can be parametrized by polar coordinates describing the distance and angle, respectively, of their normal vectors with respect to the origin. The discrete Hough transform exploits this idea by transforming a spatial image into a matrix in -space whose -th entry counts the number of image edge points that lie on the line parametrized by .L. Shapiro and G. Stockman. Computer Vision. Prentice-Hall, Inc. (2001). R. Duda and P. Hart. "Use of the Hough transformation to detect lines and curves in pictures," Comm. ACM, vol. 15, pp. 11-15 (1972).P. Hough. "Machine analysis of bubble chamber pictures." Proc. Int. Conf. High Energy Accelerators and Instrumentation. (1959). As such, one can detect lines in an image by simply searching for local maxima of its discrete Hough transform.
The grid structure of a chessboard naturally defines two sets of parallel lines in an image of it. Therefore, one expects that line detection algorithms should successfully detect these lines in practice. Indeed, the following figure demonstrates Hough transform-based line detection applied to a perspective-transformed chessboard image. Clearly, the Hough transform is able to accurately detect the lines induced by the board squares.
The following MATLAB code generates the above images using the Image Processing Toolbox:
% Load image I = imread('Perspective_chessboard.png');
% Compute edge image BW = edge(I, 'canny');
% Compute Hough transform [H theta rho] = hough(BW);
% Find local maxima of Hough transform numpeaks = 19; thresh = ceil(0.1 * max(H(:))); P = houghpeaks(H, numpeaks, 'threshold', thresh);
% Extract image lines lines = houghlines(BW, theta, rho, P, 'FillGap', 50, 'MinLength', 60);
% -------------------------------------------------------------------------- % Display results % -------------------------------------------------------------------------- % Original image figure; imshow(I);
% Edge image figure; imshow(BW);
% Hough transform figure; image(theta, rho, imadjust(mat2gray(H)), 'CDataMapping', 'scaled'); hold on; colormap(gray(256)); plot(theta(P(:, 2)), rho(P(:, 1)), 'o', 'color', 'r');
% Detected lines figure; imshow(I); hold on; n = size(I, 2); for k = 1:length(lines) % Overlay kth line x = [lines(k).point1(1) lines(k).point2(1)]; y = [lines(k).point1(2) lines(k).point2(2)]; line = @(z) ((y(2) - y(1)) / (x(2) - x(1))) * (z - x(1)) + y(1); plot([1 n], line([1 n]), 'Color', 'r'); end
Occluded chessboard detection While empty, unobstructed chessboards are optimal for camera calibration and feature extraction, detecting chessboards occupied by physical chess pieces introduces significant computer vision challenges. In real-world scenarios—such as digitizing active chess games—chess pieces obstruct lattice intersections, cast uneven shadows, and break the straight lines of the grid. Under these conditions, standard geometric algorithms like the Harris corner detector or Hough line transform often fail because they cannot resolve geometries hidden behind pieces.
To resolve these limitations and accurately locate the board's geometry, modern detection pipelines generally fall into two categories: modular iterative approaches and end-to-end deep learning frameworks.
Modular iterative approaches Because classic single-step detectors fail under extreme perspective angles and piece occlusion, researchers have developed algorithms that infer the board's full geometry from partial visible data. A widely utilized methodology, proposed by Czyzewski et al. in 2017, relies on iterative heatmap generation and perspective cropping. Code repository. Instead of attempting sub-pixel localization on the raw image, the algorithm filters the image using heuristic line-merging to deduce grid lines hidden behind pieces, and utilizes neural networks to identify occluded lattice intersections.
The algorithm crops the sub-area around the highest-probability heat map, warps the perspective to normalize skew, and recursively repeats the process. Once perspective correction converges and the exact boundaries of the board are localized, the board can be sliced into 64 isolated square crops. At this stage, downstream algorithms (such as standard convolutional neural networks) are often applied to these squares to classify the occupying pieces and transcribe the board state into Forsyth–Edwards Notation (FEN). Independent analyses have validated the robustness of this iterative paradigm for overcoming occlusion in non-planar camera angles.
End-to-end deep learning A known limitation of modular pipelines is error accumulation; if the initial occlusion-handling or chessboard warping fails, any subsequent processing will also fail. More recent approaches attempt to bypass explicit geometric detection entirely. In 2023, Masouris and van Gemert introduced an end-to-end deep learning framework alongside ChessReD, a dataset of 10,800 real-world smartphone photographs of chess games. Their model attempts to deduce the entire board state and layout directly from the raw image. While a massive improvement over prior end-to-end attempts, their model fully recognized the exact configuration in only 15.26% of unconstrained test images, highlighting how remarkably difficult detecting heavily occluded chessboards remains in computer vision.
Limitations