Markerless motion capture (MMC) is a technique used to record and analyze human movement without attaching physical markers or hardware to the subject's body.Unlike traditional optoelectronic systems that track retroreflective or active markers, markerless approaches rely on video data captured by one or more cameras and on software typically based on deep learning, to identify and track anatomical keypoints of the human movement.

The field has developed rapidly since the early 2000s, driven by advances in computer vision and the availability of several training datasets. These systems can estimate full-body pose in two or three dimensions, in real time, from a single or multi camera, depending on the accuracy required. This flexibility has enabled deployment in clinical medicine, sport, entertainment and robotics.

History The first study of MMC was published in 1985, which was the first approach to recover the 3D body structures of a person from a single view. In 1996 several studies focused on the extraction of specific parts of the human body, such as limbs, detected with three synchronized cameras by matching geometric primitives to multi-view contours, or head and hand tracking using stereo cameras and 2D Gaussian blob modeling.

By 2000, the detection of keypoints using bounding boxes allowed for the extrapolation of a 13-joint skeleton. In 2004, a 16-joint skeleton was obtained by applying inverse kinematics. In 2008, data processing incorporated volumetric voxel representations for multi-view tracking, reconstructing the body's visual hull with Gaussian blobs.

Transition to deep learning In 2011, Microsoft's Kinect used random decision forests trained on synthetic depth images to estimate 3D joint positions in real time, becoming one of the most successful examples of pre-deep-learning machine learning in motion tracking.From 2013, deep learning became the dominant paradigm for markerless motion capture. DeepPose was among the first to demonstrate the potential of this shift, using a cascade of convolutional networks with rectified linear unit (ReLU) activations to iteratively refine joint predictions across multiple stages. The approach achieved accuracy and speed that surpassed prior methods, contributing to the large adoption of deep learning in pose estimation.

Building on this foundation, a number of open-source frameworks were released in the following years. Openpose, introduced in 2017, was one of the first systems to achieve real-time multi-person 2D pose estimation. It supports the detection of up to 135 landmarks per individual, covering the full body including the hands, feet, and face.

In 2020, large technology companies have entered the field with their own contributions. For example, Google released the MediaPipe framework, designed specifically to run in real time directly on edge devices such as smartphones.

Working principles Once a model has been trained, extracting anatomical keypoints from raw images or video involves a sequence of processing steps: Data acquisition: Footage is recorded using one of two broad categories of hardware: Standard RGB cameras, which capture colour information but carry no inherent depth data, Depth-aware sensors: including RGB-D cameras, which capture both visual and depth information, as well as Time-of-Flight (ToF) devices and LiDAR scanners, which generate a physical depth map. Detection and isolation: Depending on the architecture, the system processes the scene using one of two primary methods: Top-down methods: detect one or more individuals in the frame, typically through a bounding box detector, and then estimate the pose of each person. Bottom-up: detect all anatomical keypoints present in the image without any prior person detection stage, proceeding directly to anatomical mapping. Anatomical mapping: The model analyses the subject and identifies the locations of specific body keypoints through feature extraction. Skeletal reconstruction: In some algorithms, rather than returning a set of independent keypoint coordinates, these systems connect the identified points according to a predefined body model, producing a skeletal representation that makes the pose explicit.

Technology and methodology The backbone of most markerless systems is a deep learning model, typically a convolutional neural network (CNN) or a vision transformer, trained to localize anatomical keypoints directly in 2D image space.When three-dimensional information is needed, a further stage known as "3D lifting" is applied.

This approach depends on the available camera setup:

In monocular systems, which operate from a single viewpoint, they must rely on the networks trained to infer it from visual cues, exploiting learned priors about human body proportions and typical postures. In multi-camera configurations, it is possible to recover 3D coordinates through epipolar geometry and triangulation, using the geometric relationships between calibrated viewpoints. Dataset Deep learning models for markerless motion capture are trained and evaluated using annotated datasets that pair video footage with accurate ground-truth pose data. This ground truth is most commonly obtained from marker-based motion capture systems, which remain the reference standard in the field.

An alternative strategy, adopted by several large-scale datasets, consists of aggregating and unifying existing marker-based recordings collected across multiple studies, producing broader training corpora without the need for new dedicated capture sessions.

Early benchmarks such as Human3.6M were recorded in a controlled studio with a small number of actors and synchronized cameras alongside marker-based ground truth. Later datasets such as MPI-INF-3DHP, introduced multi-view captures across green-screen, indoor, and outdoor scenes. 3D Poses in the Wild (3DPW) recorded everyday activities outdoors using a handheld camera combined with body-worn inertial sensors for ground truth.

For 2D pose estimation, the COCO dataset provide keypoint annotations across a large number of natural images, and it is commonly used to train or evaluate the 2D stage of pipelines such as OpenPose.

A different approach is taken by AMASS, which unifies many existing marker-based archives under the SMPL body model, producing a large pool of motion data for training and synthetic data generation. Furthermore, fully markerless datasets such as FreeMan have aimed to improve real-world generalization, using multi-view consumer-camera footage across varied subjects and environments without specialized hardware.

Domain-specific datasets also exist for clinical applications. For instance, pairing 3D motion sequences with disease-severity scores (e.g. UPDRS-based ratings) to support movement-quality assessment rather than general pose estimation. + Examples of datasets used for training and benchmarkingDatasetYearTypeCamera setupHuman3.6M20143D, indoor4 cameras + marker-based referenceMPI-INF-3DHP20173D, indoor/outdoorMulti-view, markerless referenceCOCO20142D, natural imagesSingle image3DPW20183D, outdoor, in the wildSingle camera + IMUsAMASS20193D, syntheticUnified archivesFreeMan20233D, in the wildMulti-view consumer cameras

Demographic bias and representational fairness Markerless motion capture systems may inherit the biases present in the dataset used to train their models. The problem has emerged in widely used benchmark datasets. COCO has been found to underrepresent women, older adults, and people with darker skin tones. This imbalance may produce measurable drops in detection accuracy for those groups, which is a concern wherever the technology is deployed in sensitive contexts.

As a result, system performance is increasingly reported across demographic subgroups rather than as a single aggregate accuracy figure. Model cards and dataset datasheets are two formats developed with this goal in mind, and their adoption has been gradually growing.

Available systems Algorithms may vary with respect to the number of cameras they require, and whether they are designed to handle a single subject or multiple people simultaneously. Furthermore, implementations exist both as open-source research frameworks and as commercial systems.

Open source frameworks A growing number of open-source frameworks for markerless motion capture have been made publicly available, typically through code-sharing platforms such as Github. + Examples of markerless open source frameworksAlgorithm2D/3DCameras neededMulti-personVNect3D1NoOpenPose2D1YesDeepLabCut2D1YesBlazePose3D1NoSportsCap3D1NoAlphaPose2D1YesYOLO-Pose2D1YesOpenCap3D2No Commercial systems Beyond open-source research frameworks, the industry offers a range of commercial systems designed for environments where high kinematic fidelity is critical. These solutions typically integrate proprietary hardware and software into pre-calibrated, multi-camera configurations. Commercial markerless motion capture solutions include systems such as Vicon, OptiTrack, Qualisys, The Captury, and Theia.

Applications Markerless motion capture has been adopted across several fields, owing primarily to its ability to operate without the controlled laboratory conditions that marker-based systems typically require.

Clinical

In clinical contexts, markerless systems have been explored for several applications, particularly in the rehabilitation field. Parkinson's disease has been one of the most studied patient groups for markerless assessment, with most studies focusing on gait parameters such as stride length, cadence, and arm swing.

Because no markers or wearable hardware are required, assessments can be performed in standard clinical rooms or in the patient's home, reducing the gap between controlled lab measurements and real-world function. However, clinical adoption remains cautious, as validation studies have shown that accuracy for certain joint rotations, particularly at the trunk and hip, can still fall short of what marker-based systems provide.

Sport Markerless systems allow biomechanical analysis to be conducted on the field, on the court, or in the pool, rather than only in dedicated motion capture studios, making movement analysis more accessible for coaches and sports scientists without interfering in the athlete's natural movement. The ability to perform unobtrusive in-competition capture of world-class athletes, previously impossible with marker-based solutions, opened up a new field of analysis with tennis cited as a sport where existing broadcast infrastructure could support analysis with adequate fidelity.

Entertainment The entertainment industry was one of the earliest adopters of motion capture and markerless techniques. In fact it has increasingly supplemented marker-based pipelines in film, television, and video game production.

In consumer gaming, the Microsoft Kinect represented one of the first mainstream deployments of markerless body tracking at scale. Released in November 2010 as an accessory for the Xbox 360, it used a combination of an RGB camera and an infrared depth sensor to track full-body movement without any physical controller, enabling gesture-based interaction in living room settings. It sold over eight million units in its first sixty days, setting a Guinness World Record as the fastest-selling consumer electronics device at the time.

Robotics and human-robot interaction In robotics research, markerless motion capture is used to study how humans move during collaborative tasks, providing the kinematic data needed to program robots that can anticipate and respond to human actions safely. It also appears in imitation learning pipelines, where a robot learns a motor skill by observing a human demonstration recorded on standard video rather than through direct teleoperation.

Limitations and ethics

Technical challenges