Planned research · not yet implemented

FUTURE PROJECT B · COMPUTER VISION & PERCEPTION

Video-Based Tracking & Multi-Sensor Perception

Carrying radar multi-target tracking and Kalman estimation into computer vision — then fusing video with GPS, IMU and other sensors.

This planned research would build a multi-sensor tracking and perception system that detects moving objects in video, keeps stable identities for them, estimates their motion, and combines complementary sensor information to make tracking more reliable.

The emphasis is on what happens after a detector fires: track initiation, association, prediction, missed detections and termination — the problems I have already worked on with radar plots. Computer vision becomes a new measurement source with its own difficulties: occlusion, illumination, camera geometry and processing cost.

Explore the Pipeline See the Mathematics All Future Projects

Concept illustration: a road camera frame with vehicles in bounding boxes labelled by track ID, dashed predicted gates where each track expects its next detection, one track coasting through a missed detection, and a side panel showing the predict, associate, update and manage cycle. — Concept illustration Per-frame tracking cycle with predicted gates, a coasting track and a tentative new track. No measured data.

Status: Planned research; no video tracker or fusion module has been built yet
Builds on: Radar MTT, adaptive Kalman & alpha-beta tracking, multi-radar fusion
First milestone: Fixed-camera multi-object tracking with adaptive state estimation
Target domains: Traffic monitoring, robotics, UAV perception, autonomous systems

Problem & motivation

Detection Is Not Tracking

Modern neural detectors find objects in individual frames very well. They do not, by themselves, say which detection in frame k is the same vehicle as a detection in frame k − 1, how fast it is moving, or what to do when it disappears behind a bus for half a second.

Those are tracking and estimation problems — exactly the layer I have worked on with radar. This project treats tracking and estimation as the core of the system, with the detector as one replaceable front end, and evaluates the result with video-specific metrics rather than demonstration clips alone.

Three engineering layers

Perception — what objects can the sensors observe?
Tracking — which observations belong to the same moving object, and how is it moving?
Sensor fusion — how can measurements from different sensors be combined into a more reliable estimate?

How existing skills transfer

Left column: completed experience. Right column: the planned transfer.
Existing foundation	Planned transfer
Radar multi-target tracking	Track initiation, plot-to-track association, Kalman and adaptive alpha-beta tracking → video multi-object tracking and state estimation.
Multi-sensor data fusion	Measurement association, coordinate transforms and uncertainty across radars → camera–camera and camera–GPS/IMU fusion.
GPS navigation & tracking	Positions, speed, distance and trajectory history → georeferenced video tracks and visual-vs-GPS trajectory comparison.
Embedded Linux & real-time	Linux, Python, C++ and integration → efficient video pipelines, latency measurement and edge deployment.

New skills to develop: image formation and camera geometry, deep-learning detection, visual appearance features, optical flow and video-specific evaluation methods.

Proposed architecture

A Computer-Vision Front End on a Tracking Core

The neural network that detects objects is kept separate from the algorithm that estimates their motion. That separation lets a classical tracker be benchmarked against a modern framework without redesigning the system.

Perception & tracking architecture

Proposed design

Video & sensor inputs
- Camera video
- GPS
- IMU
- Optional radar / other sensors
Perception layer

Object detection
Pretrained detector: box, class, confidence per frame.
Motion features
Optical flow, appearance embeddings where useful.
Tracking & state estimation
- Data association
- Track initiation & termination
- Kalman filtering
- Motion estimation
Multi-sensor fusion
- Camera geometry
- Time synchronisation
- Coordinate transforms
- GPS/IMU fusion
- Optional radar association
Tracking & perception display
- Object IDs
- Trajectories
- Velocity estimates
- Confidence
- Track history
- Performance statistics

Track lifecycle

Every track moves through explicit states, the same discipline used for radar tracks. Rules are deliberately simple so they can be tuned and measured.

Concept diagram While coasting, the filter predicts without correction, so the track's covariance — and its gate — grows.

Processing pipeline & work packages

From Frames to Stable Tracks

Work Package 1 starts with a fixed camera and recorded video of several moving vehicles or pedestrians, then compares two tracking approaches on identical data.

Read frames with timestamps
Preserve accurate acquisition times; never assume a constant frame interval.
out: frame, t_k
Detect objects
Pretrained detector produces boxes, classes and confidences.
out: {box_j, class_j, score_j}
Predict track states
Kalman prediction of every active track to the current frame time.
out: x̂_k|k−1, P_k|k−1
Associate
Gate candidates and solve a global assignment between tracks and detections.
out: matched / unmatched pairs
Update & manage
Correct matched tracks; coast unmatched tracks; start tentative tracks; delete stale ones.
out: track list with states
Display & log
IDs, boxes, trajectories and estimated motion, saved as timestamped trajectories for evaluation.
out: overlay video + CSV / logs

Benchmark: classical vs. modern

Classical trackerDetector + Kalman filter + a straightforward gated assignment. Transparent and easy to analyse.

Modern frameworkByteTrack-style association that also uses low-confidence detections. Strong baseline from published work.

The contribution is the engineering analysis — track quality, uncertainty, computational cost and failure modes — not simply running an existing model.

Work Package 2 — adaptive motion estimation

A fixed constant-velocity model tracks a steady vehicle well but lags when it turns or brakes. Building on my adaptive Kalman work for maneuvering radar targets, the plan compares:

a constant-velocity Kalman filter,
a filter with motion-dependent process-noise adaptation,
an adaptive alpha-beta tracker, and
a standard tracking framework as baseline.

Moving cameras. Apparent image motion can come from the object, the camera, perspective, or all three. If the camera moves, camera-motion compensation is needed before image displacement is interpreted as object motion.

Concept diagram Gating removes implausible pairs; assignment resolves the rest jointly.

Mathematical foundations

State Estimation, Association and Camera Geometry

The estimator is the standard discrete-time Kalman filter; what changes from radar is the state vector, the measurement model and the association cost.

State vector

Bounding-box state in image space

x_{k} = {[\begin{matrix} u & v & w & h & \dot{u} & \dot{v} & \dot{w} & \dot{h} \end{matrix}]}^{T}

$u, v$: Box centre in pixels.
$w, h$: Box width and height (size changes as objects approach or recede).
$\dot{u}, \dots$: Rates of change — the motion the filter estimates but the detector never measures directly.

After calibration (WP3) the same filter can run on road-plane coordinates in metres, which is where physical speed becomes meaningful.

Prediction & correction

Kalman filter recursion

{\hat{x}}_{k | k - 1} = F {\hat{x}}_{k - 1}

P_{k | k - 1} = F P_{k - 1} F^{T} + Q

K_{k} = P_{k | k - 1} H^{T} S_{k}^{- 1}

{\hat{x}}_{k} = {\hat{x}}_{k | k - 1} + K_{k} ν_{k}

P_{k} = (I - K_{k} H) P_{k | k - 1}

$F, Q$: Motion model (constant velocity over the measured $Δ t$ ) and its process-noise covariance.
$H, R$: Measurement model (selects $u, v, w, h$ ) and detector-noise covariance.
$ν, S$: Innovation $z_{k} - H {\hat{x}}_{k | k - 1}$ and its covariance $H P H^{T} + R$ .
$K$: Kalman gain: how much the new detection is trusted relative to the prediction.

Missed detection: only the prediction step runs, so $P$ grows each frame and the gate widens — the formal version of "the longer we have not seen it, the less sure we are where it is".

Measurement association

Overlap cost and global assignment

IoU (A, B) = \frac{| A \cap B |}{| A \cup B |}

\min_{a_{i j} \in {0, 1}} \sum_{i, j} c_{i j} a_{i j}

s.t. \sum_{j} a_{i j} \leq 1, \sum_{i} a_{i j} \leq 1

$c_{i j}$: Cost of pairing track $i$ with detection $j$ : e.g. $1 - IoU$ or the Mahalanobis distance $d^{2} = ν^{T} S^{- 1} ν$ ; pairs outside the gate are forbidden.
$a_{i j}$: 1 if track $i$ is assigned detection $j$ ; each track and detection is used at most once.

Solved per frame with the Hungarian algorithm. An identity switch is the visible symptom when this assignment goes wrong.

Adaptive estimation (research hypothesis)

Innovation-driven process-noise scaling

ε_{k} = ν_{k}^{T} S_{k}^{- 1} ν_{k}

Q_{k} = λ_{k} Q_{0}, λ_{k} \geq 1

For a consistent filter the normalised innovation squared $ε_{k}$ follows a chi-square distribution with as many degrees of freedom as the measurement. A sustained rise in a moving average of $ε$ signals that the motion model no longer fits — for example during a turn — and the hypothesis is that temporarily inflating $Q$ by $λ_{k}$ improves maneuver response without degrading straight-line smoothness.

Research question: can a motion-adaptive tracker improve trajectory stability and maneuver response over a fixed-motion Kalman filter while meeting a defined real-time processing budget?

Camera geometry (WP3)

Pinhole projection and the ground plane

s [\begin{matrix} u \\ v \\ 1 \end{matrix}] = K_{c} [R | t] [\begin{matrix} X \\ Y \\ Z \\ 1 \end{matrix}]

plane Z = 0 : H_{g} = K_{c} [r_{1} r_{2} t]

$K_{c}$: Camera intrinsics (focal lengths, principal point), from calibration.
$R, t$: Camera rotation and translation relative to the world frame; $r_{1}, r_{2}$ are the first two columns of $R$ .
$H_{g}$: Ground-plane homography: for a fixed camera over a locally planar road, image points map to road-plane points. Moving cameras or 3-D scenes need more general geometry.

Tracking metrics

Counting what goes wrong

MOTA = 1 - \frac{\sum_{t} ({FN}_{t} + {FP}_{t} + {IDSW}_{t})}{\sum_{t} {GT}_{t}}

IDF1 = \frac{2 IDTP}{2 IDTP + IDFP + IDFN}

MOTA is dominated by detection errors; IDF1 rewards keeping the same identity on the same object for as long as possible; HOTA balances detection and association accuracy. All three will be reported, because each hides different failures.

Work Package 4 · multi-sensor perception

Camera, GPS and IMU — Fused Only Where It Makes Sense

Sensor fusion is only meaningful when measurements refer to compatible quantities, are properly time-aligned, and are expressed in a shared coordinate frame.

Camera
Objects & relative motion
Detects objects, gives visual position and object-relative motion, but in pixels until calibrated.
GPS
Platform position
Geographic position of the receiver, when available and accurate enough — not the position of every object in view.
IMU
Short-term platform motion
Acceleration and angular rate at high rate; drifts over time without corrections.

Mobile camera (vehicle, robot, drone)

GPS/IMU estimate the camera's own pose. Calibration and coordinate transforms then relate visual observations to the surrounding environment, so tracked objects can be placed on a map. This is where the project connects directly to my GPS fleet-tracking platform and to Future Project C.

Fixed traffic camera

GPS on the camera platform contributes little to tracking individual vehicles. Road-plane calibration and visual measurements are what matter — the approach planned for Future Project A (GTCS).

A common mistake to avoid: a camera observing several vehicles cannot automatically associate one of them with the GPS position of the camera platform. Fusion design starts from what each sensor actually measures.

Proposed hardware & software

Implementation Technologies

Proposed choices for a Linux-based research prototype. None of them has yet been integrated for this project.

Planned stack; selections will be confirmed by measured accuracy and latency.
Component	Proposed technology	Purpose
Operating system	Ubuntu Linux LTS	Development and edge targets
Languages	Python first; C++ for hot paths	Rapid research, then performance
Video & calibration	OpenCV	Capture, undistortion, homography, optical flow
Detection	Pretrained YOLO-family detector (PyTorch)	Boxes, classes, confidences
Tracking baseline	ByteTrack (official implementation)	Modern reference tracker
Own tracker	NumPy / SciPy Kalman + Hungarian assignment	Classical and adaptive variants
Sensor integration	ROS 2 (when GPS/IMU are added)	Timestamps, transforms, logging
Evaluation	Standard MOT evaluation tooling (MOTA, IDF1, HOTA)	Repeatable benchmark

Hardware

Development PC, adding an NVIDIA GPU only when detector throughput requires it.
Fixed USB or IP camera for recorded and live sequences.
Existing GPS receiver from my navigation platform, plus an IMU for WP4.
Embedded target (Raspberry Pi class or an edge accelerator) for the deployment stage.

Recording discipline

For every comparison: same sequences, same hardware, and recorded detector settings, frame rate, resolution and skipped frames — otherwise results are not comparable.

Engineering challenges & limitations

Where Video Tracking Breaks

Occlusion
Objects hidden by others
Leads to missed detections, fragmented tracks and swapped identities.
Approach: coasting with covariance growth, conservative deletion, and recovery-rate metrics.
Illumination
Lighting & weather
Night, glare, shadows and rain change detector behaviour.
Approach: per-condition evaluation sets, not one blended average.
Geometry
Pixels are not metres
Perspective makes image motion depend on distance from the camera.
Approach: calibration and comparison against reference measurements before any speed is reported.
Ego-motion
Moving cameras
Camera motion contaminates apparent object motion.
Approach: camera-motion compensation, or IMU-aided ego-motion estimation in WP4.
Compute
Real-time budget
Detection usually dominates cost; dropped frames degrade tracking.
Approach: measure FPS, latency and utilisation for every configuration.
Timing
Sensor synchronisation
Camera, GPS and IMU arrive at different rates with different latencies.
Approach: hardware or ROS 2 timestamps and explicit latency compensation.

Testing & performance metrics

A Repeatable Benchmark, Not Just Demo Videos

Planned evaluation framework. All algorithms run on the same sequences and hardware.
Evaluation area	Measures
Object detection	Precision, recall, mAP
Tracking continuity	ID switches, IDF1, HOTA
State estimation	Position and velocity error against reference data
Maneuver response	Tracking error during turns and acceleration
Occlusion handling	Track recovery rate and identity continuity
Computational efficiency	FPS, CPU/GPU utilisation, memory use
Processing latency	End-to-end time from frame acquisition to displayed track
Robustness	Performance across lighting, density, weather and camera conditions

Phased implementation roadmap

Six Stages from Foundation to Deployment

01Foundation
Single-camera detection & tracking
- Process recorded video
- Display boxes and track IDs
- Save timestamped trajectories
02Algorithms
Classical vs. modern tracker
- Baseline Kalman tracker
- Compare with an established tracker
- Measure tracking performance
03Research
Adaptive motion estimation
- Adaptive filtering
- Maneuver response, missed detections
- Trajectory stability
04Geometry
Real-world position & speed
- Calibrate the camera
- Ground-plane positions and velocities
- Compare with reference measurements
05Integration
Multi-sensor perception
- Second camera or positioning sensor
- Timestamp alignment, transforms
- Measurement fusion
06Deployment
Linux-based demonstration
- Optimise the pipeline
- Document limitations
- Reproducible demo with measurements

First research milestone: video-based multi-object tracking with adaptive state estimation, then extension toward camera geometry and multi-sensor fusion.

Expected contribution

Expected Contribution & Research Direction

The long-term aim is a modular, real-time multi-sensor tracking and perception platform that combines established tracking mathematics with modern computer vision — a foundation for navigation and autonomous systems. The distinctive contribution is the rigorous transfer of radar-tracking discipline (explicit track lifecycles, gating, uncertainty and adaptive estimation) into video, backed by measured comparisons.

Potential applications

Intelligent traffic monitoring
Autonomous vehicles and mobile robotics
Unmanned aerial vehicle perception
Multi-sensor surveillance research

Proposed project lead

Mohammed Ismail Ali Attala

Senior Systems Engineer · C4I/C2 Integration · Tracking & Navigation

Interested in collaborating with computer-vision, robotics and ITS teams on tracking and perception research.

Contact & Collaboration

Building Perception or Tracking Systems?

I welcome discussions about multi-object tracking, state estimation, camera geometry and multi-sensor perception.

Send an Email LinkedIn Back to Future Projects

Video-Based Tracking & Multi-Sensor Perception

Detection Is Not Tracking

Three engineering layers

How existing skills transfer

A Computer-Vision Front End on a Tracking Core

Perception & tracking architecture

Video & sensor inputs

Perception layer

Object detection

Motion features

Tracking & state estimation

Multi-sensor fusion

Tracking & perception display

Track lifecycle

From Frames to Stable Tracks

Read frames with timestamps

Detect objects

Predict track states

Associate

Update & manage

Display & log

Benchmark: classical vs. modern

Work Package 2 — adaptive motion estimation

State Estimation, Association and Camera Geometry

Bounding-box state in image space

Kalman filter recursion

Overlap cost and global assignment

Innovation-driven process-noise scaling

Pinhole projection and the ground plane

Counting what goes wrong

Camera, GPS and IMU — Fused Only Where It Makes Sense

Objects & relative motion

Platform position

Short-term platform motion

Mobile camera (vehicle, robot, drone)

Fixed traffic camera

Implementation Technologies

Hardware

Recording discipline

Where Video Tracking Breaks

Objects hidden by others

Lighting & weather

Pixels are not metres

Moving cameras

Real-time budget

Sensor synchronisation

A Repeatable Benchmark, Not Just Demo Videos

Six Stages from Foundation to Deployment

Single-camera detection & tracking

Classical vs. modern tracker

Adaptive motion estimation

Real-world position & speed

Multi-sensor perception

Linux-based demonstration

Expected Contribution & Research Direction

Potential applications

Mohammed Ismail Ali Attala

Building Perception or Tracking Systems?