Real-Time Augmented Reality

Seamlessly Blending Virtual Objects with the Real World

Project Vision

This project introduces a robust system for real-time augmented reality, capable of accurately placing and tracking virtual 3D objects in a live video feed. By mastering camera calibration and marker detection, the application creates a seamless and interactive experience where the virtual and real worlds coexist.

The core of the project is a pipeline that takes a live video stream, identifies key markers (like a chessboard or an ArUco board), and uses them to calculate the camera's precise position and orientation. This allows for the stable and realistic rendering of virtual objects, making them appear as if they are truly part of the physical environment.

Technical Architecture

The system is built on a foundation of key computer vision techniques to deliver a real-time AR experience:

  • Camera Calibration: Establishes the camera's intrinsic parameters to correct for lens distortion and ensure accurate 3D projection.
  • Marker Detection: Utilizes robust algorithms to detect and track both chessboard and ArUco markers in real-time.
  • Pose Estimation: Calculates the 6-DoF (six degrees of freedom) pose of the camera relative to the detected markers, enabling precise object placement.
  • Virtual Object Rendering: Overlays complex 3D models and other virtual elements onto the video feed, anchored to the detected markers.

Core Innovation: Real-Time Marker-Based AR

Live 3D Object Integration

The application's main strength is its ability to project virtual objects into the real world with high fidelity and low latency:

  • Multi-Marker Support: Detects both standard chessboard patterns and advanced ArUco markers for flexible use cases.
  • Dynamic Tracking: Virtual objects remain locked to their markers, even as the camera or the marker moves.
  • Complex 3D Model Support: Renders detailed 3D models from OBJ files, bringing complex virtual creations to life.
  • Interactive Visualizations: Projects not just 3D models, but also visual aids like 3D axes to show the orientation of the detected markers.

Demos & Fun Stuff!

Check out some examples of what this AR application can do.

A virtual bunny projected onto a chessboard
A cute bunny projected onto a chessboard marker.
Sentinels Valorant team logo projected onto a chessboard
Projecting the Sentinels Valorant team logo.
⚠️DO NOT PRESS⚠️
You just got RICKROLLED
You just got RICKROLLED

Technical Stack & Contributions

Core Technologies

  • C++
  • OpenCV
  • CMake

Key Computer Vision Concepts

  • Camera Calibration
  • Pose Estimation (SolvePnP)
  • Feature Detection and Matching
  • 3D Projection and Rendering

Key Contributions

  • A complete pipeline for marker-based augmented reality from camera calibration to real-time rendering.
  • Implementation of both chessboard and ArUco marker detection for robust and flexible tracking.
  • Demonstration of projecting complex 3D models onto a live video feed with accurate and stable tracking.
  • Exploration of advanced feature detectors for future markerless AR applications.

This project serves as a strong foundation for building more advanced augmented reality applications, with potential uses in everything from interactive educational tools to industrial design and entertainment.

For more information, check out the project on GitHub or contact me.