The objective of this project is the development of the first rehabilitation system that combines the benefits of virtual reality and its interaction with real objects. The system allows training movements that involve all the joints of arms and hands (shoulders, elbows, wrist, and fingers), and the manipulation of daily objects (cutlery, crockery, toilet utensils, coins, etc.). In addition, the system monitors body segments and objects, without carrying any kind of technology, only based on the information recorded by RGB and depth cameras.

I took on the role of lead developer and architect, designing the project’s architecture while accounting for all potential technological risks arising from the use of multiple technologies and custom hardware developed specifically for the project. The technologies used were:

• Unity

• Computer Vision (OpenCV, OpenPose, and YOLO)

• Ad hoc camera synchronization system

• MySQL

• PHP

The system consists of two RGB-D cameras for body tracking, one depth camera for masking and touch capabilities and a short-range projector

MOOD