The fusion of state-of-the-art sensor technologies and the ever-expanding abilities of AI is revolutionizing safety in the automotive industry. Advanced in-cabin monitoring is fast-becoming a requirement for vehicle manufacturers, greatly increasing passenger safety with a growing capacity to both prevent and quickly respond to accidents and other adverse events.
With its deep expertise in embedded systems, edge AI, and specialized knowledge of camera integration and sensor fusion, HTEC is helping steer the interior monitoring revolution towards a safer future.
The technology behind interior monitoring systems
Before we get into where IMS stands presently and where it is headed, let’s take a quick look at the technology powering advanced passenger safety.
IMS is a complex system powered by a neural network deployed on a chip. The neural network receives and processes inputs from sensors within a vehicle. The most common sensors utilized in IMS are near-infrared (NIR) monochrome illuminated cameras due to their ability to provide vision in low-light conditions. NIR cameras can be supplemented by an array of other sensors, including regular RGB cameras, depth cameras and multi-camera setups, thermal far-infrared (FIR) cameras, and radar and ultrasound systems.
The combination of sensor networks and AI in advanced IMS can detect a variety of behaviors and passenger states, from driver distraction and fatigue detection to child or pet presence monitoring.
The present state of interior monitoring technology
While in-cabin monitoring is increasingly becoming one of the chief concerns in the automotive industry, a casual glance may not necessarily reveal it. Advanced interior monitoring systems (IMS) are currently present in only a handful of high-end commercial vehicles. However, the industry is already way ahead of what is visible to the naked eye of consumers.
Currently available interior monitoring systems rely primarily on cameras, most commonly NIR cameras. RGB cameras are also commonly present, although their application is more closely related to entertainment and personalization functionalities.
Furthermore, present-day monitoring systems are primarily focused on the driver, and not the entire interior of a vehicle. This driver monitoring system (DMS), featuring a camera mounted on or near the steering wheel, is already becoming an obsolete paradigm. Vehicle safety initiatives such as NCAP (New Car Assessment Programme) are increasingly focusing on occupant monitoring systems (OMS) providing a comprehensive look at in-cabin occupancy and activity.
This shifting in-cabin perspective creates a new layer of complexity manufacturers will need to resolve. While DMS can rely on a single camera due to its sole focus on the driver, visually covering the entirety of the vehicle cabin is a bit trickier. Multiple cameras provide greater visibility, but OEMs are reluctant to take that route due to increasing complexity and increasing costs. Companies such as HTEC partner Tobii have successfully developed single-camera DMS and OMS that provide a high degree of in-cabin visibility.
Same as human senses, advanced IMS systems benefit greatly from different types of input for greater contextual awareness. Going forward, comprehensive monitoring of in-cabin occupancy and activity will most likely rely on a combination of cameras with other types of sensors.
The future of interior monitoring systems
As we have stated, IMS systems currently available on the market rely primarily on cameras as sensors, but there is plenty of innovation brewing behind the scenes. Evolving safety requirements are obliging manufacturers to respond to specific edge cases, and virtually all major OEMs are engaged in R&D activities on the IMS front. It is safe to say that some of these R&D solutions are close to production readiness, and we can expect to see them implemented in commercial vehicles within the next couple of years.
The majority of IMS R&D efforts involve sensor fusion and are focused on two areas of exploration — NIR camera-radar fusion and thermal (FIR) cameras. In collaboration with its partners, HTEC is involved in R&D projects in both areas.
Radar systems
The fusion of camera and radar systems is a promising field of exploration, as radar provides a low-power, low-cost complement to cameras, efficiently addressing camera limitations related to line of sight. Some of the main safety requirements prescribed by NCAP where NIR cameras display limitations include:
- Occupant localization: camera mounting position determines the visibility of all occupants and there can be blind spots. Radar can detect occupants not visible to the camera.
- Child presence detection: due to the typical positioning of child seats in vehicles and body dimensions of children, cameras are often limited by line of sight in this aspect. Radar successfully resolves this issue.
- Intrusion detection: due to high power consumption, cameras are not viable as means of intrusion detection in a parked vehicle. Radar represents an efficient, low power means of intrusion detection.
Furthermore, camera-radar fusion could also improve cost-effectiveness by eliminating the need for certain sensors, such as weight or seatbelt sensors.
Radar technology comes with its own limitations, however. They are sensitive to noise (in a broader sense) and can easily suffer from interference. Additionally, there are specific challenges related to its applications and deployment that will need to be resolved. However, industry-wide activity in this area promises to formulate viable solutions in the near future.
HTEC has collaborated on a proof-of-concept camera-radar fusion solution, and the results will be presented later in the year as a working demo.
Thermal (FIR) cameras
For a long time, far-infrared cameras were not considered in the automotive industry due to high manufacturing costs. However, in recent years companies have been able to produce FIR sensors at price points acceptable to the automotive industry, and experimentation with their utilization has been fairly active since.
FIR cameras pick up on thermal radiation and are capable of detecting temperature variations. This makes them suitable not only for occupancy detection, but also for monitoring body heat changes, which can be utilized for detecting developing health conditions and stress. The latter is a particularly interesting line of exploration, as the prevention of traffic accidents caused by health issues is a major industry challenge that could potentially be resolved through novel technologies. However, this line of research is still experimental and comes with an array of challenges related to the proper interpretation of body temperature signals.
Overall, the future utilization of FIR cameras in commercial vehicles is uncertain as NIR camera-radar fusion could provide a more cost-effective solution for a number of currently unaddressed edge cases. However, with the rapid pace of AI evolution and the lowering production costs for FIR cameras, they remain a viable option for exploration and potential application in future IMS systems. Presently, FIR camera technology in automotive is in the early R&D stages and far from being market-ready.
The path forward
Advances in areas such as computer vision and edge AI have transformed cameras into powerful safety tools that can provide a wealth of critical real-time safety information. While they represent the foundation of modern IMS systems, cameras have their limitations as the sole source of in-vehicle sensory input. Recent industry-wide activity strongly indicates that sensor fusion will be the future of IMS, with radar technology being the main lain of current investigation.
With deep expertise in embedded systems and crafting fully optimized, highly efficient edge AI solutions, as well as extensive experience in camera integration in IMS systems, HTEC teams are already shaping the future of automotive safety. Get in touch to explore how HTEC can help you shape impactful IMS systems of tomorrow.
