New method uses cameras to measure pulse, respiratory rate, could help telehealth

Telehealth has become an essential way for physicians to continue providing health care while minimizing in-person contact during COVID-19. But with phone or Zoom appointments, it’s harder for doctors to get a patient’s important vital signs, such as their pulse or respiratory rate, in real time.

A team led by the University of Washington has developed a method that uses a camera on a person’s smartphone or computer to take their pulse and respiratory signal from a real-time video of their face. The researchers presented this cutting-edge system in December at the Neural Information Processing Systems conference.

Now the team is coming up with a better system to measure these physiological signals. This system is less likely to be triggered by different cameras, lighting conditions, or facial features, such as skin color. The researchers will present these results on April 8 at the ACM conference on health, interference and learning.

“Machine learning is pretty good at classifying images. If you give him a series of cat photos and then tell him to find cats in other pictures, he can do that. But for machine learning to be useful in remote health sensing, we need a system that can identify the region of interest in a video that contains the strongest source of physiological information – the pulse, for example – and then measure that over time, ”said senior author Xin Liu, a UW doctoral student at the Paul G Allen School of Computer Science and Engineering.

“Each person is different,” Liu said. “This system must therefore be able to adapt quickly to each person’s unique physiological signature and separate it from other variations, such as their appearance and the environment in which they find themselves.”

The team’s system preserves privacy – it runs on the device rather than in the cloud – and uses machine learning to capture the subtle changes in the way light reflects off a person’s face. which is correlated with the evolution of blood flow. Then, it converts these changes into pulse and respiratory rate.

The first version of this system was trained with a dataset containing both videos of people’s faces and “ground truth” information: each person’s pulse and respiratory rate measured by standard instruments in the field. . The system then used the spatial and temporal information from the videos to calculate the two vital signs. It outperformed similar machine learning systems on videos where subjects were moving and talking.

But while the system performed well on some datasets, it still struggled with others that contained different people, backgrounds, and lighting. This is a common problem known as “overfitting,” the team said.

The researchers improved the system by having it produce a personalized machine learning model for each individual. Specifically, it allows you to search for important areas in a video image that likely contain physiological features that correlate with the evolution of blood flow in a face in different settings, such as skin tones, lighting conditions and colors. different environments. From there, he can focus on that area and measure the pulse and respiratory rate.

While this new system outperforms its predecessor when it has more complex data sets, especially for people with darker skin, there is still work to be done, the team said.

“We recognize that there is always a tendency for lower performance when the subject’s skin type is darker,” Liu said. “This is in part because light reflects differently on darker skin, resulting in a weaker signal for the camera. Our team is actively developing new methods to address this limitation.”

Researchers are also working on various collaborations with doctors to see how this system works in the clinic.

“Any ability to sense the pulse or respiratory rate remotely opens up new opportunities for remote patient care and telemedicine. This could include personal care, follow-up care or triage, especially when someone does not have convenient access to a clinic, ”the official said. author Shwetak Patel, professor at both the Allen School and the Department of Electrical and Computer Engineering. “It’s exciting to see academic communities working on new algorithmic approaches to solve this problem with devices people have in their homes. “


Ziheng Jiang, doctoral student at Allen School; Josh Fromm, a UW graduate who now works at OctoML; Xuhai Xu, doctoral student at the School of Information; and Daniel McDuff of Microsoft Research are also co-authors of this article. This research was funded by the Bill & Melinda Gates Foundation, Google, and the University of Washington.

For more information, contact Liu at and Patel at

Source link

About Donald P. Hooten

Check Also

Distributed deep learning method without sharing sensitive data

Data sharing is one of the major challenges of machine learning models. The advent of …