Affective Computing (2014-Present)
Predicting Perceived Emotions in Animated GIFs with 3D Convolutional Neural Networks
Animated GIFs are widely used on the Internet to express emotions, but automatic analysis of their content is largely unexplored. To help with the search and recommendation of GIFs, we aim to predict how their emotions will be perceived by humans based on their content. Since previous solutions to this problem only utilize image-based features and lose all the motion information, we propose to use 3D convolutional neural networks (CNNs) to extract spatiotemporal features from GIFs. We evaluate our methodology on a crowd-sourcing platform called GIFGIF with more than 6,000 animated GIFs, and achieve a better accuracy than any previous approach in predicting crowd-sourced intensity scores of 17 emotions. We have also found that our trained model can be used to distinguish and cluster emotions in terms of valence and risk perception.
Improving Sleep-Wake Schedule Using Sleep Behavior Visualization and a Bedtime Alarm
Humans need sleep, along with food, water and oxygen, to survive. With about one-third of our lives spent sleeping, there has been increased attention and interest in understanding sleep and the overall state of our ``sleep health." The rapid adoption of smartphones along with a growing number of sleep tracking applications for these devices presents an opportunity to use this device to encourage better sleep hygiene. Procrastinating going to bed and being unable to stick to a consistent bedtime can lead to inadequate amount of sleep time which in turn affects quality of life and overall wellbeing. To help address this problem, we developed two applications, Lights Out and Sleep Wallpaper, which provide a sensor-based bedtime alarm and a connected peripheral display on the wallpaper of the user's mobile phone to promote awareness with sleep data visualization. In this paper, we describe Lights Out and Sleep Wallpaper and results from a two-week field study with 19 participants who have a variety of sleep contexts. Results indicate that a simple bedtime alarm and a peripheral display with sleep data visualization can be an effective method for improving sleep consistency.
Wavelet-Based Motion Artifact Removal for Electrodermal Activity
Electrodermal activity (EDA) recording is a powerful, widely used tool for monitoring psychological or physiological arousal. However, analysis of EDA is hampered by its sensitivity to motion artifacts. We propose a method for removing motion artifacts from EDA, measured as skin conductance (SC), using a stationary wavelet transform (SWT). We modeled the wavelet coefficients as a Gaussian mixture distribution corresponding to the underlying skin conductance level (SCL) and skin conductance responses (SCRs). The goodness-of-fit of the model was validated on ambulatory SC data. We evaluated the proposed method in comparison with three previous approaches. Our method achieved a greater reduction of artifacts while retaining motion-artifact-free data.
SmileTracker: Automatically and Unobtrusively Recording Smiles and Their Context
SmileTracker is a system designed to capture naturally occurring instances of positive emotion during the course of normal interaction with a computer. A facial expression recognition algorithm is applied to images captured with the user's webcam. When the user smiles, both a photo and a screenshot are recorded and saved to the user's profile for later review. Based on positive psychology research, we hypothesize that the act of reviewing content that led to smiles will improve positive affect, and consequently, overall wellbeing.
Brain Computer Interfaces (2008-2014)
Logistic-weighted Regression Improves Decoding of Finger Flexion from Electrocorticographic Signals
One of the most important applications of brain computer interfaces (BCIs) is to assist people who have disrupted neuromuscular channels through which the brain communicates with and controls its external environment by reproducing their motor functions with a cursor or a robotic arm. Thanks to the progress of invasive recording techniques and decoding algorithms in the past ten years, many single neuron-based and electrocorticography (ECoG)-based studies have been able to decode continuous trajectories of limb movements. Unlike traditional BCIs which classify discrete brain states, these studies require prediction of continuous variables. In other words, they belong to regression problems rather than classification problems.
Nevertheless, the studies of limb movement translation are in fact not pure regression problems, because the limbs are not always under the motion state. Whether it is during an experiment or in the daily life, the resting state of the limbs is usually as long as their motion state, if not longer. In this case, the recorded movement data will exhibit a binary property, which was not made the best of in the previous studies.
In this paper, we propose a novel algorithm named logistic-weighted regression to synthesize the binary information and the continuous information of the movement data. The algorithm can significantly improve the decoding of finger flexion in the system described in the link below.
Got the championship in the 2013 brain computer interface competition at UPenn
Platform Development of Online Brain-Computer Interface Competition
Constructed a brain-computer interface competition platform based on steady state visual evoked potentials (SSVEP), which consists of four parts:
(1) A stimulator program displaying numbers 0-9, BACK and CALL which flickered in different frequencies or phases;
(2) A server program receiving the EEG signal from the amplifier and sending it to the client programs;
(3) Client programs using candidates' codes to process the EEG signal, get the SSVEP features and send the classification results to the display program;
(4) A display program showing the results of four teams at the same time and grading them by the accuracy and efficiency
Via the system, a subject was capable of dialling merely by eye focusing, and several teams were able to compete in judging what telephone number the subject had dialled.
Dynamic Classification of Time-embedded EEG with Sequential Hypothesis Testing (SHT) for Brain-Computer Interface
Implemented sequential hypothesis testing (SHT) algorithm in a motor imagery BCI system to improve its efficiency and accuracy.
My work in the project involved:
(1) Revised the code of the BCI2000 system to adapt it to our motor imagery experiments;
(2) Realized SHT algorithm in MATLAB and embedded it in the BCI2000 system;
(3) Established and debugged the whole hardware system with other group members;
Human Computer Interaction on Health Care (2008-2014)
Cortical Visualizers for the International Epilepsy Electrophysiology Portal and BrainMapper
The International Epilepsy Electrophysiology Portal (www.ieeg.org) is a collaborative initiative funded by the National Institutes of Neurological Disease and Stroke. This initiative seeks to advance research towards the understanding of epilepsy by providing a platform for sharing data, tools and expertise between researchers. The portal includes a large database of scientific data and tools to analyze these datasets.
BrainMapper is an offline tool used to create three-dimensional reconstructions of the brain with electrode placement and labelled brain regions for the portal. It requires pre-implant MRI, and a post-implant CT imaging data in order to automatically construct the co-registered model.
My tasks in the two projects are to develop online and offline visualizers for the display of the 3D co-registered model. The visualizers enable users to freely zoom in/out and rotate the cortical model so as to observe more details.
Facepulse - MATLAB and Simulink Student Design Challenge 2013
Facepulse is a software chronically measuring your heart rate with only a webcam.
Real-time pulse monitoring using facial imaging is very popular recently, with several applications on mobile platforms. In my opinion, it is more important to implement it on desktop computers or laptops, because that will enable chronic monitoring. However, I was unable to find such a software, so I decided to make one by myself.
Adaptive Reduction of Motion Artifact in Optical Skin Monitoring Using Accelerometers
Optical skin monitoring uses a digital camera to collect information related to the subtle color changes of skin caused by cardiac signals and body movement (Jonathan & Leahy, 2010; Poh, McDuff, & Picard, 2010). Given illumination of the skin area with a white LED mobile phone flash, the color change information can be described as reflection Photoplethysmography (PPG) imaging. From the PPG signals, heart rate, respiratory rate and even blood oxygen saturation can be extracted via digital signal processing (Shelley, 2007), which means that these physiological parameters can be measured now via only a smart phone without any accessories.
Sport monitoring is an important potential application of the optical skin monitoring method for the following reasons: On the one hand, it is significant to measure physiological parameters during exercise for athletes expecting to improve exercise efficiency and avoid sport injury; On the other hand, athletes are always highly concerned about their equipment’s portability, which is the main advantage of optical skin monitoring. However, in order to realize this application, the problem of motion artifact has to be solved first. As optical skin monitoring is based on the detection of subtle color changes of skin, it is vulnerable to the motion of the skin, for the relative motion between the skin and the camera will change the lighting conditions of the skin area. So far, no work has been done to deal with this problem, and all the relevant research is still focusing on stationary measurement.
In this project, we make full use of the built-in accelerometers of smart phones to implement an active noise cancellation method for optical skin monitoring. The correlation between acceleration and the distorted color change signal is analyzed to confirm that the motion artifact can be predicted from the measured acceleration. A signal distortion model is then established to determine the structure of the active noise cancellation filter. Based on these analyses, our experiments prove the effectiveness of the proposed method in reducing motion artifact for optical skin monitoring.
GoodNite - A Portable Sleep Stage Monitor
Sleep quality is central to the quality of life. GoodNite is designed to help you analyze your sleep and improve it. It’s composed of a lightweight wireless headband and a bedside display.
Compared with the existing sleep stagers, GoodNite combines frontal EEG and head motion information to ensure high staging accuracy without sacrifice of the portability.
GoodNite can visualize your sleep with personalized sleep graphs and, via the monitor of your sleep stages, more functions are available including a wise alarm waking you up at the most appropriate time, pseudo-sunlight simulating sunrise and sleep-assisting music fading as you fall asleep.
High-precision Brain Function Mapping and Neural Monitoring System Based on ECoG
How to reduce the risk of damaging the epileptic patient's vital function areas during resection of epileptic focus remains a challenge for neurosurgeon. Clinically used electrical cortical stimulation (ECS) method shows limits on accuracy, efficiency and reliability.
In this study, a cortical function mapping method with 3D visualization was implemented by analyzing and projecting the power change of high gamma (HG) oscillation in ECoG on patient's own MRI brain model. The method was tested on epileptic patients with subdural electrodes for three tasks (hand movement, tongue movement and silent reading). The proposed 3D cortical function mapping on the patient's individual brain structure provides direct and accurate reference for resection surgery planning.
Identified as a national initiative by Chinese National Academy of Sciences
Got the grand prize in the 6th "Challenge Cup" Extracurricular Academic Science and Technology Competition for Capital College Students
Got the third prize in the 12th "Challenge Cup" Extracurricular Academic Science and Technology Competition for National College Students
iNeck - A Little Health Guardian for Your Neck
Cervical spondylosis is a disease which keeps perplexing human beings, and its incidence has been rocketing due to the unhealthy lifestyle of contemporary people. According to the statistics, the incidence of cervical spondylosis in China has reached to 17.3% with the suffering population more than 100 million and, to our disappointment, the incidence among adolescents has been increasing even faster.
Suffering from cervical spondylosis will lead to dizziness, ache of head, neck, back, shoulders or arms, inflexibility of neck and myasthenia of limbs, cephalagra or even encephalanalosis if severe. These symptoms will all do great harm to the patients’ normal life.
The main predisposition of cervical spondylosis is the neck strain arising from long-time lowering one’s head, as a result of which long-time reading burying one’s head, driving or using a PC for long hours may all result in cervical spondylosis. According to relevant research, one who has lowered his head for more than one hour should relax his neck at once for at least ten minutes, but workers or students can hardly develop the habit only by themselves.
Therefore, iNeck was developed. As part of it, a very light inclinometer can be clipped on an arm of glasses or a headset, which is connected via a cable to the device with a size of an mp3 player. The headset will play a musical hint once the inclinometer detects a long-time burying. In addition, the device provides a connector for myoelectricity electrodes, which can collect the electromyography (EMG) of users’ necks so as to judge the severity of the users’ neck suffering.
Got the third prize in the 4th "CapitalBio Cup" Medical Instrument Design Contest
Rapid Diagnosis - A New Patten of Online Self-Diagnosis Platforms
Realized long-distance self-diagnosis for non-critical patients; Through a data interface connected to home medical equipment, implemented an efficient doctor-patient interactive platform; Helped ease the difficult situation of seeing a doctor in China
Public Health Emergency Network based on Mobile Phones
Emergency treatment is the temporary help given to an injured or a sick person before professional medical treatment can be provided. This timely assistance, comprising of simple medical techniques, is most critical to the victims and is, often, life saving.
However, due to the large population and the imperfect emergency system of hospitals in China, it is usually very hard for patients to receive professional emergency treatment in time. Fortunately, any layperson can be trained to administer first aid, which can be carried out using minimal equipments, and basic training in first aid skills has been taught in schools as well as work places widely in China.
Therefore, the target of the project is to lead the public who received emergency training to first aid scenes through phone’s positioning function and information network.
Human Computer Interaction on Daily Life (2008-2014)
High-speed Keyboard for Chinese Character Input Based on the Optimal Mapping of Fingers and Letters
High-speed Keyboard for Chinese Character Input Based on the Optimal Mapping of Fingers and Letters
Rearranged the traditional keyboard by calculating the operating frequencies of different letters and punctuations when inputting Chinese characters;
The rearranged keyboard enabled the most flexible fingers to control the most commonly used keys so as to improve the input speed.
Got "Best Freshman Award" and the third prize in the 27th "Challenge Cup" Extracurricular Academic Science and Technology Competition for Tsinghua University students
Human Computer Interaction Platform based on Electromyography and Tri-axial Acceleration Sensor
Enabled the operation of computer games and office software through the pattern recognition of upper limbs’ electromyography and posture
Touch Mouse for Playing Music Games
The requirements of different computer games for input devices are quite different. Some of them like real-time strategy (RTS) games require high precision and perfect tactile feedback, while some others like music games (MUG) usually require high input frequency and low fatigue.
This design replaced the micro switches in a mouse with conductive fabric so as to enable the users to press the two buttons with very little force, which lowered the fatigue of users significantly. With clearly shorter response time than touch pads and touch screens, it is particularly suitable for playing music games.
Mobile Campus - Multi-function Campus Information Platform Based on Network
Mobile Campus is an app for mobile devices, which offers a suite of information tools for campus life at Tsinghua University.
Its function includes Mobile Info, Syllabus Reminder, Campus Guidance, Portable Lib and so on.
Biomedical Signal Processing (2008-2014)
Predictive Matching Pursuit and Its Application to Real-time Seizure Detection
Matching pursuit is an effective method of signal representation widely used in neural signal analysis. However, its traditional algorithm is so slow and resource-consuming that it can be hardly applied in real-time, especially when the analyzed data has a lot of channels such as in electroencephalography (EEG). The main objective of this research is to develop an improved matching pursuit algorithm called predictive matching pursuit and verify its speed improvement in neural signal applications.
Study on the Function of Cerebral Cortex with Image Fusion and Neural Signal Processing
Through processing medical images and neural signals, clearly displayed the distribution of important functional areas of the brain on the individualized 3-dimensional model and obtained 5-dimensional functional mapping including spatial, temporal and spectrum features.
The study not only increased the accuracy of preoperative planning for epilepsy surgery, but also provided a new approach to the analysis of brain neural signals.
Received Seed Funding support from Tsinghua University
Analysis of Neural Signals on Sound and Speech Perception
Cooperated with JHMI (Johns Hopkins Medical Institution) of Johns Hopkins University to analyse the experimental ECoG data of epileptic patients; Studied the relevance and connectivity between the language centres and the auditory centre to help understand the neural circuits that processed sound and speech information.
Mono-lead ECG Amplifier
The objective is to design a sensitive amplifier circuit that can detect ECG (Electrocardiogram) signals obtained from metal electrodes applied at the left arm (LA), right arm (RA) and right leg(RL).
The circuit basically amplifies the difference between the right-arm (RA) lead and the left-arm (LA) lead with the right-leg (RL) lead as the ground or reference node. The circuit consists of a “difference” amplifier which is essentially a 2-stage instrumentation amplifier (IA) followed by a band-pass filter, a 2-stage power frequency notch filter and another amplifier stage.
Infrared Human Body Temperature Measurement
Designed the amplifier and filter circuit for pyroelectric detector P2288 of Hamamatsu Corporation so as to realize non-contact measurement of human body temperature.