Hindawi Publishing Corporation
EURASIP Journal on Advances in Signal Processing
Volume 2008, Article ID 823695, 17 pages
doi:10.1155/2008/823695
Research Article
Simultaneous Eye Tracking and Blink Detection with
Interactive Particle Filters
Junwen Wu and Mohan M. Trivedi
Computer Vision and Robotics Research Laboratory, University of California, San Diego, La Jolla, CA 92093, USA
Correspondence should be addressed to Junwen Wu, juwu@ucsd.edu
Received 2 May 2007; Revised 1 October 2007; Accepted 28 October 2007
Recommended by Juwei Lu
We present a system that simultaneously tracks eyes and detects eye blinks. Two interactive particle filters are used for this purpose,
one for the closed eyes and the other one for the open eyes. Each particle filter is used to track the eye locations as well as the scales
of the eye subjects. The set of particles that gives higher confidence is defined as the primary set and the other one is defined
as the secondary set. The eye location is estimated by the primary particle filter, and whether the eye status is open or closed
is also decided by the label of the primary particle filter. When a new frame comes, the secondary particle filter is reinitialized
according to the estimates from the primary particle filter. We use autoregression models for describing the state transition and a
classification-based model for measuring the observation. Tensor subspace analysis is used for feature extraction which is followed
by a logistic regression model to give the posterior estimation. The performance is carefully evaluated from two aspects: the
blink detection rate and the tracking accuracy. The blink detection rate is evaluated using videos from varying scenarios, and
the tracking accuracy is given by comparing with the benchmark data obtained using the Vicon motion capturing system. The
setup for obtaining benchmark data for tracking accuracy evaluation is presented and experimental results are shown. Extensive
experimental evaluations validate the capability of the algorithm.
Copyright © 2008 J. Wu and M. M. Trivedi. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly
cited.
1. INTRODUCTION
Eye blink detection plays an important role in human-
computer interface (HCI) systems. It can also be used in
driver’s assistance systems. Studies show that eye blink du-
rationhasacloserelationtoasubject’sdrowsiness[
1]. The
openness of eyes, as well as the frequency of eye blinks, shows
the level of the person’s consciousness, which has potential
applications in monitoring driver’s vigourous level for addi-
tional safety control [2]. Also, eye blinks can be used as a
method of communication for people with severe disabili-
ties, in which blink patterns can be interpreted as semiotic
messages [3–5]. This provides an alternate input modality to
control a computer: communication by “blink pattern.” The
duration of eye closure determines whether the blink is vol-
untary or involuntary. Blink patterns are used by interpreting
voluntary long blinks according to the predefined semiotics
dictionary, while ignoring involuntary short blinks.
Eye blink detection has attracted considerable research
interest from the computer vision community. In literature,
most existing techniques use two separate steps for eye track-
ing and blink detection [2,3,5–8]. For eye blink detection
systems, there are three types of dynamic information in-
volved: the global motion of eyes (which can be used to infer
the head motion), the local motion of eye pupils, and the
eye openness/closure. Accordingly, an effective eye tracking
algorithm for blink detection purposes needs to satisfy the
following constraints:
(i) track the global motion of eyes, which is confined by
the head motion;
(ii) maintain invariance to local motion of eye pupils;
(iii) classify the closed-eye frames from the open-eye
frames.
Once the eyes’ locations are estimated by the tracking al-
gorithm, the differences in image appearance between the
open eyes and the closed eyes can be used to find the frames
in which the subjects’ eyes are closed, such that eye blink-
ing can be determined. In [2], template matching is used to
track the eyes and color features are used to determine the
2 EURASIP Journal on Advances in Signal Processing
openness of eyes. Detected blinks are then used together with
pose and gaze estimates to monitor the driver’s alertness. In
[6,9], blink detection is implemented as part of a large fa-
cial expression classification system. Differences in intensity
values between the upper eye and lower eye are used for eye
openness/closure classification, such that closed-eye frames
can be detected. The use of low-level features makes the real-
time implementation of the blink detection systems feasible.
However, for videos with large variations, such as the typi-
cal videos collected from in-car cameras, the acquired images
are usually noisy and with low-resolution. In such scenarios,
simple low-level features, like color and image differences,
are not sufficient. Temporal information is also used by some
other researchers for blinking detection purposes. For exam-
ple, in [3,5,7], the image difference between neighboring
frames is used to locate the eyes, and the temporal image cor-
relation is used thereafter to determine whether the eyes are
open or closed. This system provides a possible new solu-
tion for a human-computer interaction system that can be
used for highly disabled people. Besides that, motion infor-
mation has been exploited as well. The estimate of the dense
motion field describes the motion patterns, in which the eye
lid movements can be separated to detect eye blinks. In [8],
dense optical flow is used for this purpose. The ability to dif-
ferentiate the motion related to blinks from the global head
motion is essential. Since face subjects are nonrigid and non-
planar, it is not a trivial work.
Such two-step-based blink detection system requires that
the tracking algorithms are capable of handling the appear-
ance change between the open eyes and the closed eyes. In
this work, we propose an alternative way that simultaneously
tracks eyes and detects eye blinks. We use two interactive
particle filters, one tracks the open eyes and the other one
tracks the closed eyes. Eye detection algorithms can be used
to give the initial position of the eyes [10–12], and after that
the interactive particle filters are used for eye tracking and
blink detection. The set of particles that gives higher con-
fidence is defined as the primary particle set and the other
one is defined as the secondary particle set. Estimates of the
eyes’ location, as well as the eye class labels (open-eye ver-
sus closed-eye), are determined by the primary particle filter,
which is also used to reinitialize the secondary particle fil-
ter for the new observation. For each particle filter, the state
variables characterize the location and size of the eyes. We use
autoregression (AR) models to describe the state transitions,
where the location is modeled by a second-order AR and the
scale is modeled by a separate first-order AR. The observa-
tion model is a classification-based model, which tracks eyes
according to the knowledge learned from examples instead
of the templates adapted from previous frames. Therefore, it
can avoid accumulation of the tracking errors. In our work,
we use a regression model in tensor subspace to measure the
posterior probabilities of the observations. Other classifica-
tion/regression models can be used as well. Experimental re-
sults show the capability of the algorithm.
The remaining part of the paper is organized as follows.
In Section 2, the theoretical foundation of the particle filter
is reviewed. In Section 3, the details of the proposed algo-
rithm are presented. The system flowchart in Figure 1 gives
an overview of the algorithm. In Section 4, a systematic ex-
perimental evaluation of the performance is described. The
performance is evaluated from two aspects: the blink detec-
tion rate and the tracking accuracy. The blink detection rate
is evaluated using videos collected under varying scenarios,
and the tracking accuracy is evaluated using benchmark data
collected with the Vicon motion capturing system. Section 5
gives some discussion and concludes the paper.
2. DYNAMIC SYSTEMS AND PARTICLE FILTERS
The fundamental prerequisite of a simultaneous eye tracking
and blink detection system is to accurately recover the dy-
namics of eyes, which can be modeled by a dynamic system.
Open eyes and closed eyes appear to have significantly dif-
ferent appearances. A straightforward way is to model the
dynamics of open-eye and closed-eye individually. We use
two interactive particle filters for this purpose. The poste-
rior probabilities learned by the particle filters are used to
determine which particle filter gives the correct tracks, and
this particle filter is thus labeled as the primary one. Figure 1
gives the diagram of the system. Since the particle filters are
the key part of this blink detection system, in this section,
we present a detailed overview of the dynamic system and its
particle filtering solutions, such that the proposed system for
simultaneous eye tracking and blink detection can be better
understood.
2.1. Dynamic systems
A dynamic system can be described by two mathematical
models. One is the state-transition model, which describes
the system evolution rules, represented by the stochastic pro-
cess {St}∈Rns×1(t=0, 1, ...), where
St=FtSt−1,Vt.(1)
Vt∈Rnv×1is the state transition noise with known proba-
bility density function (PDF) p(Vt). The other one is the ob-
servation model, which shows the relationship between the
observable measurement of the system and the underlying
hidden state variables. The dynamic system is observed at
discrete times tvia realization of the stochastic process, mod-
eled as follows:
Yt=HtSt,Wt.(2)
Yt(t=0, 1, ...) is the discrete observation obtained at time t.
Wt∈Rnwis the observation noise with known PDF p(Wt),
which is independent from Vt. For simplicity, we use capital
letters to refer to the random processes and lowercase letters
to denote the realization of the random processes.
Given that these two system models are known, the prob-
lem is to estimate any function of the state f(St) using the
expectation E[ f(St)|Y0:t]. If Ftand Htare linear, and the
two noise PDFs, p(Vt)andp(Wt), are Gaussian, the sys-
tem can be characterized by a Kalman filter [13]. Unfortu-
nately, Kalman filters only provide the first-order approxi-
mations for general systems. Extended Kalman Filter (EKF)
[13] is one way to handle the nonlinearity. A more general
J. Wu and M. M. Trivedi 3
Predicting/regenerating the
open-eye particles according
to previous eye tracking
Regenerating/predicting the
closed-eye particles according
to previous eye tracking
Generating initial
particles
One set for open-eye
tracking
One set for closed-eye
tracking
Each particle: consider
a binary classification
Each particle: consider
a binary classification
Tensor PCA for feature
extraction
Tensor PCA for feature
extraction
Open-eye/non-eye: posterior
for open-eye
Use logistic regression
Closed-eye/non-eye: posterior
for closed-eye
Use logistic regression
Posterior: open-eye Posterior: closed-eye
Popen >Pclosed
No
Yes
Estimation of the open eye
location
Estimation of the closed eye
location
Output of logistic regression:
weight of each particle
Output of logistic regression:
weight of each particle
Figure 1: Flow-chart for eye blink detection system. For every new frame observation, new particles are first predicted from the known
important distribution, and then updated accordingly based on the posterior estimated by logistic regressor in the tensor subspaces. The
best estimation gives the class label (open-eye/closed-eye) as well as the eye location.
framework is provided by particle filtering techniques. Par-
ticle filtering is a Monte Carlo solution for general form dy-
namic systems. As an alternative to the EKF, particle filters
have the advantage that with sufficient samples, the solutions
approach the Bayesian estimate.
2.2. Review of a basic particle filter
Particle filters are sequential analogues of Markov chain
Monte Carlo (MCMC) batch methods. They are also known
as sequential Monte Carlo (SMC) methods. Particle filters
are widely used in positioning, navigation, and tracking for
modeling dynamic systems [14–20]. The basic idea of par-
ticle filtering is to use point mass, or particles, to represent
the probability densities. The tracking problem can be ex-
pressed as a Bayes filtering problem, in which the posterior
distribution of the target state is updated recursively as a new
observation comes in
pSt|Y0:t∝pYt|St;Y0:t−1St−1
pSt|St−1;Y0:t−1
×pSt−1|Y0:t−1dSt−1.
(3)
The likelihood p(Yt|St;Y0:t−1) is the observation model,
and p(St|St−1;Y0:t−1) is the state transition model.
There are several versions of the particle filters, such
as sequential importance sampling (SIS) [21,22]/sampling-
importance resampling (SIR) [22–24], auxiliary particle fil-
ters [22,25], and Rao-Blackwellized particle filters [20,22,
26,27], and so forth. All particle filters are derived based on
the following two assumptions. The first assumption is that
4 EURASIP Journal on Advances in Signal Processing
the state-transition is a first-order Markov process, which
simplifies the state transition model in (3)to
pSt|St−1;Y0:t−1=pSt|St−1.(4)
The second assumption is that the observations Y1:tare con-
ditionally independent given known states S1:t, which im-
plies that each observation only relies on the current state;
then we have
pYt|St;Y0:t−1=pYt|St.(5)
These two assumptions simplify the Bayes filter in (3)to
pSt|Y0:t∝pYt|StSt−1
pSt|St−1pSt−1|Y0:t−1dSt−1.
(6)
Exploiting this, particle filter uses a number of particles
(ω(i),s(i)
t) to sequentially compute the expectation of any
function of the state, which is E[ f(St)|y0:t], by
EfSt|y0:t=fstpst|y0:tdst=
i
ω(i)
tfs(i)
t.
(7)
In our work, we use the combination of SIS and SIR.
Equation (6) tells us that the estimation is achieved by a pre-
diction step, st−1p(st|st−1)p(st−1|y0:t−1)dst−1, followed by
an update step, p(yt|st). At the prediction step, the new state
si
tis sampled from the state evolution process Ft−1(s(i)
t−1,·)to
generate a new cloud of particle filters. With the predicted
state si
t, an estimate of the observation is obtained, which is
used in the update step to correct the posterior estimate. Each
particle is then reweighted in proportion to the likelihood of
the observation at time t. We adopt the idea of “resampling
when necessary” as suggested by [21,28,29], which suggests
that resampling is only necessary when the effective number
of particles is sufficiently low. The SIS/SIR algorithm can be
summarized as in Algorithm 1.
π(s(i)
t|s(i)
0:t−1,y0:t)=π(s(i)
t|s(i)
t−1,y0:t) is also called
the proposal distribution. A common and simple choice is to
use the prior distribution [30] as the proposal distribution,
which is also known as a bootstrap filter. We use the boot-
strap filter in our work, and by this way the weight update
can be simplified to
ω(i)
t=ω(i)
t−1pyt|s(i)
t.(12)
This indicates that the weight update is directly related to the
observational model.
3. PARTICLE FILTERS FOR EYE TRACKING AND
BLINK DETECTION
The appearance of eyes is presented to have significant
changes when blinks occur. To effectively handle such ap-
pearance changes, we use two interactive particle filters, one
for open eyes and the other one for closed eyes. These two
particle filters are only different in the observation measure-
ment. In the following sections, we present the three ele-
ments of the proposed particle filters: state transition model,
observation model, and prediction/update scheme.
(1) For i=1, ...,N, draw samples from the importance dis-
tributions (prediction step):
s(i)
t∼πst|s0:t−1,y0:t;(8)
(2) Evaluate the importance weights for every particle up to a
normalized constant (update step):
ω(i)
t=ω(i)
t−1
pyt|s(i)
tps(i)
t|s(i)
t−1
πs(i)
t|s(i)
0:t−1,y0:t;(9)
(3) Normalize the importance weights:
ω(i)
t=
ω(i)
t
N
j=1ω(j)
t
,i=1, ...,N; (10)
(4) Compute an estimate of the effective number of the parti-
cles:
Neff=1
N
i=1ω(i)
t; (11)
(5) If Neff<θ,whereθis a given threshold, we perform resam-
pling. Nparticles are drawn from the current particle set
with probabilities proportional to their weights. Replace
the current particle set with this new one, and reset each
new particle’s weight to 1/N.
Algorithm 1: SIS/SIR particle filter.
3.1. State transition model
The system dynamics, which are described by the state vari-
ables, are defined by the location of the eye and the size of
the eye image patches. The state vector is St=(ut,vt;ρt),
where (ut,vt) defines the location and ρtis used to define
the size of eye image patches and normalize them to a fixed
size. In other words, the state vector (ut,vt;ρt) means that the
image patch under study is centered at (ut,vt) and its size is
40ρt×60ρt,where40×60 is the fixed size of the eye patches
we use in our study.
A second-order autoregressive (AR) model is used for es-
timating the eyes’ movement. The AR model has been widely
used in particle filter tracking literature for modeling the mo-
tion. It can be written as
ut=u+Aut−1−u+Bµt,
vt=v+Avt−1−v+Bµt,(13)
where
ut=ut
ut−1,vt=vt
vt−1.(14)
uand vare the corresponding mean values for uand v.As
pointed out by [31],thisdynamicmodelisactuallyatem-
poral Markov chain. It is capable of capturing complicated
J. Wu and M. M. Trivedi 5
object motion. Aand Bare matrices representing the deter-
ministic and the stochastic components, respectively. Aand
Bcan be either obtained by a maximum-likelihood estima-
tion or set manually from prior knowledge. µtis the i.i.d.
Gaussian noise.
We use a first-order AR model to model the scale transi-
tion, which is
ρt−ρ=Cρt−1−ρ+Dηt.(15)
Similar to the motion model, Cis the parameter describing
the system deterministic component, and Dis the parameter
describing the system stochastic component. ρis the mean
value of the scales, and ηtis the i.i.d. measurement noise.
We assu m e ηtis uniformly distributed. The scale is crucial
for many image appearance-based classifiers. An incorrect
scale causes a significant difference in the image appearance.
Therefore, the scale transition model is one of the most im-
portant prerequisites for obtaining an effective particle fil-
ter for measuring the observation. Experimental evaluation
shows that the AR model with uniform i.i.d. noise is appro-
priate for tracking the scale changes.
3.2. Classification-based observation model
In literature, many efforts have been done to address the
problem of selecting the proposal distribution [15,32–35]. A
carefully selected proposal distribution can alleviate the sam-
ple depletion problem, which refers to the problem that the
particle-based posterior approximation collapses over time
to a few particles. For example, in [35], AdaBoost is incor-
porated into the proposal distribution to form a mixture
proposal. This is crucial in some typical occlusion scenarios,
since “cross over” targets can be represented by the mixture-
model. However, the introduction of complicated proposal
distributions greatly increases the computational complex-
ity. Also, since blink detection is usually a single-target track-
ing problem, the proposal distribution is more likely to be
single-mode. Therefore, we only use bootstrap particle filter-
ing approach, and avoid the nontrivial proposal distribution
estimation problem.
In this work, we focus on a better observation model
p(yt|st). The rationale is based on the observation that
combined with the resampling step, a more accurate likeli-
hood learning from a better observation model can move
the particles to areas of high likelihood. This will in turn
mitigate the sample depletion problem, leading to a signif-
icant increase in performance. In literatures, many existing
approaches use simple online template matching [16,18,
19,36] to get the observation model, where the templates
are constructed from low-level features, such as color, edges,
contour, and so forth, from previous observations. The like-
lihood is usually estimated based on a Gaussian distribution
assumption [26,34]. However, such approaches in a large ex-
tent rely on a reasonably stable feature detection algorithm.
Also, usually a large number of the single low-level feature
points are needed. For example, the contour-based method
requires that the state vector be able to describe the evolution
of all contour points. This results in a high-dimensional state
space. Correspondingly, the computational cost is expensive.
One solution is to use abstracted statistics of these single fea-
ture points, such as using color histogram instead of direct
color measurement. However, this causes a loss in the spatial
layout information, which implies a sacrifice in the localiza-
tion accuracy. Instead we use a subspace-based classification
model for measuring the observation such that a more accu-
rate probability evaluation can be obtained. Statistics learned
from a set of training samples are used for classification in-
stead of simple template matching and online updating. This
can greatly alleviate the problem of error accumulation. The
likelihood estimation problem, p(y(i)
t|s(i)
t), becomes a prob-
lem of estimating the distribution of a Bernoulli variable,
which is p(y(i)
t=1|s(i)
t). y(i)
t=1 means that the current
state generates a positive example. In our eye tracking and
blink detection problem, it represents that an eye patch is lo-
cated, including both open eye and closed eye. Logistic re-
gression is a straightforward solution for this purpose. Obvi-
ously, other existing classification/regression techniques can
be used as well.
Such classification-based particle filtering framework
makes simultaneous tracking and recognition feasible and
straightforward. There are two different ways to embed the
recognition problem. The first approach is to use a single par-
ticle filter, whose observation model is a multiclass classifier.
Thesecondapproachistousemultipleparticlefilters,where
foreachparticlefilteritsobservationmodelusesabinary
classifier designed for a specific object class. The particle filter
who gets the highest posterior is used to determine the class
label as well as the object location, and at the next frame t+1,
the other particle filters are reinitialized accordingly. We use
the second approach for simultaneous eye tracking and blink
detection. Individual observation models are built for open
eye and closed eye separately, such that two interactive sets
of particles can be obtained. The observation models contain
two parts: tensor subspace analysis for feature extraction, and
logistic regression for class posterior learning. The two parts
are individually discussed in Sections 3.2.1 and 3.2.2.Poste-
rior probabilities measured by particles from these two par-
ticle filters are individually denoted as po=p(yt=1oe |st)
and pc=p(yt=1ce |st), respectively, where yt=1oe refers to
the presence of an open eye and yt=1ce refers to the presence
of a closed eye.
3.2.1. Subspace analysis for feature extraction
Most existing applications of using particle filters for visual
tracking involve high-dimensional observations. With the in-
crease of the dimensionality in observations, the number of
particles required increases exponentially. Therefore, lower
dimensional feature extraction is necessary. Sparse low-level
features, such as the abstracted statistics of the low-level
features, have been proposed for this purpose. Examples
of the most commonly used features are color histogram
[35,37], edge density [15,38], salient points [39], con-
tour points [18,19], and so forth. The use of such features
makes the system capable of easily accommodating the scale
changes and handling occlusions; however, performance of