For both groups we combined all the answers in which subjects gave the same confidence
assessment and calculated how often they were right. Good calibration means that the
fraction of correct answers should be about equal to the stated confidence level. For example,
on questions where subjects said they were 80 percent confident, they should be right about
80 percent of time. If subjects are well calibrated, then a graph of the percent correct against
the confidence levels should lie near a 45◦ line.
To keep our analysis focused, we abstract from other channels emphasized
in the literature through which expectations may have an immediate impact
on current economic activity such as time-to-build, capital adjustment costs,
or consumption smoothing. Also, it should be clear that we do not believe
that the economic expansion experienced by the U.S. economy during the
second half of the 1990s was entirely driven by expectations of future higher
Naively collecting translations by crowdsourcing the task to non-professional translators yields disﬂuent, low-quality results if no quality control is exercised. We demonstrate a variety of mechanisms that increase the translation quality to near professional levels. Speciﬁcally, we solicit redundant translations and edits to them, and automatically select the best output among them.