Program Evaluation Methods

Manuel Arellano

2010-11

Program

 

1.     Empirical approaches, potential outcomes, and causality

 

1.1.Structural and treatment effect approaches.

1.2.Descriptive analysis vs. causal inference.

1.3.Potential outcomes and causality.

 

2.     Social experiments

 

2.1.Experimental testing of welfare programs in the US.

2.2.Employment effects of job and earnings subsidies.

2.3.Experimental evaluation of labor histories.

2.4.Econometric models of labor histories.

 

3.     Matching

 

3.1.Exogeneity, matching, and multiple regression.

3.2.Methods based on the propensity score.

3.3.The common support condition.

3.4.Monetary incentives and schooling in the UK.

 

4.     Instrumental variables

 

4.1.Instrumental variable estimation using natural experiments.

4.2.Interpreting IV estimates when effects are heterogeneous.

4.3.Local average treatment effects and marginal treatment effects.

4.4.Estimating the distributions of potential outcomes.

4.5.The econometric selection model.

 

5.     Regression-discontinuity

 

5.1.Identification from discontinuities in assignment rules.

5.2.Parametric ans semiparametric estimation methods.

5.3.Financial aid offers and college enrollment decisions.

 

6.     Differences in differences

 

6.1.Comparisons based on policy changes.

6.2.Identifying the average treatment effect for the treated.

6.3.Changes in the distribution of effects vs. changes in means.

6.4.Minimum wages and employment.

 

7.     Further topics

 

7.1.Continuous treatments.

7.2.Treatment effects in duration analysis.

 

 

Readings

 

Lesson 1: Empirical approaches, potential outcomes, and causality

 

 · General

1)     Angrist, J. and A. Krueger (2000): “Empirical Strategies in Labor Economics”, Handbook of Labor Economics, O. Ashenfelter and D. Card (eds.), North Holland, 1277-1366.

2)     Heckman, J. J. (2001): “Micro Data, Heterogeneity, and the Evaluation of Public policy: Nobel Lecture”, Journal of Political Economy, 109, 673-748.

3)     Meyer, B. (1995): “Natural and Quasi-experiments in Economics”, Journal of Business and Economic Statistics, 13, 151-161.

 

 · Potential outcomes and causality

1)     Holland, P. W. (1986): “Statistics and Causal Inference”, Journal of the American Statistical Association, 81, 945-970.

2)      Rubin, D. B. (1974): “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies”, Journal of Educational Psychology, 66, 688-701.

 

 

Lesson 2: Social experiments

 

1)     Banerjee, A. and E. Duflo (2009): “The Experimental Approach to Development Economics”, Annual Review of Economics, 1, 151-178.

2)     Duflo, E., R. Glennerster, and M. Kremer (2008): “Using Randomization in Development Economics Research: A Toolkit”. In T.P. Schultz and J. Strauss (eds.): Handbook of Development Economics, Vol. 4, 3895-3962.

3)     Card, D. and D. R. Hyslop (2005): “Estimating the Effects of a Time-Limited Earnings Subsidy for Welfare-Leavers”, Econometrica, 73, 1723-1770.

4)     Ham, J. C. and R. J. LaLonde (1996): “The Effect of Sample Selection and Initial Conditions in Duration Models: Evidence from Experimental Data on Training”, Econometrica, 64, 175-205.

5)     Hesselius, P., P. Johansson, and L. Larsson (2005): “Monitoring Sickness Insurance Claimants: Evidence from a Social Experiment”, IFAU, Uppsala.

6)     Holla, A. and M. Kremer (2008): “Pricing and Access: Lessons from Randomized Evaluations in Education and Health”. In W. Easterly and J. Cohen (eds.): What Works in Development? Thinking Big vs. Thinking Small, Brookings Institution Press, forthcoming.

7)     LaLonde, R. J. (1995): “Evaluating the Econometric Evaluations of Training Programs with Experimental Data”, American Economic Review, 76, 604-620.

8)     Miguel, E. and M. Kremer (2004): “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities”, Econometrica, 72, 159-217.

9)     Moffitt, R. A. (2004): “The Role of Randomized Field Trials in Social Science Research: A Perspective from Evaluations of Reforms of Social Welfare Programs”, American Behavioral Scientist, 47(5), 506-540.

 

 

Lesson 3: Matching

 

1)     Dearden, L., C. Emmerson, C. Frayne, and C. Meghir (2004): “Can Education Subsidies Stop School Drop-outs? An Evaluation of Education Maintenance in England”, Institute for Fiscal Studies, London.

2)     Heckman, J. J., H. Ichimura, and C. Taber (1997): “Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme”, Review of Economic Studies, 64, 605-654.

3)     Imbens, G. (2004): “Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review”, Review of Economics and Statistics, 86, 4-29.

4)     Rosenbaum, P. R. and D. B. Rubin (1983): “The Central Role of the Propensity Score in Observational Studies for Causal Effects”, Biometrika, 70, 41-55.

5)     Rubin, D. B. (2001): “Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation”, Health Services & Outcomes Research Methodology, 2, 169-188.

 

 

Lesson 4: Instrumental variables

 

 · Methods

1)     Imbens, G. W. and J. Angrist (1994): “Identification and Estimation of Local Average Treatment Effects”, Econometrica, 62, 467-475.

2)     Imbens, G. W. and D. B. Rubin (1997): “Estimating Outcome Distributions for Compliers in Instrumental Variable Models”, Review of Economic Studies, 64, 555-574.

3)     Abadie, A. (2002): “Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models”, Journal of the American Statistical Association, 97, 284-292.

4)     Vytlacil, E. (2002): “Independence, Monotonicity, and Latent Index Models: An Equivalence Results” Econometrica, 70, 331-341.

5)     Heckman, J. J. and E. Vytlacil (2005): “Structural Equations, Treatment Effects, and Econometric Policy Evaluation”, Econometrica, 73, 669-738.

6)     Angrist, J., G. Imbens, and K. Graddy (2000): “The Interpretation of Instrumental Variable Estimators in Simultaneous Equations Models with an Application to the Demand for Fish”, Review of Economic Studies, 67, 499-528.

 

· Examples

1)     Gentzkow, M. and J. M. Shapiro (2008): “Preschool Television Viewing and Adolescent Test Scores: Historical Evidence from the Coleman Study”, Quarterly Journal of Economics, 123, 279-323.

2)     Banerjee, A., S. Cole, E. Duflo, and L. Linden (2007): “Remedying Education: Evidence from Two Randomized Experiments in India”, Quarterly Journal of Economics, 122, 1235-1264.

3)     Moffitt, R. (2008): “Estimating Marginal Treatment Effects in Heterogeneous Populations”, unpublished.

 

 

Lesson 5: Regression-discontinuity

 

1)     Card, D., R. Chetty, and A. Weber (2007): “Cash-on-Hand and Competing Models of Intertemporal Behavior: New Evidence from the Labor Market”, Quarterly Journal of Economics, 122, 1511-1560.

2)     Hahn, J., P. Todd, and W. van der Klaauw (2001): “Estimation of Treatment Effects with a Quasi-Experimental Regression-Discontinuity Design”, Econometrica, 69, 201-209.

3)     Van der Klaauw, W. (2002): “Estimating the Effect of Financial Aid Offers on College Enrollment: A Regression-Discontinuity Approach”, International Economic Review, 43, 1249-1287.

4)     Angrist, J. and V. Lavy (1999): “Using Maimonides’ Rule to Estimate the Effect of Class Size on Scholastic Achievement”, Quarterly Journal of Economics, 114, 533-575.

 

 

Lesson 6: Differences in differences

 

1)     Card, D. and A. Krueger (1994): “Minimum Wages and Employment: A Case Study of the Fast Food Industry”, American Economic Review, 84, 772-793.

2)     Meyer, B., K. Viscusi and D. Durbin (1995): “Workers’ Compensation and Injury Duration: Evidence from a Natural Experiment”, American Economic Review, 85, 322-340.

3)     Athey, S. and G. W. Imbens (2006): “Identification and Inference in Nonlinear Difference-in-differences Models”, Econometrica, 74, 431-497.

4)     Bertrand, M., E. Duflo, and S. Mullainathan (2004): “How Much Should We Trust Differences-in-Differences Estimates?”, Quarterly Journal of Economics, 119, 249-75.

5)     Montalvo, J. G. (2011): “Voting After the Bombing: A Natural Experiment on the Effect of Terrorist Attacks on Democratic Elections”, Review of Economics and Statistics, forthcoming.

 

 

Lesson 7: Further topics

 

· Continuous treatments

1)     Hirano, K. and G. W. Imbens (2004): “The Propensity Score with Continuous Treatments”. Draft chapter for Missing Data and Bayesian Methods in Practice: Contributions by Donald Rubin’s Statistical Family, Wiley, forthcoming.

2)     Florens, J. P., J. Heckman, C. Meghir, and E. Vytlacil (2004): “Identification of Treatment Effects Using Control Functions in Models with Continuous, Endogenous Treatment and Heterogeneous Effects”, Econometrica, 76, 1191-1206.

 

· Treatment effects in duration analysis

1)     Abbring, J. H. and G. J. van den Berg (2003): “The nonparametric identification of treatment effects in duration models”, Econometrica, 71, 1491-1517.

2)     Abbring, J. H. and G. J. van den Berg (2005): “Social Experiments and Instrumental Variables with Duration Outcomes”, unpublished.

 

 

Slides

 

Econometric Methods of Program Evaluation, lecture at Instituto de Estudios Fiscales, Madrid, 14 December 2010.

 

http://www.cemfi.es/~arellano/Pol_Eval_PROGRAM.htm

 

 

(pdf file)