At last--a social scientist's guide through the pitfalls of modern statistical computing

Addressing the current deficiency in the literature on statistical methods as they apply to the social and behavioral sciences, Numerical Issues in Statistical Computing for the Social Scientist seeks to provide readers with a unique practical guidebook to the numerical methods underlying computerized statistical calculations specific to these fields. The authors demonstrate that knowledge of these numerical methods and how they are used in statistical packages is essential for making accurate inferences. With the aid of key contributors from both the social and behavioral sciences, the authors have assembled a rich set of interrelated chapters designed to guide empirical social scientists through the potential minefield of modern statistical computing.

Uniquely accessible and abounding in modern-day tools, tricks, and advice, the text successfully bridges the gap between the current level of social science methodology and the more sophisticated technical coverage usually associated with the statistical field.

Highlights include:

* A focus on problems occurring in maximum likelihood estimation

* Integrated examples of statistical computing (using software packages such as the SAS, Gauss, Splus, R, Stata, LIMDEP, SPSS, WinBUGS, and MATLAB¯®)

* A guide to choosing accurate statistical packages

* Discussions of a multitude of computationally intensive statistical approaches such as ecological inference, Markov chain Monte Carlo, and spatial regression analysis

* Emphasis on specific numerical problems, statistical procedures, and their applications in the field

* Replications and re-analysis of published social science research, using innovative numerical methods

* Key numerical estimation issues along with the means of avoiding common pitfalls

* A related Web site includes test data for use in demonstrating numerical problems, code for applying the original methods described in the book, and an online bibliography of Web resources for the statistical computation

Designed as an independent research tool, a professional reference, or a classroom supplement, the book presents a well-thought-out treatment of a complex and multifaceted field.

JEFF GILL is Associate Professor of Political Science at the University of California, Davis.

MICHAEL P. McDONALD is Assistant Professor of Government and Politics at George Mason University in Fairfax, Virginia.

Preface xi

1 Introduction: Consequences of Numerical Inaccuracy 1

1.1 Importance of Understanding Computational Statistics 1

1.2 Brief History: Duhem to the Twenty-First Century 3

1.3 Motivating Example: Rare Events Counts Models 6

1.4 Preview of Findings 10

2 Sources of Inaccuracy in Statistical Computation 12

2.1 Introduction 12

2.1.1 Revealing Example: Computing the Coefficient Standard Deviation 12

2.1.2 Some Preliminary Conclusions 13

2.2 Fundamental Theoretical Concepts 15

2.2.1 Accuracy and Precision 15

2.2.2 Problems, Algorithms, and Implementations 15

2.3 Accuracy and Correct Inference 18

2.3.1 Brief Digression: Why Statistical Inference Is Harder in Practice Than It Appears 20

2.4 Sources of Implementation Errors 21

2.4.1 Bugs, Errors, and Annoyances 22

2.4.2 Computer Arithmetic 23

2.5 Algorithmic Limitations 29

2.5.1 Randomized Algorithms 30

2.5.2 Approximation Algorithms for Statistical Functions 31

2.5.3 Heuristic Algorithms for Random Number Generation 32

2.5.4 Local Search Algorithms 39

2.6 Summary 41

3 Evaluating Statistical Software 44

3.1 Introduction 44

3.1.1 Strategies for Evaluating Accuracy 44

3.1.2 Conditioning 47

3.2 Benchmarks for Statistical Packages 48

3.2.1 NIST Statistical Reference Datasets 49

3.2.2 Benchmarking Nonlinear Problems with StRD 51

3.2.3 Analyzing StRD Test Results 53

3.2.4 Empirical Tests of Pseudo-Random Number Generation 54

3.2.5 Tests of Distribution Functions 58

3.2.6 Testing the Accuracy of Data Input and Output 60

3.3 General Features Supporting Accurate and Reproducible Results 63

3.4 Comparison of Some Popular Statistical Packages 64

3.5 Reproduction of Research 65

3.6 Choosing a Statistical Package 69

4 Robust Inference 71

4.1 Introduction 71

4.2 Some Clarification of Terminology 71

4.3 Sensitivity Tests 73

4.3.1 Sensitivity to Alternative Implementations and Algorithms 73

4.3.2 Perturbation Tests 75

4.3.3 Tests of Global Optimality 84

4.4 Obtaining More Accurate Results 91

4.4.1 High-Precision Mathematical Libraries 92

4.4.2 Increasing the Precision of Intermediate Calculations 93

4.4.3 Selecting Optimization Methods 95

4.5 Inference for Computationally Difficult Problems 103

4.5.1 Obtaining Confidence Intervals with Ill-Behaved Functions 104

4.5.2 Interpreting Results in the Presence of Multiple Modes 106

4.5.3 Inference in the Presence of Instability 114

5 Numerical Issues in Markov Chain Monte Carlo Estimation 118

5.1 Introduction 118

5.2 Background and History 119

5.3 Essential Markov Chain Theory 120

5.3.1 Measure and Probability Preliminaries 120

5.3.2 Markov Chain Properties 121

5.3.3 The Final Word (Sort of) 125

5.4 Mechanics of Common MCMC Algorithms 126

5.4.1 MetropolisHastings Algorithm 126

5.4.2 Hit-and-Run Algorithm 127

5.4.3 Gibbs Sampler 128

5.5 Role of Random Number Generation 129

5.5.1 Periodicity of Generators and MCMC Effects 130

5.5.2 Periodicity and Convergence 132

5.5.3 Example: The Slice Sampler 135

5.5.4 Evaluating WinBUGS 137

5.6 Absorbing State Problem 139

5.7 Regular Monte Carlo Simulation 140

5.8 So What Can Be Done? 141

6 Numerical Issues Involved in Inverting Hessian Matrices 143
Jeff Gill and Gary King

6.1 Introduction 143

6.2 Means versus Modes 145

6.3 Developing a Solution Using Bayesian Simulation Tools 147

6.4 What Is It That Bayesians Do? 148

6.5 Problem in Detail: Noninvertible Hessians 149

6....