Studying the Edge of the State Space: An Agent-Based Modeling Approach to Forecasting Rare Events

March 21st, 2014

Miguel Garces, Matthew Reichert, Ian S. Lustick

In this paper, we begin with a thought exercise to generate a definition of rare events that captures those events that pose the ‘rare events problem,’ and introduce the concept of a ‘state space’ as a useful way to conceptualize it. Second, we review three approaches that seek to increase the N of rare event data: one from King, Keohane, and Verba (1994) and two from King and Zeng (2001), identifying the strengths and limitations of each. Next, we propose agent-based modeling as an alternative approach to the rare-event problem that moves past existing limitations. Finally, we provide a proof-of-concept for the agent-based modeling approach, using a model of Virtual Egypt with alternating inductive and process-tracing type techniques to study the rare event of a military takeover in Egypt in July 2013.

Download Here

If you are interested in replicating this experiment, please contact the authors for replication resources.

Using Empirical Data to Test Theoretically Grounded Operationalizations of Protest in an Agent-Based Model

April 10th, 2013

Brandon Alcorn, Miguel Garces, Ian S. Lustick

In this paper we will report on an attempt at Validation & Verification of our agent-based country models using empirical “medium hostility” data from the W-ICEWS program. We will investigate how well our model compares to week-to-week changes in hostility in Bangladesh, Egypt, Pakistan, the Philippines, Thailand, and Venezuela.

Download Here

If you are interested in replicating this experiment, please contact the authors for replication resources.

Verification as a Form of Validation: Deepening Theory to Broaden Application of DOD Protocols to the Social Sciences

September 5th, 2012

Ian S. Lustick, Matthew R. Tubin

The original DOD objective which the process of Verification, Validation, and Accreditation exists to serve was to evaluate the credibility of models and simulations. The red-line between verification (following the blueprint correctly) and validity (accuracy for a particular domain) was convenient, but suppressed consideration of flaws in design concepts. Protocols for evaluating models in domains where theory or data is not unchallenged, e.g. social science, requires clarifying the meaning of verification as “construct validity” and developing a model of verification and validation which situates these operations across multiple levels of an epistemological hierarchy. Pathologies of either inference or generalizability can present themselves at any level and be inherited by others.

Download Here

Granular ABM Simulations for Operational Use: Forecasting and What-if Experiments with Models of Kandahar and Kunduz

August 1st, 2012

Brandon Alcorn, Miguel Garces, Ian S. Lustick

In this paper we report and discuss results of an effort to adapt country-level modeling
techniques developed in DARPA’s Integrated Crisis Early Warning System
(ICEWS) program to produce forecasts and analyses at a much more granular level,
focusing on districts in Kandahar and Kunduz, Afghanistan. The challenge of this
work in Afghanistan is not only to bridge theoretical models to operational needs at
an appropriate level of granularity, but also to overcome modeling obstacles that are
unique to Afghanistan, at least in comparison with the national models that were the
focus of the ICEWS project. We briefly describe our modeling methodology,
including specific steps taken to account for the challenges listed above, before
moving to a analysis of the results of our experiments. Findings and forecasts are reported from baseline models of Kandahar and Kunduz provinces built with the
most recent open-source data available as well as several “what-if” and “in light of”
analyses that explore counterfactual futures of Kandahar and Kunduz provinces
associated with an ISAF drawdown, militia disarmament, and a string of political

If you are interested in replicating this experiment, please contact the authors for replication resources.

Validating and Verifying Validation and Verification: The Methodological Challenge of a Public Policy Imperative

August 1st, 2012
Ian S. Lustick

Verification and Validation are legally required evaluations that the Defense Department
must make of any system, model, or simulation it deploys. With extensive experience and
detailed protocols for verifying and validating natural science based systems, DoD now faces the problem of how to conduct such evaluations for the social science based systems, models, and simulations it is increasingly interested in acquiring. Employing an “epistemological ladder” or hierarchy linking ontology to data, this paper will analyze the verification and validation process as entailing analogous questions about warrantability of inferences and accuracy or generalizability of findings appropriate at any step in the ladder. Analysis of the confusing impact of the “two cultures” idea and its importance within influential philosophy of science texts, along with the particular history of validation and verification concepts within the American military, will be used as the basis for presenting a unified approach to the problem. The usefulness of the approach is tested in a mapping exercise of key debates between quantitatively and qualitatively oriented political science methodologists and of the conventional wisdom reflected in recent DoD documents.

Deploying Constructivism for the Analysis of Rare Events: How Possible is the Emergence of “Punjabistan”?

June 22nd, 2012

Ian S. Lustick; Edited volume by Kanchan Chandra

The purpose of this volume is to demonstrate that a rigorous conceptual framework can enable constructivist insights to be deployed for the solution of a variety of theoretical and empirical problems. In this chapter I offer a use case in which the framework set forth by the editor, if not the exact details of its entire vocabulary, is employed to solve a difficult empirical and policy-relevant problem. The general problem involves is to evaluate a future for Pakistan involving the secession of its Punjabi core – a future whose probability experts have had difficulty assessing. Since secession is itself a rare event, and secession of the center an even rarer event, data relevant to addressing this problem must be generated by a computer simulation model designed and implemented in conformance with available social theories, including constructivist theory, along with information and Pakistani society relevant to the categories of those theories. The thrust of this chapter is to demonstrate that by integrating constructivist approaches to political contestation, via the framework offered in this volume, which specific knowledge of a complex and important case – the future of Punjabi-dominated Pakistan – an agent-based modeling approach can be used to analyze the conditions under which secession of the center can take place and to estimate its likelihood.

If you are interested in replicating this experiment, please contact the authors for replication resources.

Nesting a Data-driven Agent-based Model within a Theoretical Framework

March 15th, 2012

Miguel Garces, Brandon Alcorn

Traditionally agent-based modeling has been a field that focuses largely on abstract models that offer insights into real world phenomena, but rarely with enough complexity to play a major role within academic or policy discussions about particular problems in particular places. In this paper we outline an approach to building country virtualization models that combines traditional micro-level interactions in an agent-based model with macro-level theory to simulate complex political systems in a range of countries in Southeast Asia. We discuss the advantages, and potential downfalls, to adding complexity into agent-based models and present results that highlight some of the analytic advantages to having multiple levels of analysis present in a model and a distribution of counterfactual model results by which we can analyze and forecast political outcomes.

Download Here

If you are interested in replicating this experiment, please contact the authors for replication resources.

Virtual Strategic Analysis and Forecasting Tool (V-SAFT)

December 16th, 2011

Lustick Consulting was awarded a contract through the Office of Naval Research to develop V-SAFT.

We will provide a completed Virtual Strategic Analysis and Forcasting Tool (V-SAFT), an integrated, semi-automatic tool for producing and exploiting virtualized models of complex sociocultural arenas.

The V-SAFT ultimate objective is to provide commanders with the capacity to monitor the velocity, scope, and magnitude of change in politically fragile societies.  Model-building instruments within V-SAFT will double as situation awareness instruments for operators, featuring wiki-style windows into the human terrain of complex societies.  V-SAFT will enhance forecasting capabilities by extending the range of questions commanders can ask and by greatly improving the reliability of answers.  In particular, V-SAFT will be designed to assess the likelihood of rare but high-impact events and to afford trace-back opportunities to support mitigation strategies and contingency planning.

VirThai: A PS-I Implemented Agent-Based Model of Thailand as a Predictive and Analytic Tool

March 17th, 2011

Brandon Alcorn, Miguel Garces, Allen Hicken

In this paper we report on the deployment of “VirThai,” a virtualization type agent-based model of contemporary Thailand, to produce predictions for events of political and policy interest over the course of 2011. Predictions are inferred from distributions of outcomes across large batches of counterfactual futures. We will assess the performance of the model as a forecasting tool, as a technique for understanding the mechanisms that drive political outcomes, and as a means for stimulating new insights or lines of reasoning among country experts.

Download Here

UPDATE: This paper has since been published in the 12th volume of the Stanford Journal of East Asian Affairs. Download Here

Check out Replication Resources for our forecast results.

Forecasting Political Futures For Thailand: Leveraging Social Science Theory in a Virtualization Model

February 10th, 2011

Ian Lustick, Roy Eidelson, Matthew Tubin, Brandon Alcorn, Miguel Garces

This poster was presented at “HSCB Focus 2011: Integrating Social Science Theory and Analytic Methods for Operational Use“.

Download Here

From the website, the purpose of this meeting was to “showcase research and applications in the general HSCB modeling area and to engage OSD HSCB Modeling Program personnel as well as leading scientific and technical experts working in HSCB related fields in a technical exchange. A specific focus of this conference will be to promote communication between the development and user communities and to facilitate the transition of HSCB capabilities into operational use. In addition to personnel from the OSD HSCB Modeling Program, representatives from both DoD and other Government agencies are expected to attend and showcase their programs in this area.  Researchers and developers from industry, academia, and government labs, including current HSCB program awardees, are invited to present their work and ideas related to HSCB technologies. Additionally, representatives from end-user communities within DOD and elsewhere in the US government are strongly encouraged to present requirements, use cases, and challenge problems to the community.”