Introduction

This section highlights parts of the Modeling Team’s portfolio that are not directly tied to the CalCAT dashboard. Its purpose is to give users additional context and resources about modeling team projects that extend beyond CalCAT. This section is designed for users who want to explore the broader range of modeling activities and learn more about the analyses and tools the team develops outside the main CalCAT products.

Decision intelligence (DI) tools and case studies

Decisions are an irrevocable allocation of resources toward achieving our goals. Resources include time, energy, assets, money, and people. We all make decisions every day in our personal and professional lives. These decisions vary in complexity, consciousness, and consequences. Decision intelligence is a multi-disciplinary approach to decision making that informs our individual and team decision processes with best practices and suggested supportive tools. Practically, this includes recognizing and resisting cognitive biases, thinking rationally and probabilistically, and structuring decision-making with practical tools like consequence and strategy tables. When needed, decision consequences and their costs can be modeled with decision trees and influence diagrams.

Applying decision intelligence in public health is not trivial, as we are optimizing for population health and not for a specific individual or based on cost. Public health trends and interventions exist amidst complex systems, and tradeoffs involve other sectors with competing priorities (e.g., hospitalizations vs. cost vs. missed days of work or school). By applying decision intelligence, we aspire to improve California’s public health through evidence-based decision support and improved situational awareness, preparedness, and response capabilities.


Appropriate decision intelligence tools are a function of analytical and context complexity (figure credit: Handbook of Decision Analysis, Second Edition [Parnell et al.], adapted by Tomás Aragón). Modeling involves high analytical complexity and therefore intersects heavily with decision analysis approaches. Higher context complexity (related to the nature of individuals and groups involved in the decision) lends itself to the use of decision quality checklists. In the most complex decisions along multiple dimensions, rigorous frameworks like the dialog decision process may be useful to assist in decision-making.

Forecast evaluation tools and background

Forecast evaluation assesses how well models predict future disease activity, such as influenza hospital admissions or other respiratory pathogen trends. In public health, we do this to understand which modeling approaches provide reliable early warning signals, help allocate resources, and support risk communication during rapidly evolving situations. Evaluating forecasts is essential because model performance can vary by season, pathogen evolution, and other changeable conditions like human behavior. For example, model accuracy often declines during periods of rapid epidemic growth or atypical seasonal patterns, reinforcing why continuous evaluation is critical for decision‑making support (Mathis et al. 2024). Model performance can also vary significantly by target location (White et al. 2023, Cramer et al. 2022). Previous work by CDPH scientists highlighted the differences in model performance across different variant periods during the COVID-19 pandemic, as well as variability across different counties within California (White et al. 2023).

Several standardized metrics are commonly used to assess forecasts:

  • Weighted Interval Score (WIS) is a comprehensive scoring metric for probabilistic forecasts; it rewards forecasts that are both accurate and appropriately uncertain (Bracher et al. 2021). Log-transforming WIS scores puts large and small jurisdictions on a more equal footing by reducing the influence of big swings in areas with higher case counts. This makes forecast comparisons fairer and highlights meaningful differences in model performance rather than differences driven simply by population size or count magnitude (Bosse et al. 2023).

  • Mean Absolute Error (MAE) measures how far off a model’s point predictions are, on average, from observed values.

  • Coverage (also known as calibration) refers to how often observed values fall within a model’s prediction intervals—for example, a well‑calibrated model should capture roughly 50% of observations within its 50% prediction interval.

  • For binary events (e.g., whether hospitalizations will increase vs. not), the Brier score evaluates the accuracy of probabilistic predictions by measuring the squared difference between the forecasted probability and the actual outcome. It is best suited for yes/no or categorical event forecasting. Together, these metrics provide complementary views of model accuracy, uncertainty, and calibration, enabling more robust comparisons across teams and seasons.

Several national forecasting hubs provide ongoing real-time evaluation of model performance throughout a season:

SaTScan

Background

Traditionally, infectious disease outbreaks are detected by astute clinicians or epidemiologists who notice unusual increases in case counts. This empirical approach frequently introduces delays in outbreak response, prompting growing interest among health departments in automated surveillance methods capable of flagging potential outbreaks earlier. In practice, the spatiotemporal scan statistic has emerged as the predominant approach (Gleason et al., 2022; Levin-Rector et al., 2024) as implemented in the SaTScan software (SaTScan). The underlying software code is available on Github.

The spatiotemporal scan statistic conceptualizes clusters as three-dimensional cylinders, where the base and height correspond to the spatial (e.g., census tract) and temporal (e.g., days, months, years) dimensions of the data. The algorithm creates all permutations of potential clusters across a defined region and time period, computing a likelihood ratio for each by comparing observed case counts within the cylinder to expected counts derived from a user-specified probability model fit to historical data. The cylinder with the highest likelihood ratio is designated the most likely cluster; secondary clusters are identified iteratively by removing detected clusters and repeating the procedure. Statistical significance is assessed independently for each cluster via Monte Carlo simulation, comparing simulated likelihood ratios from random permutations of case locations against the observed value.

CDPH Models

CDPH’s COVID-19 spatiotemporal cluster‑detection system adapts New York City Department of Health and Mental Hygiene’s SaTScan-based approach (Greene et al., 2021). Using daily polymerase chain reaction (PCR) testing data from the California Reportable Disease Information Exchange (CalREDIE), CDPH identifies abnormal increases in COVID-19 testing percent positivity by including all PCR positives as the events “cases” and all PCR tests as the population denominator. Extensive preprocessing is performed to geocode individuals, identify and exclude residents of congregate settings, and ensure that household and dorm‑based clusters are removed so results better reflect community transmission.

Once data are prepared, CDPH runs 12 SaTScan analyses every week: one statewide and one for each public health region (Regional Public Health Office), each with and without congregate setting residents. Parameter choices were selected based on sensitivity analyses and program priorities. Following SaTScan detection, a network analysis consolidates overlapping clusters into broader community clusters based on shared census tracts. Results are then interpreted within health officer regions and contextualized using Healthy Places Index quartiles to assess potential health inequities. For California local health jurisdictions, these results are displayed on a dashboard available on the CDPH Extranet. Similar statewide dashboards for legionellosis and coccidioidomycosis are available for internal (CDPH) audiences.

Dashboard framework - CalCORVID

CDPH has made the dashboard framework publicly available for adaptation to other surveillance contexts. Users need only supply a SaTScan results file to generate a fully featured R Shiny dashboard that is customizable to jurisdiction-specific data and readily shareable. Key features include out-of-the-box cluster visualization without requiring manipulation of spatial objects, longitudinal views of results over time, interactive layers such as the Social Vulnerability Index (SVI) for contextualizing findings, and support for automating ongoing surveillance workflows. The GitHub repository and corresponding manuscript provide more detail.

West Nile Virus (WNV) modeling and forecasting

West Nile Virus (WNV) is endemic to California, with transmission typically occurring between the months of May to December (most activity concentrated in the Central Valley and Southern California). California Department of Public Health (CDPH), along with the member districts of the Mosquito and Vector Control Agency of California monitor mosquito abundance & infection rates, sentinel chickens, dead birds, and reported human cases (seasonal updates available at Westnile.ca.gov). These surveillance metrics are used to direct mosquito control efforts across the state. During WNV season, this data informs a forecasting challenge that targets the monthly number of symptomatic human cases in six different regions of the state: Inland Southern California, Coastal Southern California, North & South San Joaquin Valley, North & South Sacramento Valley. Several external partners have contributed forecasts in the last two seasons (2024, 2025). The CDPH generates forecasts that rely on machine learning and Bayesian methodologies to detect signals in covariates that are traditionally associated with WNV activity (e.g., temperature, dead bird reports, sentinel chickens, rainfall, among others). In addition, CDPH includes human judgement forecasts in its ensemble to leverage expertise within and outside the organization. Latest forecasts can be downloaded from this year’s repository.

Human judgement forecasting (HJF)

Human judgement forecasting (HJF) is a methodology that relies on subject-matter experts to combine historical data, personal experience, and intuition to predict the outcomes of certain questions, which can be expressed quantitatively or qualitatively. This approach leverages the human ability to combine information from a wide variety of unstructured sources to anticipate outcomes (Farrow et al. 2017, McAndrew et al. 2024).

During the respiratory virus season, CDPH scientists use available surveillance data to predict influenza, COVID-19, and respiratory syncytial virus (RSV) hospital admissions on a weekly basis at 1-4 week horizons. Each participant provides their best estimate of the most likely value, and the interquartile range around it, which is then converted into a predictive cumulative distribution. Submissions from all team members are transformed into a quantile-oriented median HJF ensemble that is displayed on CalCAT and used as part of the CalCAT ensemble for respiratory disease forecasts.

Publications