Design and Analysis of Experiments 2021

11-12:30 PM Eastern Time

Organizer: Min Yang, University of Illinois-Chicago

• William Li, Shanghai Advanced Institute of Finance
  • Title: On optimal designs in information-based optimal subdata - A systematic view of a data reduction strategy with application to second-order model
  • Abstract: With the urgent need of analyzing extraordinary amount of data, information-based optimal subdata selection (IBOSS) approach has gained considerable attention in the recent literature due to its ability of maintaining rich information within the full datset with limited subdata size. On the other hand, there is still lack of systematically exploring the framework, especially on the characterization of the optimal subset, the key step of developing the associated algorithm. Motivated by a real finance case study concerning the impact of corporate attributes on firm value, we systematically explore the framework consisting of the exact steps one can follow when employing the idea of IBOSS for data reduction. Considering the second-order effect model that contains main effects, quadratic effects, and interaction effects, we develop a novel algorithm of selecting an informative subdata. Empirical studies including a real example demonstrate that the new algorithm adequately addresses the trade-off between the computation complexity and statistical efficiency, one of six core research directions for theoretical data science research proposed by the US National Science Foundation.
• Yanxi Liu, University of Illinois at Chicago
  • Title: Information-based Optimal Subdata Selection for Clusterwise Linear Regression
  • Abstract: Technological advancements have accelerated in recent years. The amount of data being collected and the size of the data are increasing exponentially. Over time, it becomes more challenging to deal with not just massive amounts of data but also their complexity. The relationship between input and output variables may not be homogeneous anymore. Conventional statistical models such as generalized linear models (GLMs) may not be well-suited to heterogeneous relationships. Using a Mixture of Expert models is a good solution. The mixture of Expert models can combine different models. It can detect heterogeneous patterns while maintaining the benefits of conventional statistical modeling techniques. It does, however, need a considerable amount of computer resources, particularly when working with huge quantities of data. The subdata approach is a technique for resolving this issue. Inspired by Wang, Yang, and Stufken (2019), the purpose of this project is to develop an algorithm for clusterwise linear regression, a type of Mixture of Experts, to select optimal subdata from the full data set, which preserves the maximum amount of information while requiring minimal computing resources. In this project, the proposed subdata selection is proved to be asymptotically optimal, i.e., no other method is statistically more efficient than the proposed one when the full data size is large.
• Roshan Joseph, Georgia Institute of Technology
  • Title: Supervised compression of big data
  • Abstract: The phenomenon of big data has become ubiquitous in nearly all disciplines, from science to engineering. A key challenge is the use of such data for fitting statistical and machine learning models, which can incur high computational and storage costs. One solution is to perform model fitting on a carefully-selected subset of the data. Various data reduction methods have been proposed in the literature, ranging from random subsampling to optimal experimental design-based methods. However, when the goal is to learn the underlying input-output relationship, such reduction methods may not be ideal, since it does not make use of information contained in the output. To this end, we propose a supervised data compression method called supercompress, which integrates output information by sampling data from regions most important for modeling the desired input-output relationship. An advantage of supercompress is that it is nonparametric – the compression method does not rely on parametric modeling assumptions between inputs and output. As a result, the proposed method is robust to a wide range of modeling choices. We demonstrate the usefulness of supercompress over existing data reduction methods, in both simulations and a taxicab predictive modeling application.

12-1:30 PM Eastern Time

Organizer: John Stufken, University of North Carolina Greensboro

• Jesús López Fidalgo, University of Navarra
  • Title: Active Learning considering the marginal distribution of the covariates
  • Abstract: The Big Data sample size introduces statistical and computational challenges to extract useful information from data sets. The subsampling procedure is widely used to downsize the data volume and allows computing estimators in regression models. Usually, subsampling is performed defining a weight for each point and selecting a subset according to these weights. The subsample can be chosen at random (Passive Learning), but in order to obtain better estimators, the optimal experimental design theory can be used searching for an “influential” sub-sample (Active Learning). This has been developed in the literature for linear and logistic regression, obtaining algorithms based on D-optimality and A-optimality. To the authors knowledge the distribution of the explanatory variables has never been considered for obtaining a subsample. We study the effect of the explanatory variables distribution on the estimation as well as the optimal design. We first assume normality of the covariates and latter we measure the impact of skewness and kurtosis on the estimation and optimal designs. Then, we propose a novel method to obtain optimal subsampling through D-optimality, taking into account the marginal distribution of the covariates. The D-optimal design is computed by an exchange algorithm to obtain the subsample.
• Kalliopi Mylona, King's College London
  • Title: Optimal split-plot designs for precise pure-error estimation of the variance components
  • Abstract: In this work, we present a novel approach to design split-plot experiments which ensures that the two variance components can be estimated from pure error and guarantees a precise estimation of the response surface model. Our novel approach involves a new Bayesian compound D-optimal design criterion which pays attention to both the variance components and the fixed treatment effects. One part of the compound criterion (the part concerned with the treatment effects) is based on the response surface model of interest, while the other part (which is concerned with pure-error estimates of the variance components) is based on the full treatment model. We demonstrate that our new criterion yields split-plot designs that outperform existing designs from the literature both in terms of the precision of the pure-error estimates and the precision of the estimates of the factor effects.

This is a joint work with Steven G. Gilmour (King’s College London) and Peter Goos (KU Leuven)

• Rakhi Singh, UNC Greensboro
  • Title: Design selection for 2-level supersaturated designs
  • Abstract: The commonly used design optimality criteria are inadequate for selecting supersaturated designs. As a result, there is extensive literature on alternative optimality criteria within this context. Most of these criteria are rather ad hoc and are not directly related to the primary goal of experiments that use supersaturated designs, which is factor screening. Especially, unlike almost any other optimal design problem, the criteria are not directly related to the method of analysis. An assumption needed for the analysis of supersaturated designs is the assumption of effect sparsity. Under this assumption, a popular method of analysis for 2-level supersaturated designs is the Gauss-Dantzig Selector (GDS), which shrinks many estimates to 0. We develop new design selection criteria inspired by the GDS and establish that designs that are better under these criteria tend to perform better as screening designs than designs obtained using existing criteria. This presentation is based on joint work with John Stufken, University of North Carolina at Greensboro.

11-12:30 PM Eastern Time

Organizer: Hongquan Xu, University of California, Los Angeles

• Jessica Jaynes, California State University
  • Title: Orthogonal Array Composite Designs for Drug Combination Experiments with Applications for Tuberculosis
  • Abstract: The aim of this research is to provide an overview of the orthogonal array composite design (OACD) methodology, show that they can be robust to missing data under practical scenarios and provide an application for tuberculosis. We compare the 𝐷-efficiencies of OACDs to the commonly used central composite designs (CCD) when there are a few missing observations and demonstrate that OACDs are more robust than the popular CCDs to missing observations for two scenarios. The first scenario assumes one observation is missing either from one factorial point or one additional point. The second scenario assumes two observations are missing either from two factorial points or from two additional points, or from one factorial point and one additional point. Two real-world applications of OACDs pertaining to tuberculosis are provided: a 155-run OACD with nine drugs and a 50-run OACD with six drugs.
• Robert Mee, University of Tennessee
  • Title: Two-level parallel flats designs
  • Abstract: Regular \(2^{n-p}\) designs are also known as single flat designs. Parallel flats designs (PFDs) consisting of three parallel flats (3-PFDs) are the most frequently utilized PFDs, due to their simple structure. Generalizing to \(f\) -PFD with \(f>3\) is more challenging. This talk summarizes recent work on a general theory for the \(f\) -PFD for any \(f\geq 3\). We propose a method for obtaining the confounding frequency vectors for all nonequivalent \(f\) -PFDs, and to find the least \(G\) -aberration (or highest D-efficiency) \(f\) -PFD constructed from any single flat. We also characterize the quaternary code design series as PFDs. Finally, we show how designs constructed by concatenating regular fractions from different families may also have a parallel flats structure. Examples are given throughout to illustrate the results.
• Lin Wang, George Washington University
  • Title: Orthogonal subsampling for big data linear regression
  • Abstract: The dramatic growth of big datasets presents a new challenge to data storage and analysis. Data reduction, or subsampling, that extracts useful information from datasets is a crucial step in big data analysis. We propose an orthogonal subsampling (OSS) approach for big data with a focus on linear regression models. The approach is inspired by the fact that an orthogonal array of two levels provides the best experimental design for linear regression models in the sense that it minimizes the average variance of the estimated parameters and provides the best predictions. The merits of OSS are three-fold: (i) it is easy to implement and fast; (ii) it is suitable for distributed parallel computing and ensures the subsamples selected in different batches have no common data points; and (iii) it outperforms existing methods in minimizing the mean squared errors of the estimated parameters and maximizing the efficiencies of the selected subsamples. Theoretical results and extensive numerical results show that the OSS approach is superior to existing subsampling approaches. It is also more robust to the presence of interactions among covariates and, when they do exist, OSS provides more precise estimates of the interaction effects than existing methods. The advantages of OSS are also illustrated through analysis of real data.

12-1:30 PM Eastern Time

Organizer: David Steinberg, Tel Aviv University

• Susan Murphy, Harvard University
  • Title: Micro-Randomized Trials & Online Decision-Making Algorithms
  • Abstract: A formidable challenge in designing sequential treatments in health is to determine when and in which context it is best to deliver treatments to individuals. Operationally designing the sequential treatments involves the construction of decision rules that input current context of an individual and output a recommended treatment. Micro-randomized experiments, in which each individual is randomized many times can be use to provide data for constructing these decision rules. Further there is much interest in personalization during the experiment, that is, in real time as the individual experiences sequences of treatment. Here we discuss our work in designing online "bandit" learning algorithms for use in personalizing mobile health interventions Reinforcement Learning provides an attractive suite of online learning methods for personalizing interventions in a Digital Health.
• Julie Beckley, Etsy
  • Title: Improving Internet Experiment Validity
  • Abstract: Large scale internet experiments are critical for making the best decisions for customers. While randomized controlled experiments are the cleanest way to measure treatment effects, ensuring experiments are behaving as expected is much harder. Based on my experience at Etsy and Netflix, I will go over several case studies of unexpected challenges and the methodological solutions we implemented to resolve them.
• Weitao Duan, LinkedIn
  • Title: Experiment Velocity and Trustworthiness at LinkedIn
  • Abstract: Controlled experiments, or A/B tests, have been the gold standard for testing a product feature and making launch decisions. Many technology companies, such as Google, Facebook, LinkedIn, and Microsoft, have built large-scale in-house experimentation platforms and fully adopted A/B testing in their decision-making process. Despite the increasing engagement on the site, many of our experiments at LinkedIn suffer from low sample size and low experiment power. To be able to make inference, we adopted a different randomization unit to greatly increase experiment power. In this talk, we will share our lessons and learnings on this. In the second half of the talk, we will discuss the topic of experimentation within an advertising marketplace. We will show that the typical A/B test could lead to severe bias and introduce the budget split design to remove the cannibalization bias.

11-12:30 PM Eastern Time

Organizer: Robert Gramacy, Virginia Tech

• Nathan Wycoff, Georgetown University
  • Title: Learning And Deploying Active Subspaces On Black Box Simulators
  • Abstract: Surrogate modeling of computer experiments via local models, which induce sparsity by only considering short range interactions, can tackle huge analyses of complicated input-output relationships. However, narrowing focus to local scale means that global trends must be relearned over and over again. We first demonstrate how to use Gaussian processes to efficiently perform a global sensitivity analysis on an expensive black box simulator. We next propose a framework for incorporating information from this global sensitivity analysis into the surrogate model as an input rotation and rescaling preprocessing step. We further discuss applications to derivative free optimization via locally defined subspaces. Numerical experiments on observational data and benchmark test functions provide empirical validation.
• Max Balandat, Facebook
  • Title: Multi-Objective Bayesian Optimization over High-Dimensional Search Spaces
  • Abstract: The ability to optimize multiple competing objective functions with high sample efficiency is imperative in many applied problems across science and industry. Multi-objective Bayesian optimization (BO) achieves strong empirical performance on such problems, but even with recent methodological advances, it has been restricted to simple, low-dimensional domains. Most existing BO methods exhibit poor performance on search spaces with more than a few dozen parameters. In this work we propose MORBO, a method for multi-objective Bayesian optimization over high-dimensional search spaces. MORBO performs local Bayesian optimization within multiple trust regions simultaneously, allowing it to explore and identify diverse solutions even when the objective functions are difficult to model globally. We show that MORBO significantly advances the state-of-the-art in sample-efficiency for several high-dimensional synthetic and real-world multi-objective problems, including a vehicle design problem with 222 parameters, demonstrating that MORBO is a practical approach for challenging and important problems that were previously out of reach for BO methods.
• Matthias Poloczek, Amazon
  • Title: Scalable High-dimensional Bayesian Optimization

  • Abstract: Bayesian optimization has become a powerful method for the sample-efficient optimization of expensive black-box functions. These functions do not have a closed-form and are evaluated for example by running a complex simulation of a marketplace, by a physical experiment in the lab or in a market, or by a CFD simulation. Use cases arise in machine learning, e.g., when tuning the configuration of an ML model or when optimizing a reinforcement learning policy. Many of these applications are high-dimensional, i.e., the number of tunable parameters exceeds 20, and thus difficult for current approaches due to the curses of dimensionality and the heterogeneity of the underlying functions. Of particular interest are constrained settings, where we are looking for a solution that satisfies inequality constraints of the form c(x) <= 0 and is globally optimal for the objective function among all feasible solutions. These constrained problems are particularly challenging because the sets of feasible points are often small and non-convex. Due to the lack of sample efficient methods, practitioners usually fall back to evolutionary strategies or heuristics.

    In this talk I will start with a brief introduction to Bayesian optimization and then present the trust region Bayesian optimization algorithm (TuRBO) that addresses the above challenges via a local surrogate and a suitable sampling strategy. Then we will turn our attention to optimization under expensive black-box constraints and introduce the scalable constrained Bayesian optimization algorithm (SCBO). I will show comprehensive experimental results that demonstrate that TuRBO and SCBO achieve excellent results and outperform the state-of-the-art methods.

    References:

    • A Tutorial on Bayesian Optimization
    • A Framework for Bayesian Optimization in Embedded Subspaces
    • Scalable Global Optimization via Local Bayesian Optimization
    • Scalable Constrained Bayesian Optimization

12-1:30 PM Eastern Time

Organizer: Peter Muller, the University of Texas at Austin

• Tianjian Zhou, Colorado State University
  • Title: Probability-of-Decision Designs to Accelerate Dose-Finding Trials
  • Abstract: Cohort-based enrollment can slow down phase I dose-finding trials since the outcomes of the previous cohort must be fully evaluated before the next cohort can be enrolled. This results in frequent suspension of patient enrollment. We propose a class of probability-of-decision (POD) designs to accelerate dose-finding trials, which enable dose assignments in real-time in the presence of pending toxicity outcomes. With uncertain outcomes, the dose assignment decisions are treated as random variables, and we calculate the posterior distribution of the decisions. The posterior distribution reflects the variability in the pending outcomes and allows a direct and intuitive evaluation of the confidence of all possible decisions. Optimal decisions are calculated based on the 0-1 loss, and extra safety rules are constructed to enforce sufficient protection from exposing patients to risky doses. A new and useful feature of POD designs is that they allow investigators and regulators to balance the trade-off between enrollment speed and making risky decisions by tuning a pair of intuitive design parameters. The performances of POD designs are evaluated through numerical studies.
• Daniel Schwartz, University of Chicago
  • Title: Bayesian Uncertainty-Directed Designs with Model Averaging for Faster and More Informative Dose-Ranging Trials
  • Abstract: In this paper we make three contributions to the design and analysis of Phase 2b non-oncology dose-ranging trials, which are critical for drug developers to find the optimal dose to carry forward to Phase 3. First, we use a Bayesian "uncertainty-directed" design (Ventz et al. 2018) that adaptively randomizes patients to doses in a way that explicitly maximizes information about which dose is optimal. This typically means assigning new patients to doses that have been previously understudied relative to how strongly the data suggest they could be the optimal dose. Second, we efficiently and robustly incorporate pharmacological knowledge through Bayesian model averaging of parametric dose-response curves. And third, we provide very fast posterior computation for this Bayesian adaptive design using a Sequential Monte Carlo algorithm that makes it easier for trialists to conduct extensive simulation studies to reliably check Frequentist error. These practical designs show promise to accelerate Phase 2b trials and produce higher quality evidence before Phase 3.
• Meizi Liu, University of Chicago
  • Title: PoD-BIN: A Probability of Decision Bayesian Interval De- sign for Time-to-Event Dose-Finding Trials with Multiple Toxicity Grades
  • Abstract: We consider a Bayesian framework based on “probability of decision” for dose- finding trial designs. The proposed PoD-BIN design evaluates the posterior predictive probabilities of up-and-down decisions. In PoD-BIN, multiple grades of toxicity, catego- rized as mild toxicity (MT) and dose-limiting toxicity (DLT), are modeled simultaneously, and the primary outcome of interests is time-to-toxicity for both MT and DLT. This allows the possibility of enrolling new patients when previously enrolled patients are still being fol- lowed for toxicity, thus potentially shortening the trial length. The Bayesian decision rules in PoD-BIN utilize the probability of decisions to balance the trade-off between the need to speed up the trial and the risk of exposing patients to overly toxic doses. We demonstrate via numerical examples the resulting trade-off of speed and safety of PoD-BIN and com- pare it to existing designs. PoD-BIN appears to be able to control the frequency of making risky decisions and, at the same time, shorten the trial duration in the simulation.

11-12:30 PM Eastern Time

Organizer: Nancy Flournoy, University of Missouri

• Richard Emsley, King's College London.
  • Title: Frequentist and Bayesian approaches to rescuing disrupted trials
  • Abstract: There is a severe threat to the validity of clinical trials that were underway before the COVID-19 pandemic and potentially huge research waste. Many studies were paused and recruitment restarted without due consideration of whether all studies should restart, or to required changes in sample sizes. There is also a need for solutions to practical and statistical issues (e.g. increased missing data) that have arisen from the virus and its sequelae. This talk will present some of these challenges, and discuss frequentist and Bayesian approaches to rescuing disrupted trials. The contents are drawn from a report on this topic from the NISS Ingram Olkin Forum Series on Unplanned Clinical Trial Disruptions.
• Kelly Van Lancker, Ghent University.
  • Title: Potential estimands and estimators for clinical trials impacted by COVID-19
  • Abstract: The COVID-19 pandemic continues to affect the conduct of clinical trials of medical products globally. Complications may arise from pandemic-related operational challenges such as site closure, travel limitations and interruptions to the supply chain for the investigational product, or from health-related challenges such as COVID-19 infected trial participants. Some of these complications lead to unforeseen intercurrent events in the sense that they affect either the interpretation or the existence of the measurements associated with the clinical question of interest. The ICH E9(R1) Addendum on estimands provides a rigorous basis to discuss potential pandemic-related trial disruptions and embed them in the context of study objectives and design elements. In this talk, we focus on the use of the hypothetical strategy and to a lesser extent the treatment-policy strategy to frame clinical questions in the presence of the unforeseen intercurrent events due to the COVID-19 pandemic. It should be noted that different hypothetical strategies could be considered and care has to be taken that the envisaged scenario, in which the intercurrent event would not occur, is precisely described. For their estimation, we will consider different causal inference and missing data methods such as multiple imputation and (augmented) inverse probability weighting. To clarify, we describe the features of a stylized trial in neuroscience, and how it may have been impacted by the pandemic. This stylized trial will then be re-visited by discussing the changes to the estimand and the estimator to account for pandemic disruptions.
• Diane Uschner, George Washington University.
  • Title: Randomization tests toaddress disruptions inclinical trials
  • Abstract: In early 2020, the World Health Organization declared the novel corona virus disease (COVID-19) a pandemic. On top of prompting various trials to study treatments and vaccines for COVID-19, COVID-19 also had numerous consequences for ongoing clinical trials. People around the globe restricted their daily activities to minimize contagion, which led to missed visits and cancelling or postponing of elective medical treatments. For some clinical indications, COVID-19 may lead to a change in the patient population or treatment effect heterogeneity. We will measure the effect of the disruption on randomization tests and derive a methodological framework for randomization tests that allows for the assessment of clinical trial disruptions. We show that randomization tests are robust against clinical trial disruptions in certain scenarios, namely if the disruption can be considered an ancillary statistic to the treatment effect. As a consequence, randomization tests maintain type I error probability and power at their nominal levels.

12-1:30 PM Eastern Time

Organizer: Dave Woods, University of Southampton

• Tim Waite, University of Manchester, UK
  • Title: Minimax efficient random experimental design strategies with application to model-robust design for prediction
  • Abstract: Fisher stressed the importance of randomizing an experiment via random permutation of the allocation of treatments to experimental units; in an industrial context this usually amounts to randomizing the run order of the design. In this talk we take the idea of experimental randomization much further by introducing flexible new random design strategies in which the design to be applied is chosen at random from a distribution of possible designs. We discuss the philosophical justification for doing so from a game-theoretic perspective and it is shown that the new strategies give stronger bounds on both the expectation and survivor function of the loss distribution. The consequences of this approach are explored in several problems, including global prediction from a linear model contaminated by a discrepancy function from an L2-class. In this problem the performance improvement is dramatic: the new approach gives bounded expected loss, in contrast to previous designs for which the expected loss was unbounded.
• Lida Mavrogonatou, University of Cambridge
  • Title: Optimal Bayesian experimental design for model selection through minimisation of f-divergences
  • Abstract: A systematic understanding of studied phenomena has been embraced in a range of scientific disciplines where a collection of components are studied as parts of a system rather than as isolated processes. As direct observation of the studied system is often not possible, observable information is collected through experiments and subsequently used for inference of unobservable components. Given a predefined budget, Bayesian optimal experimental design methods are often employed to identify the most useful (in terms of a targeted objective) experimental conditions while accounting for potential sources of uncertainty. Unfortunately, currently adopted methods fail to address challenges arising within a modern scientific framework, due to the increased computational complexity of models that can realistically capture the studied structures. In this talk, I will present an efficient estimation framework that is shown to overcome ongoing challenges through the use of variational approximation methods. The proposed approach is applicable to optimal experimental design problems for model selection. A suitable class of metrics that are used to quantify the benefit from each experimental condition (commonly known as utility functions) is established in which the benefit is expressed as an f-divergence between predictive distributions of the competing models.
• Lulu Kang, Illinois Institute of Technology
  • Title: A Maximin Φp-Efficient Design for Multivariate Generalized Linear Models
  • Abstract: Experimental designs for a generalized linear model (GLM) often depend on the specification of the model, including the link function, the predictors, and unknown parameters, such as the regression coefficients. To deal with the uncertainties of these model specifications, it is important to construct optimal designs with high efficiency under such uncertainties. Existing methods such as Bayesian experimental designs often use prior distributions of model specifications to incorporate model uncertainties into the design criterion. Alternatively, one can obtain the design by optimizing the worst-case design efficiency with respect to the uncertainties of model specifications. In this work, we propose a new Maximin \(\Phi_p\) -Efficient (or Mm-\(\Phi_p\) for short) design which aims at maximizing the minimum \(\Phi_p\) -efficiency under model uncertainties. Based on the theoretical properties of the proposed criterion, we develop an efficient algorithm with sound convergence properties to construct the Mm-\(\Phi_p\) design. The performance of the proposed Mm-\(\Phi_p\) design is assessed through several numerical examples.

• Derek Bingham, Simon Fraser University
• Nancy Flournoy, University of Missouri

• Bradley Jones, SAS
• Natee Ting, Boehringer-Ingelheim Pharmaceuticals Inc.

• Ying Hung, Rutges
• Min Yang, University of Illinois Chicago

• Angela Dean, Ohio State University
• Dibyen Majumdar, University of Illinois-Chicago
• Max Morris, Iowa State University
• Werner G. Müller, Johannes Kepler University Linz

1. Nicholas Alfredo Larsen, North Carolina State University, Department of Statistics, for the poster: HODOR: A two-stage Hold-Out Design for Online Randomized experiments
2. Torsten Reuter, Otto von Guericke University Magdeburg, for the poster: Optimal Subsampling Design for Big Data Regression
3. Mohammed Saif Ismail Hameed, KU Leuven, for the poster: A tailored analysis of data from OMARS designs
4. (Honorable mention) Gautham Sunder, Carlson School of Management, for the poster: Hyperparameter Optimization of Deep Neural Networks with Application to Medical Device Manufacturing