Defining and assessing immediacy in single‐case experimental designs

Tincani M., Travers J.

Questionable research practices (QRPs) are a variety of research choices that introduce bias into the body of scientific literature. Researchers have documented widespread presence of QRPs across disciplines and promoted practices aimed at preventing them. More recently, Single-Case Experimental Design (SCED) researchers have explored how QRPs could manifest in SCED research. In the chapter, we describe QRPs in participant selection, independent variable selection, procedural fidelity documentation, graphical depictions of behavior, and effect size measures and statistics. We also discuss QRPs in relation to the file drawer effect, publication bias, and meta-analyses of SCED research. We provide recommendations for researchers and the research community to promote practices for preventing QRPs in SCED.

Jamshidi L., Heyvaert M., Declercq L., Fernández-Castilla B., Ferron J.M., Moeyaert M., Beretvas S.N., Onghena P., Van den Noortgate W.

Ledford J.R.

Slocum et al. (this issue) provide well-reasoned arguments for the use of nonconcurrent multiple baseline designs in behavior analytic work, despite historical preference for concurrent designs (i.e., simultaneous baseline initiation) and contemporary guidelines in related fields suggesting that nonconcurrent designs are insufficient for evaluating functional relations (What Works Clearinghouse, 2020). I provide a commentary, highlighting major contributions of this article and suggesting areas of further consideration. In sum, I agree with authors that researchers should avoid wholesale dismissal of nonconcurrent designs. I also agree that understanding how multiple-baseline designs control for and allow for detection of threats to internal validity is critical so that authors can apply the variation of the design that allows them to draw confident conclusions about relations between independent and dependent variables.

Aydin O., Tanious R.

Visual analysis and nonoverlap-based effect sizes are predominantly used in analyzing single case experimental designs (SCEDs). Although they are popular analytical methods for SCEDs, they have certain limitations. In this study, a new effect size calculation model for SCEDs, named performance criteria-based effect size (PCES), is proposed considering the limitations of 4 nonoverlap-based effect size measures, widely accepted in the literature and that blend well with visual analysis. In the field test of PCES, actual data from published studies were utilized, and the relations between PCES, visual analysis, and the 4 nonoverlap-based methods were examined. In determining the data to be used in the field test, 1,052 tiers (AB phases) were identified from 6 journals. The results revealed a weak or moderate relation between PCES and nonoverlap-based methods due to its focus on performance criteria. Although PCES has some weaknesses, it promises to eliminate the causes that may create issues in nonoverlap-based methods, using quantitative data to determine socially important changes in behavior and to complement visual analysis.

Slocum T.A., Pinkelman S.E., Joslyn P.R., Nichols B.

Multiple baseline designs—both concurrent and nonconcurrent—are the predominant experimental design in modern applied behavior analytic research and are increasingly employed in other disciplines. In the past, there was significant controversy regarding the relative rigor of concurrent and nonconcurrent multiple baseline designs. The consensus in recent textbooks and methodological papers is that nonconcurrent designs are less rigorous than concurrent designs because of their presumed limited ability to address the threat of coincidental events (i.e., history). This skepticism of nonconcurrent designs stems from an emphasis on the importance of across-tier comparisons and relatively low importance placed on replicated within-tier comparisons for addressing threats to internal validity and establishing experimental control. In this article, we argue that the primary reliance on across-tier comparisons and the resulting deprecation of nonconcurrent designs are not well-justified. In this article, we first define multiple baseline designs, describe common threats to internal validity, and delineate the two bases for controlling these threats. Second, we briefly summarize historical methodological writing and current textbook treatment of these designs. Third, we explore how concurrent and nonconcurrent multiple baselines address each of the main threats to internal validity. Finally, we make recommendations for more rigorous use, reporting, and evaluation of multiple baseline designs.

Kratochwill T.R., Horner R.H., Levin J.R., Machalicek W., Ferron J., Johnson A.

In this paper, we provide a critique focused on the What Works Clearinghouse (WWC) Standards for Single-Case Research Design (Standards 4.1). Specifically, we (a) recommend the use of visual-analysis to verify a single-case intervention study's design standards and to examine the study's operational issues, (b) identify limitations of the design-comparable effect-size measure and discuss related statistical matters, (c) review the applicability and practicality of Standards 4.1 to single-case designs (SCDs), and (d) recommend inclusion of content pertaining to diversity, equity, and inclusion in future standards. Within the historical context of the WWC Pilot Standards for Single-Case Design (1.0), we suggest that Standards 4.1 may best serve as standards for meta-analyses of SCDs but will need to make clear distinctions among the various types of SCD studies that are included in any research synthesis. In this regard, we argue for transparency in SCD studies that meet design standards and those that do not meet design standards in any meta-analysis emanating from the WWC. The intent of these recommendations is to advance the science of SCD research both in research synthesis and in promoting evidence-based practices.

Barnard-Brak L., Watkins L., Richman D.

• We examined the optimal number of baseline sessions that produced minimal bias. • As the number of baseline sessions increased, the bias in effect size estimates decreased. • Second, we examined what would be the minimum number of baseline sessions associated with varying levels of bias. • As the number of baseline sessions increases, the standard deviation for the phase decreased. • When considering five or ten percent bias, the optimal level of standard deviation was 0.59 or less. Recommendations vary considerably for the minimum or optimal number of baseline sessions to conduct within single-case experimental design clinical analyses or research studies. We examined the optimal number of baseline sessions that produced minimal bias. First, we examined the relation between the number of baseline sessions and the degree of bias in calculating estimates of treatment effect size. As the number of baseline sessions increased, the bias in effect size estimates decreased, r = -0.36, p < 0.001. s, we examined what would be the minimum number of baseline sessions associated with varying levels of bias. Bias of approximately ten percent was associated with four to five baseline sessions. Bias of about five percent was associated with six to seven baseline sessions. Third, we examined the relation between standard deviation and varying levels of bias. As the number of baseline sessions increases, the standard deviation for the phase decreased, r = -0.89, p < 0.001. Fourth, we examined what value of standard deviation in the baseline phase was associated with equal to or more than five versus ten percent bias. When considering five or ten percent bias, the optimal level of standard deviation was 0.59 or less.

Epstein L.H., Bickel W.K., Czajkowski S.M., Paluch R.A., Moeyaert M., Davidson K.W.

The biomedical research community has long recognized that much of the basic research being conducted, whether in the biological, behavioral or social sciences, is not readily translated into clinical and public health applications. This translational gap is due in part to challenges inherent in moving research findings from basic or discovery research to applied research that addresses clinical or public health problems. In the behavioral and social sciences, research designs typically used in the early phases of translational research are small, underpowered "pilot" studies that may lack sufficient statistical power to test the research question of interest. While this approach is discouraged, these studies are often employed to estimate effect sizes before embarking on a larger trial with adequate statistical power to test the research hypothesis. The goal of this paper is to provide an alternative approach to early phase studies using single case designs (SCDs).Review basic principles of SCDs; provide a series of hypothetical SCD replication experiments to illustrate (1) how data from SCDs can be analyzed to test the effects of an intervention on behavioral and biological outcomes and (2) how sample sizes can be derived for larger randomized controlled trials (RCTs) based on clinically meaningful effects from SCDs; and review feedback between SCDs and RCTs.The paper illustrates the use of SCD reversal and multiple baseline designs for early phase translational research.SCDs provide a flexible and efficient platform for the use of experimental methods in early phase translational research. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Fingerhut J., Xu X., Moeyaert M.

A variety of measures have been developed to quantify intervention effects for single-case experimental design studies. Within the family of non-overlap indices, the Tau-U measure is one of the mos...

Manolov R., Moeyaert M., Fingerhut J.E.

Natesan Batley P., Hedges L.V.

Although statistical practices to evaluate intervention effects in single-case experimental design (SCEDs) have gained prominence in recent times, models are yet to incorporate and investigate all their analytic complexities. Most of these statistical models incorporate slopes and autocorrelations, both of which contribute to trend in the data. The question that arises is whether in SCED data that show trend, there is indeterminacy between estimating slope and autocorrelation, because both contribute to trend, and the data have a limited number of observations. Using Monte Carlo simulation, we compared the performance of four Bayesian change-point models: (a) intercepts only (IO), (b) slopes but no autocorrelations (SI), (c) autocorrelations but no slopes (NS), and (d) both autocorrelations and slopes (SA). Weakly informative priors were used to remain agnostic about the parameters. Coverage rates showed that for the SA model, either the slope effect size or the autocorrelation credible interval almost always erroneously contained 0, and the type II errors were prohibitively large. Considering the 0-coverage and coverage rates of slope effect size, intercept effect size, mean relative bias, and second-phase intercept relative bias, the SI model outperformed all other models. Therefore, it is recommended that researchers favor the SI model over the other three models. Research studies that develop slope effect sizes for SCEDs should consider the performance of the statistic by taking into account coverage and 0-coverage rates. These helped uncover patterns that were not realized in other simulation studies. We underline the need for investigating the use of informative priors in SCEDs.

Natesan Batley P., Nandakumar R., Palka J.M., Shrestha P.

Recently, there has been an increased interest in developing statistical methodologies for analyzing single case experimental design (SCED) data to supplement visual analysis. Some of these are simulation-driven such as Bayesian methods because Bayesian methods can compensate for small sample sizes, which is a main challenge of SCEDs. Two simulation-driven approaches: Bayesian unknown change-point model (BUCP) and simulation modeling analysis (SMA) were compared in the present study for three real datasets that exhibit “clear” immediacy, “unclear” immediacy, and delayed effects. Although SMA estimates can be used to answer some aspects of functional relationship between the independent and the outcome variables, they cannot address immediacy or provide an effect size estimate that considers autocorrelation as required by the What Works Clearinghouse (WWC) Standards. BUCP overcomes these drawbacks of SMA. In final analysis, it is recommended that both visual and statistical analyses be conducted for a thorough analysis of SCEDs.

Lancioni G.E., Desideri L., Singh N.N., Sigafoos J., O’Reilly M.F.

Manolov R., Tanious R.

The current text deals with the assessment of consistency of data features from experimentally similar phases and consistency of effects in single-case experimental designs. Although consistency is frequently mentioned as a critical feature, few quantifications have been proposed so far: namely, under the acronyms CONDAP (consistency of data patterns in similar phases) and CONEFF (consistency of effects). Whereas CONDAP allows assessing the consistency of data patterns, the proposals made here focus on the consistency of data features such as level, trend, and variability, as represented by summary measures (mean, ordinary least squares slope, and standard deviation, respectively). The assessment of consistency of effect is also made in terms of these three data features, while also including the study of the consistency of an immediate effect (if expected). The summary measures are represented as points on a modified Brinley plot and their similarity is assessed via quantifications of distance. Both absolute and relative measures of consistency are proposed: the former expressed in the same measurement units as the outcome variable and the latter as a percentage. Illustrations with real data sets (multiple baseline, ABAB, and alternating treatments designs) show the wide applicability of the proposals. We developed a user-friendly website to offer both the graphical representations and the quantifications.

Katz B.R., Lattal K.A.

Among the tactics of experimental science discussed by Sidman (1960) were those used to study transitional behavior. Drawing from his insights, this review considers an often cited but infrequently analyzed aspect of the transition from reinforcement to extinction: the extinction burst. In particular, the review seeks to answer the question posed in its title. The generic definition of an extinction burst as an increase in response rate following the onset of extinction is found to be wanting, raising more questions than it answers. Because questions of definition in science usually come down to those of measurement, the answer to the title's question is suggested to be found in how behavior prior to extinction is maintained and measured, when and how extinction is introduced, and where in time and how behavior early in extinction is measured. This analysis suggests that a single, uniform, and precise definition of the extinction burst is misguided. Examining how each of these facets contributes to what has been described generically as the extinction burst is a small, but important, part of Sidman's methodological legacy to the experimental analysis of behavior.

Manolov R., Rochat L.

Cox S.K., Root J.R., McConomy A., Davis K.

Replications provide credibility by demonstrating under what conditions experimental findings can be repeated, the premise behind evidence-based practices. Replications in single-case research also investigate generalization of findings across groups. For groups with high variability, such as individuals with autism, assumptions of generalizability should be based on learners who are similar in critical ways. The purpose of this study was to use Coyne et al.'s framework for replication and the next generation guidelines for single-case research to extend understanding of “for whom” and “under what conditions” modified schema-based instruction (an established evidence-based practice for individuals with autism) is effective. In this distal conceptual replication of Root et al., contextual and instructional variables of theoretical and practical importance were intentionally manipulated or maintained and reported to model transparency and support replicability. Four high school students receiving special education under the Individuals With Disabilities Education Act category of autism were taught mathematical and social problem-solving behaviors within the context of percentage-of-change word problems. Researchers used modified schema-based instruction and augmented reality in a one-on-one setting and assessed generalization to purchasing in the food court of a mall biweekly. We frame our discussion around the recommendations for replication research from Coyne et al. and recommendations for single-case research from Ledford et al., concluding with suggestions for future replications that use single-case research designs.

Manolov R., Tanious R.

AbstractOverlap is one of the data aspects that are expected to be assessed when visually inspecting single-case experimental designs (SCED) data. A frequently used quantification of overlap is the Nonoverlap of All Pairs (NAP). The current article reviews the main strengths and challenges when using this index, as compared to other nonoverlap indices such as Tau and the Percentage of data points exceeding the median. Four challenges are reviewed: the difficulty in representing NAP graphically, the presence of a ceiling effect, the disregard of trend, and the limitations in using p-values associated with NAP. Given the importance of complementing quantitative analysis and visual inspection of graphed data, straightforward quantifications and new graphical elements for the time-series plot are proposed as options for addressing the first three challenges. The suggestions for graphical representations (representing within-phase monotonic trend and across-phases overlaps) and additional numerical summaries (quantifying the degree of separation in case of complete nonoverlap or the proportion of data points in the overlap zone) are illustrated with two multiple-baseline data sets. To make it easier to obtain the plots and quantifications, the recommendations are implemented in a freely available user-friendly website. Educational researchers can use this article to inform their use and application of NAP to meaningfully interpret this quantification in the context of SCEDs.

Manolov R., Onghena P.

AbstractRandomization tests represent a class of significance tests to assess the statistical significance of treatment effects in randomized single-case experiments. Most applications of single-case randomization tests concern simple treatment effects: immediate, abrupt, and permanent changes in the level of the outcome variable. However, researchers are confronted with delayed, gradual, and temporary treatment effects; in general, with “response functions” that are markedly different from single-step functions. We here introduce a general framework that allows specifying a test statistic for a randomization test based on predicted response functions that is sensitive to a wide variety of data patterns beyond immediate and sustained changes in level: different latencies (degrees of delay) of effect, abrupt versus gradual effects, and different durations of the effect (permanent or temporary). There may be reasonable expectations regarding the kind of effect (abrupt or gradual), entailing a different focal data feature (e.g., level or slope). However, the exact amount of latency and the exact duration of a temporary effect may not be known a priori, justifying an exploratory approach studying the effect of specifying different latencies or delayed effects and different durations for temporary effects. We provide illustrations of the proposal with real data, and we present a user-friendly freely available web application implementing it.

Manolov R., Lebrault H., Krasny-Pacini A.