--- title: "Evaluating transitions" author: "Written by [Mohammed Saqr](https://saqr.me) and [Sonsoles López-Pernas](https://sonsoles.me)" output: rmarkdown::html_vignette: toc: true toc_depth: 2 vignette: > %\VignetteIndexEntry{Evaluating transitions} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r options, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>", fig.width = 6.5, fig.height = 4.5, fig.align = "center", dpi = 72) old_options <- options(digits = 3, width = 90) ``` A fitted lag sequential model is an estimate, not a finding. The adjusted residual tests each transition against independence, but that test rests on large-sample assumptions that sequential data can violate: transitions within one sequence are not independent observations, and a few long or atypical sequences can carry a large share of the table. A claim about a transition therefore needs evidence matched to its scope. A claim about one edge needs an estimate of how precisely that edge is known; a claim that a significant edge is real needs to survive the loss of part of the sample; a claim about the whole network needs to reproduce in independent halves of the data; and a claim that the structure exceeds chance needs a null that does not rest on the residual's approximation. `lagdynamics` pairs each kind of claim with a procedure. All of them take the fitted model as their first argument and are read with `transitions()`, the same verb that reads the fit. | Claim | Evidence | Function | |---|---|---| | a transition is precisely estimated | edge-level uncertainty | `certainty_lsa()`, `bootstrap_lsa()` | | a significant transition is not fragile | robustness to case loss | `stability_lsa()` | | the whole network is reproducible | split-half reliability | `reliability_lsa()` | | the structure exceeds chance | an empirical null | `permute_lsa()` | Differences between groups are a claim of their own and are covered in [Comparing groups](https://pak.dynasite.org/lagdynamics/articles/group-comparison.html). ## Data `engagement` records the weekly engagement state of 138 students over 15 weeks; two rows are entirely missing, so the model uses 136 sequences. - **Sequence:** each row is one student. - **Order:** the column order, weeks 1 to 15. - **State:** `Active`, `Average`, or `Disengaged`. The student is the unit that all resampling procedures below draw, drop, or split, because the transitions of one student are dependent. ```{r fit} library(lagdynamics) fit <- lsa(engagement) transitions(fit, significant = TRUE, sort = "strength") ``` Seven of the nine transitions are significant: the three self-transitions are over-represented, and the four moves into or out of `Active` are avoided. The procedures below ask how much of this survives scrutiny. ## Edge uncertainty A claim about one transition requires an estimate of how much it could vary. `certainty_lsa()` derives that uncertainty analytically: each state's outgoing transitions are modelled as Dirichlet-Multinomial, which gives a posterior mean, a credible interval, and a certainty decision for every transition probability without resampling. ```{r certainty} cert <- certainty_lsa(fit) cert transitions(cert) ``` The persistence `Active -> Active` is pinned down tightly: a probability of 0.698 with a 95% credible interval from 0.661 to 0.731. Two edges are not classed as stable. `Disengaged -> Active` has an interval from 0.088 to 0.158 around a probability of 0.120, wide relative to its size because `Disengaged` is the least visited source state, and `Active -> Disengaged` (0.035, interval 0.023 to 0.051) is equally imprecise relative to its size. A residual can be large while the probability behind it is still uncertain, which is why both are reported. `bootstrap_lsa()` reaches the same question by resampling. It draws students with replacement, refits the model on each resample, and records a percentile interval for the count, the residual, the probability, and Yule's Q of every edge (Efron, 1979). Resampling whole sequences preserves the dependence within each student that the analytic model ignores, so the bootstrap is preferred when students may differ systematically. The seed is set once here and governs every resampling procedure in this article; the bootstrap draws 500 resamples. ```{r bootstrap} set.seed(2026) boot <- bootstrap_lsa(fit, R = 500) boot transitions(boot, significant = TRUE) |> head(4) ``` Seven of the nine edges are stable under the bootstrap: their residual interval excludes zero. `transitions(boot, significant = TRUE)` keeps those seven, and the first four are shown. The residual of `Active -> Active` ranges from 18.8 to 24.31 across resamples, far from zero. Its probability interval, 0.640 to 0.748, is wider than the analytic credible interval (0.661 to 0.731) because resampling students carries the variation between students that the Dirichlet-Multinomial model treats as absent. `Disengaged -> Active` shows why the two procedures answer different questions: its residual interval (-12.6 to -8.35) is firmly negative, so the avoidance itself is not in doubt, while its probability (0.080 to 0.164) remains imprecise. The bootstrap forest shows the interval of every edge at once. ```{r forest, fig.height = 5} plot(boot) ``` Each spoke is one edge: the square is the bootstrap estimate of the adjusted residual, the bar its 95% interval, and the dashed ring the null value of zero. The seven significant edges lie clearly inside or outside the ring, over-represented edges outside and avoided edges inside, while the two grey, non-significant edges, `Average -> Disengaged` and `Disengaged -> Average`, sit on it. No significant edge has an interval that reaches the null. ## Robustness to case loss `stability_lsa()` repeatedly drops a share of the students, refits, and records the fraction of replicates in which each edge is significant (`stability`). By default it keeps 80% of the students per replicate and flags an edge as stable when it is significant in at least 95% of replicates. The call below runs 200 replicates. ```{r stability} stab <- stability_lsa(fit, R = 200) stab transitions(stab) ``` None of the 200 replicates failed, and all seven significant edges remain significant in every replicate (stability 1.000), so the significance pattern does not depend on particular students. Of the two non-significant edges, `Average -> Disengaged` is significant in 0.5% of replicates and `Disengaged -> Average` in 25.5%: the latter has a residual of -1.7, close to the threshold, and crosses it in some subsamples. Its non-significance is therefore a borderline result rather than a firm absence of effect. ## Reproducibility of the whole network `reliability_lsa()` raises the claim from one edge to the whole network. It splits the students at random into two halves, fits a model to each, and correlates the two vectors of edge weights (transition probabilities by default); the distribution of this split-half correlation over replicates measures how reproducible the network is within the sample (Epskamp, Borsboom, & Fried, 2018). The call below draws 100 splits. ```{r reliability} rel <- reliability_lsa(fit, R = 100) rel as.data.frame(rel, what = "summary") ``` The mean split-half correlation of the transition probabilities is 0.966 (sd 0.034), with 95% of replicates between 0.866 and 0.998: two independent halves of the students produce nearly the same network. The printout and the summary table also report failed replicates, splits in which a half could not be fitted (for example because a state is absent from it); here the count is 0 of 100, so the correlation rests on every split drawn. A non-zero count is reported rather than silently dropped and is read before the correlation. ## An empirical null `permute_lsa()` builds the null of no sequential structure by shuffling the order of states within each student and refitting. Shuffling within a sequence keeps every student's mix of states, so the null asks whether a transition is more or less frequent than that mix implies, and the p-value does not depend on the residual's large-sample approximation. The call below runs 500 permutations, so the smallest attainable p-value is 1/501. ```{r permute} perm <- permute_lsa(fit, R = 500) perm transitions(perm) ``` Seven edges are significant under the permutation null, as under the residual test, but not the same seven. Five edges reach p = 0.00200, the smallest value 500 permutations can produce. `Disengaged -> Active`, with a residual of -10.5, has a permutation p-value of 0.078 and is not significant. Shuffling within each student keeps that student's mix of states, so a student who is rarely both disengaged and active produces few such moves under the null as well; the avoidance is explained by which students are disengaged, not by the order of their weeks. The reverse holds for `Disengaged -> Average`: its residual (-1.7) is not significant, but its permutation p-value is 0.022, so disengaged students move to `Average` less often than their own mix of states implies. `Disengaged -> Active` is also the edge whose probability `certainty_lsa()` did not class as stable, so the claim that disengaged students avoid becoming active rests on the residual alone and is not reported as an established sequential effect. ## When to use which - **A descriptive reading of a transition** needs only the fit: `transitions(fit)`. - **How precisely a transition is known:** `certainty_lsa()` when sequences are homogeneous and a fast analytic answer is enough, `bootstrap_lsa()` when sequences may differ. - **Whether a significant transition depends on particular cases:** `stability_lsa()`. - **Whether the whole network reproduces:** `reliability_lsa()`. - **Whether a transition exceeds what each sequence's own mix of states implies:** `permute_lsa()`. - **Whether two groups differ:** `compare_lsa()` and `bayes_compare_lsa()`, in [Comparing groups](https://pak.dynasite.org/lagdynamics/articles/group-comparison.html). A transition that is significant, precisely estimated, stable under case loss, and beyond the permutation null is the kind to report as a finding. ## References Efron, B. (1979). Bootstrap methods: Another look at the jackknife. *Annals of Statistics, 7*(1), 1-26. Epskamp, S., Borsboom, D., & Fried, E. I. (2018). Estimating psychological networks and their accuracy: A tutorial paper. *Behavior Research Methods, 50*(1), 195-212. ```{r cleanup, include = FALSE} options(old_options) ```