The changelog of the R package spiDE, from NEWS.md.
spiDE (development)
- The vignettes are restructured into four articles with no overlap: “Getting started with spiDE” (
spiDE, the whole pipeline on the toy data), “The spiDE model” (spiDE-model, all the statistics, including the gene-set and niche-abundance models), “Designing a study with spiDE” (spiDE-design, new: patients and cells needed, minimum detectable effects, the small-design defaults) and “Validation: calibration and accuracy” (spiDE-calibration). They no longer link to the research site. pooled.df = "capped"is removed. The pooled test refers its t statistic to limma’s df scaled by the Kish effective share of the patients (the 0.99.37 default); the rule used before 0.99.37 is gone. Thepooled.dfargument oftestSpiDE(),spiDE()andspiGSEA()and theSpiDEResultsslotpooled.dfare removed, andshow()no longer prints the df rule. Results saved by earlier versions still load;spiGSEA()now always uses the proportional rule.spiDE()gainsname(default"Niche"), the niche reducedDim prefix, passed tobuildNiches()andfitSpiDE().testNicheAbundance()no longer tests an index type against a merged niche that contains it (mergeNiches()), asfitSpiDE()does not, and reports theindexandnichelabels as spelled in the data ("T cell", not"T.cell").- Internal:
buildNiches()no longer copies the object once per sample;fitSpiDE()builds one base list per index type (the order of the elements offit@index[[k]]changes, their values do not); shared helpers for the Cochran and Kish arithmetic, the sparse group sums, the per-patient long tables and the condition inputs; small dead code removed;MASSandwithrdropped from Suggests.
spiDE 0.99.37
New plot colours (“option V”). Slope direction is gold (up) and violet (down), also the diverging t scale; the two conditions are dark violet and pink; magnitudes use viridis “magma” (light to dark on the page, written out as hex, no new dependency); the nuisance colours stay Okabe-Ito. The palette was checked under simulated deuteranopia, protanopia and tritanopia.
plotNicheMap()draws its panel in the sticker’s ink (#140B2E) with dim non-focal cells (#3B3158), a white scale bar, and a brighter dark-to-bright magma ramp so the lowest densities stay visible; legend, title and axes keep the light page’s style.spiDEColours()has a new"spatial"role for that ramp; the other role names are unchanged, but their colours are new.plotPatientSlopes(): the condition strip is horizontal and wraps, so a long label such as “Non-responder” is no longer clipped.
spiGSEA()orders its rows by p, breaking ties (to 12 digits) by index, niche and set name. Complementary gene sets have exactly opposite statistics and a tied p, and their order had depended on floating-point noise, so it differed between platforms (the 0.99.37 test of the pooled df failed on macOS and R-devel for this reason).Two new defaults for small designs (5-7 patients per condition). Each was confirmed on untouched real-data nulls, against criteria written before those nulls were scored; the calibration vignette has the numbers and how each was chosen.
- The condition test’s family: pooled or heterogeneity (
testSpiDE(procedure = "pooled_or_heterogeneity"), the slopes engine’s default). The condition test now runs on the triplets that pass the pooled test or the new heterogeneity filter (procedure = "heterogeneity"on its own).- The heterogeneity filter keeps the triplets whose slopes vary between patients more than their sampling variances allow: Cochran’s Q across patients (the statistic behind the between-patient variance), with Benjamini-Hochberg over every triplet at
fdr. - Like the pooled filter it never looks at the condition.
- A condition effect is between-patient variation, so this filter keeps effects whose two conditions cancel in the pooled slope.
- The pooled filter alone missed them. With few patients it passed almost nothing for the condition test to see.
- Keeping the pooled filter’s triplets as well means that every triplet the previous default tested is still tested (
research/condtest/README.md).
- The heterogeneity filter keeps the triplets whose slopes vary between patients more than their sampling variances allow: Cochran’s Q across patients (the statistic behind the between-patient variance), with Benjamini-Hochberg over every triplet at
- The pooled test’s df: proportional (
testSpiDE(pooled.df = "proportional")). The moderated t is referred to limma’s df scaled by the Kish effective share of the patients. The previous rule took the smaller of limma’s df and the Kish effective patients less one. That always bound, dropped the prior’s df and left the pooled test conservative with few patients (research/smalldf/README.md). - The previous behaviour is
procedure = "filtered", pooled.df = "capped". The sandwich engine keepsprocedure = "filtered"; it refuses the heterogeneity filter, which needs per-patient slopes. Its tests do not usepooled.df.
- The condition test’s family: pooled or heterogeneity (
The results table has a new column,
p.heterogeneity(the slopes engine’s label-free heterogeneity test;NAfor the sandwich engine). One input check for the procedure and the engine, shared bytestSpiDE()andspiDE()(checkProcedure());spiDE()makes it before fitting.SpiDEResultshas a new slot,pooled.df. Results saved by earlier versions lack the slot and are read as"capped", the rule they were tested with.spiGSEA(pooled.df = NULL)follows the results it is given. Its condition test has no filter and is unchanged.spiDE()passesprocedureandpooled.dfthrough totestSpiDE().
spiDE 0.99.36
- Gene sets for the between-patient test.
testNicheAbundance(genesets = )tests sets instead of genes.- The score. Each gene is standardised across patients. A set’s score in a patient is the mean of its genes’ standardised pseudobulk expression.
type = "competitive", the default, subtracts the mean over the other genes;"self-contained"subtracts nothing. - The test. The scores go through the same regression, leverage cap and per-term multiple testing as genes.
- Coverage. Gene sets are now available for all four tests:
spiGSEA()for the within-patient slopes,testNicheAbundance()for the between-patient associations. - Not yet checked against permutation nulls.
- One gene-set input check, shared by both functions (
checkGenesets()).
spiDE 0.99.35
testNicheAbundance(): three fixes after permutation nulls. Permuting the condition across patients, the old test made condition:niche calls in nearly every permutation, as many as on the real labels.- One multiple-testing family per term.
q.globalwas a single Benjamini-Hochberg adjustment over both terms. A strong"niche"signal then loosened the threshold for the"condition:niche"rows. It is now computed over every row of the same term. - Centred condition coding. With a condition,
"niche"is now the association averaged over the two conditions; it was the first level’s."condition:niche"is unchanged: the difference, second level minus first. - A leverage cap (
max.leverage = 3, the weakest cap that keeps the condition term’s permutation null within its limit on all three public cohorts; see the calibration vignette). With a few dozen patients, one patient at the edge of a niche type’s abundance could carry a slope. A gene unusual in that patient then got a p-value far smaller than the moderated t allows. Patients whose leverage in the design exceedsmax.leveragetimes the average are now down-weighted until none does. The weights come from the design only, never from expression. New columns:leverageanddownweighted. - Covariates. A covariate constant over an index type’s patients is dropped from that design, where it stopped the fit before.
- Not calibrated everywhere. In designs with only a few cells per patient and index type, the condition term still makes false calls at the default cap (the calibration vignette, “The between-patient test”).
- One multiple-testing family per term.
spiDE 0.99.34
- Plots. Nine ggplot2 functions, one per question, in one house style (
theme_spiDE()) and one colour-blind-safe palette (spiDEColours()):plotNicheMap()(the niche covariate in tissue),plotPatientSlopes()(each patient’s slope: the evidence the tests weigh),plotNicheResponse()(the cells behind a slope, adjusted for each patient’s level or raw),plotCallMap()(calls per cell-type pair, heatmap or graph),plotTopTriplets()(a forest of the strongest triplets or chosen genes),plotSpillover()(whether the calls are the niche type’s own markers),plotTripletHeatmap()(genes’ or gene sets’ profiles across pairs),plotPValues()(p-values by expression band) andplotPatientEffects()(what the patient intercepts, or slopes, capture). Each returns one ggplot with its data inp$data; any two combine with patchwork. - The fit keeps each patient’s intercept.
fitSpiDE()now stores, for both engines, the patients’ intercepts from the shared, condition-free fit, their mean log1p niche densities and mean log library size;patientIntercepts()returns them. The slopes engine’s fit still never sees the condition. Fits from 0.99.33 and earlier work in every plot exceptplotNicheResponse(adjust = TRUE)andplotPatientEffects(), which ask for a refit. - ggplot2 and scales are now imported; ggrepel and patchwork are suggested.
- Documentation: which effects spiDE tests. The model vignette has a new section, “What kinds of effects can spiDE find?”, with a figure of mock expression trends: a slope within patients, shared or condition-specific (
testSpiDE()); an association between patients, shared or condition-specific (testNicheAbundance()); and the two effects of a condition that spiDE does not test, a shift in a gene’s overall level and a change in composition. The quick start points to it. ?testNicheAbundancecorrected. With a condition, the term"niche"is the association within the first level of the condition (the design is treatment-coded), not across both conditions as the help page said; run the test without a condition for the association across all patients. The function itself is unchanged.
spiDE 0.99.33
- Slopes engine: a niche type missing from a patient no longer drops the patient.
- The bug. A patient’s slopes were estimated jointly over all the index type’s niche columns. A niche type absent from the patient’s tissue (or of constant density across its cells) leaves a zero column, and the rank check then discarded the patient for every niche, not only the empty one.
- Who it hit. Designs with rare niche types. On the public cohorts a single patient (GSE289194). With many fine cell types almost every patient lacks one, and nearly all slopes were lost.
- The fix. A constant column is dropped for that patient only, which gives no slope for that niche. The patient’s other slopes are those it would have had with every niche present, because a zero column carries no information about the rest. The rank check now runs on the correlation scale of the patient’s information matrix, so it judges collinearity alone. A niche column of small spread (a sparse niche’s tail) no longer drops the patient. Nor do columns on very different scales. A patient that passed the check before gets the same slopes.
- An existing fit must be refitted: the slopes are computed in
fitSpiDE().
- Documentation: how much data spiDE needs.
- What. A power benchmark measured the pooled and condition-specific tests across the number of patients, the cells per patient, the effect size, the kind of gene, the tissue architecture and the panel. Counts were simulated from four real tissues.
- Where. Two reports on the research site, linked from the Validation menu: the results, with an interactive study planner, and the simulation design. The calibration vignette has a new section, “How much data is needed”, with the headline numbers.
spiDE 0.99.32
- Slopes engine: a new condition test.
- What it is. The condition x niche contrast is now weighted least squares across patients, with the pooled test’s weights
1 / (v_model * factor + tau2), an HC2 standard error and Bell-McCaffrey df. - Why the weights changed. Through 0.99.31 the test was limma weighted by each gene’s own tile variance
v_tile.v_tileis computed at the shared, condition-free fit, so it grows with a patient’s departure from the pooled slope. As a weight it gave the patients furthest from that slope the least say, which pulled each condition’s mean toward it and attenuated the contrast (research/bench2/diag/FINDINGS.md). - Why the SE changed. With the gene-shared weights, limma’s model-based SE let spatially patterned marker genes through a real-data block null. The robust SE does not (
research/release/README.md). - What is unchanged. The fit and the pooled test: an existing fit needs only
testSpiDE()again. - The calibration vignette has the re-measured nulls.
- What it is. The condition x niche contrast is now weighted least squares across patients, with the pooled test’s weights
- Strata. A condition fully confounded with
stratais now refused by the slopes engine. Before, the strata column was silently dropped and the condition tested unadjusted. spiGSEA()rewritten.- The method. A gene set is now tested like a gene. Each patient gets a set slope: the mean of its genes’ slopes, each scaled by the gene’s between-patient spread, less the other genes’ mean (
type = "competitive", the default) or not ("self-contained"). The set slopes go through the pooled and condition-specific tests. - Correlation between genes needs no estimate: the patient is the unit.
- Calibration. The gene-set null is calibrated on the five cohorts (the calibration vignette), so the function is no longer experimental.
- Output. Adds
q.global,se,dfandn_patients; dropsrho. - Engine. Slopes engine only.
- The method. A gene set is now tested like a gene. Each patient gets a set slope: the mean of its genes’ slopes, each scaled by the gene’s between-patient spread, less the other genes’ mean (
testSpiDE()andspiGSEA()takeBPPARAMfor the condition-specific test, andspiDE()passes its own on.- Vignettes rewritten for users. The model is explained step by step, plainly and with the formulae; the calibration vignette gives the checks on real and simulated data.
spiDE 0.99.31
- Sandwich engine:
stratano longer enter the pooled model. In 0.99.30fitSpiDE(engine = "sandwich", strata = )put the treatment-coded strata x niche columns into the pooled model as well as the condition model, so the pooled slope was the slope of the first stratum’s patients rather than the average over all patients, with the df of that stratum alone. The condition test was unaffected, but its filtered family was drawn from the wrong pooled test. Found on the YTMA LUAD stage cohort, the only cohort run with strata (research/release/), where the sandwich engine’s pooled test called a small fraction of what the slopes engine’s did. The slopes engine never used strata in its fit and is unchanged; so is every run without strata. A test now guards this.
spiDE 0.99.30
A new model; the mixed model is archived
spiDE 0.99.30 replaces the joint mixed-effects model with per-patient models, and the API changes with it. Nothing below is backward compatible.
- Why. On five comparisons in four spatial cohorts the mixed model of 0.99.22 made false calls on every null grid: it had no between-patient variance in the niche slopes, so its standard errors shrank with the number of cells. The simplification study (
research/simplify/, 2026-09-28/29) measured the replacement engines; the calibration vignette quotes the numbers. fitSpiDE()now fits, for every index cell type, one negative binomial GLM per gene with an intercept per patient (SpaNorm::polishNB(), patient intercepts absorbed), so every niche coefficient is a within-patient slope.engine = "slopes"(the default) then estimates each patient’s own slopes with a within-patient spatial sandwich variance;engine = "sandwich"fits the condition-specific model and uses a patient-clustered CR2 sandwich with Bell-McCaffrey degrees of freedom. One bandwidth per fit.- Depth.
depth = "loglib"(default, one slope per gene on the centred log library size),"nonlinear"(a 3-df natural spline of it),"spatial_spline"(a library-size spline within each patient: per section, log library size times a smooth tensor spline of position, absorbed with the patient’s intercept through SpaNorm’s compactnbBlockDesign()(requires SpaNorm >= 1.7.15), as in SpaNorm’s library-size term) or"none", andoffset =an assay of per-gene log offsets (e.g. SpaNorm’s library-size component). A pre-registered benchmark on simulations fitted to real cohorts (research/bench2/) found every option calibrated and their biases within its tie margin, so"loglib", the simplest, stays the default; the calibration vignette has the numbers. - Fit status. A gene the solver cannot fit, or whose coefficients run off (|coefficient| > 20 on the log scale), is left out rather than tested; a gene whose dispersion optimum is on its search bound takes that bound, with its mean refitted there. Both are counted per index type in the fit (
status). testSpiDE()always runs the pooled test (is the niche slope non-zero across patients?) and, given acondition, the condition-specific test (does it differ between conditions?). By default (procedure = "filtered") the condition-specific test is corrected only among triplets whose pooled test passes; the filter never looks at the condition. The slopes engine combines patients with weightedlimma, an expression-trended prior and an effective-patient df.spiDE()chainsbuildNiches(),fitSpiDE()andtestSpiDE().results()returns one table per test (test = "pooled","condition"or"both") with columnsgene,index,niche,test,estimate,se,t,df,p,n_patients,q,in_family.patientSlopes()(new) returns every patient’s slopes.compositionTest()is renamedtestNicheAbundance(), and its columns now matchresults()(index,niche,estimate,n_patients,q,q.global). It asks a between-patient question the within-patient models absorb by design.spiGSEA()is reimplemented on the new results and is experimental: no gene-set null has been run.- Removed:
polishSpiDE(),nicheDesign(),computeSizeFactors(),fits(),bandwidths(), the argumentsrandom,df.method,re.celltype,dispersion,combine,polish,backendand the GPU path, multi-bandwidth fits and the cross-bandwidth combination, and the hierarchical FDR cascade. - The mixed model is archived, with its tests, long tests, fixtures, vignettes, design documents and this file’s history, as the research package
spiDEmixed(research/mixed), numerically identical to 0.99.22. Objects saved by spiDE 0.99.22 or earlier are refused by this version with a pointer tospiDEmixed::readSpiDE(), which reads them. Research code that calls the old functions calls them asspiDEmixed::or loads the pinned libraryresearch/libs/mixed-final. toySpiDEnow has 16 patients of 150 cells (6 of 80 before): the tests take patients as their units.- Vignettes: a new quick start, a model vignette and a calibration vignette. The mixed model’s validation reports are no longer on the package site; they are archived on the research site.
spiDE 0.99.22
Changes
- The per-gene polish engine moved to SpaNorm.
polishSpiDE()’s damped Newton solver, sane restart and profile dispersion are nowSpaNorm::polishNB()andSpaNorm::nbProfilePsi()(requires SpaNorm >= 1.7.13); spiDE keeps what is its own – which columns the solver absorbs and starts from (.absorbSpec()/.absorbBatchSpec(),.testedStartCols()), the ridge penalty, and the variance-component (tau2) loop around the polish. Outputs are unchanged:longtests/testthat/test-polish-golden.Rpins the golden fixture to tolerance 0 before and after the rewire.
spiDE 0.99.21
Changes
- The quasi-likelihood moderation no longer loses a stage to limma’s robust fit.
limma::squeezeVar(robust = TRUE)estimates an outlierdf2through the F tail, and with unequaldf1an extreme(s2, df)pair makes that tailNaN, after whichfitFDistUnequalDF1()stops on an emptywhich.min(). Measured on the restricted cohort design at bandwidth 70 (2026-09-14), where it killed the test stage after the fit and the polish had already run. The per-gene sites ininference.Rdrop the offending gene; this call is across genes, so dropping is not available and the treatment is to fall back to the non-robust moderation – ordinary empirical Bayes, which is whatrobust = FALSEhas always done – with a warning naming the cause, rather than lose the stage.
spiDE 0.99.20
Changes
- The polish stage costs one cold pass, not four. On the 0.99.19 cohort runs (769 genes, four workers) the stage took 620-700 min: a cold pass of 245-395 min and three re-polish passes of 110-168 min each, and four of fifteen tasks hit their 12 h limit inside the fourth pass. Three causes, each measured, each fixed:
- Forked polish workers now run BLAS single-threaded. A forked worker inherits the parent’s OpenBLAS thread count, so four workers on eight cores ran 32 threads. At the cohort’s design shape, four workers with four threads each on four cores take 2.3-2.6 s per Newton step against 0.24-0.28 s with one thread each (9x), while a single worker gains only 1.5x from four threads. Both blocked stages, the polish and the inference (whose per-gene gram is the same BLAS work), now dispatch through a wrapper that sets one BLAS thread inside each worker when RhpcBLASctl (now in Suggests) is installed; the parent process is left alone.
- The re-polish after a variance-component step is warm. It starts from a converged fit at a nearby penalty, so it is a few damped Newton steps per gene at the held dispersion: no profile-psi search, and no log-mean restart check, which had thrown about 100 of 769 genes back to the sane start on every re-polish pass at bandwidth 10 (3-5 in the cold pass). It agrees with a cold re-polish to 1e-3 on the coefficients; on the clustered fixture the variance components agree to four digits and the t-statistics to 0.01, and re-profiling the held dispersion at the end of the loop would move them by at most 0.014.
- The variance-component loop converges instead of stopping at a cap. Schall’s update is a linearly convergent fixed-point map, and with the cap of three the (sample x cell type) component was still 6-12% above its extrapolated limit in thirteen of fifteen cohort runs (the per-sample one is within 1% after two steps). That matters: a tenfold error in a near-zero nested component moves individual t-statistics by up to 0.13. The loop is now Steffensen-accelerated (Aitken extrapolation after every two plain steps, per component, only while the steps contract), with
tau2.maxit = 10,tau2.tol = 1e-2onlog(tau2)andtau2.accelerate = TRUE(FALSEis the plain map), and it ends with one more warm pass at the reported penalty so coefficients, penalty and df are consistent.
@polishgainsrepolish.iterations, the Newton steps spent in the re-polish passes,repolish.cappedandrepolish.singular(a warm pass that hit its cap or a singular system, which leaves the gene at its previous converged fit), andpsi_fitnbisfitNB’s dispersion again: the 0.99.19 loop handed each re-polish the previous pass’s polished value, so the “polished / fitNB” ratio the cohort driver reports read about 1..toyClustered(sd_nested = )plants a (sample x cell type) intercept, so a fixture can have an interior nested variance component.- From the review of the loop: its tolerance is an estimated distance to the fixed point rather than a step size (a step-size rule declares convergence ten times early on a slowly contracting map); `
carries the loop'siterations,convergedand per-steptraceas the attribute“tau2”; under the profile rule the stage ends with one profile pass at the reported coefficients; a singular final information degrades the reference df to the between-sample scalar, as the fit does; the cold pass no longer restarts a gene whose only fitted log-means below -10 are at zero-count cells; andresults()keeps the triplet p-value (p.niche`) beside its FDR columns. RhpcBLASctl moved to Imports so the single-threaded workers do not depend on an optional package. - The polish’s two dispersion rules are documented with their cost (
polishSpiDE(psi = "moderated")skips the dispersion search and the two re-polishes it triggers) and benchmarked from one fit polished both ways: the harness times the polish on its own and the timing table carries one row per arm and size (research/config.R, armpolish-rules). Measured: the rules are equivalent on null and power, the moderated rule saves 13-19% of the polish on the benchmark and half on the cohort, and classic BH over all triplet tests does not reclaim the cascade’s conservatism (the cascade is slightly the more powerful at every level); seevignette("spiDE-calibration").
spiDE 0.99.19
Changes
- The pipeline is fit -> polish -> test -> gsea, and the polish is a stage.
fitSpiDE()fits the shared model and nothing more:converge,converge.maxit,converge.tolandpolish.psiare gone from it.polishSpiDE()is the stage the user runs, or skips, according to their data;spiDE()runs it by default (polish = TRUE) andpolish = FALSEtests the shared fit asfitNBreturned it. - The variance components are re-estimated from the converged fit. The fit’s Schall loop reads the shared fit’s own coefficients and dispersion, which can sit far from every gene’s optimum: on the clustered test fixture it reported a between-sample variance of 10 against a planted 0.49 (from sample intercepts three times too wide), under every release since 2026-08-08.
polishSpiDE(tau2 = TRUE)(the default) takes one Schall step on the polished coefficients with the gene-averaged weights at the polished mean and dispersion, re-polishes at the new penalty, iterates totau2.tol, and refreshes the Satterthwaite reference df. The fixture’s component lands in the planted window. Calibration had survived the old value because the QL scale is robust to the dispersion and an inflated component only weakens the ridge, but the reference df, the shrinkage and the Newton weights all read it. - The polish’s dispersion rule is the profile value again (
polishSpiDE(psi = "profile")). The moderated rule, made the default in 0.99.18 for a third of the cost, keeps whatever the shared fit left, which on the same fixture is fifteen times the converged value; the variance-component step needs a dispersion consistent with the converged mean. The two rules remain indistinguishable on the null and in the real cohort’s calls, and the cost argument has weakened now that the GPU carries the cohort’s real grid in a third of the CPU time. - The two-stage estimator has left the package.
twoStageSpiDE()and its stages are archived as a standalone research package,spiDEtwostage(research/twostage/in the research repository), which returns its own result object;SpiDEResultsloses thediagnosticsslot and the validity relaxation that allowed an emptyfits. The mixed-effects estimator is the recommended approach: it fits all cells jointly, so thin cell types borrow strength, and on the real cohort it was the better-calibrated of the two. The two-stage benchmark stays on the research site as the record of that comparison.
spiDE 0.99.18
Changes
- The standard errors are scaled by the quasi-likelihood dispersion.
testSpiDE(dispersion = "ql")is the default: each gene’s edgeR-v4 quasi-likelihood dispersion (its adjusted NB deviance over its effective residual df,SpaNorm::qlDispersion()) moderated across genes withlimma::squeezeVar(), in place of the working Pearson dispersion, which remains available asdispersion = "pearson". Measured on the synthetic benchmark (40 replicates, intercept mode, Satterthwaite df) it is the only configuration that holds the nominal type-I error at every sample size: 0.050–0.055 at S = 4, 10, 16 and 30 against 0.091, 0.076, 0.071 and 0.068 for the Pearson scale. Its recall at FDR 0.05 is that of the design with neither 0.99.17 switch (0.360 against 0.361 at S = 30, and 0.402 for the Pearson scale, whose extra recall is bought with its inflated null) and its realised FDP sits far below nominal (0.014 at S = 30, 0.000 at S = 4 where the Pearson scale reaches 0.59). On the real cohort it calls the same triplets as before (111 of 114 shared) at no extra cost. It runs on either backend: the deviance moments are one shared (log mu, log phi) table per gene block and the rest is elementwise over genes x cells on the device. A fixed-effects, unconverged fit (random = "none", converge = FALSE) has no such scale and keeps its legacy NB-dispersion scale with a message. - The convergence stage keeps the moderated dispersion.
fitSpiDE(polish.psi = "moderated")is the default: the coefficients are converged per gene underfitNB’s cross-gene moderated dispersion, and the per-gene profile-ML re-estimate is available aspolish.psi = "profile". The two are indistinguishable on the null at every sample size (to the third decimal) and call the same triplets on the real cohort, and the moderated one runs the cohort’s real grid in two thirds of the wall time (178 against 266 minutes), because the profile step was the expensive part of the stage.polishSpiDE()takes the same argument with the same default. - The quasi-likelihood machinery now lives in SpaNorm (>= 1.7.10) as
nbUnitDeviance(),nbDevianceMoments()andqlDispersion(), with both backends and the oracle tests againstedgeR::glmQLFit(); spiDE carries only the wiring. The 2026-08-31 refutation of the QL dispersion is withdrawn: it was scored on the composition bias that the nested intercept removes, which no per-gene scale could fix.
spiDE 0.99.17
New Features
re.celltype(defaultTRUE) adds a nested (sample x cell type) random intercept to the niche design alongside the per-sample one, so that every tested niche slope is a within-(sample, cell type) slope. Without it the slopes also carry the between-sample composition effect – a patient-level association between a cell type’s mean niche density and its mean expression in that type – which a shuffle null that permutes within (sample, cell type) preserves exactly. On the YTMA cohort that confound was the cause of the triplet-level FDR failure: with the nested block the shuffle null is calibrated in every expression band (sd(null t)0.96-0.99, no expression gradient, no|t| > 4.89exceedances against 95 before) where before it ran to 1.42 for the brightest genes.re.celltype = FALSEreproduces pre-correction fits.random = "slope"is not a substitute: per-sample slopes on theCellType:nichebases leave the group means untouched.converge(defaultTRUE) converges each gene to its own penalised negative-binomial optimum afterSpaNorm::fitNBreturns, and re-estimates its dispersion by profile maximum likelihood at the converged mean.fitNB’s multi-gene IRLS shares one cell weight vector across genes and stops on the aggregate log-likelihood, which leaves bright, cell-type-restricted genes one to four standard errors short of their own optimum with a dispersion about 1.6 times too large. The stage is per-gene, so it is blocked and parallelised overblock.size/BPPARAMas inference is, with the nested indicator block absorbed by a Schur complement so the per-iteration cost does not grow with the number of groups. NewSpiDEFit@polishrecords the per-gene diagnostics. Note this replaces edgeR’s cross-gene moderated dispersion with a per-gene one, and that it raises the null variance of bright genes – their previous standard errors were inflated by a dispersion estimated off the optimum.It also sharpens real signal substantially, which was not the reason for building it. On the toy fixture’s planted niche effect the test statistic goes from 1.68 (FDR 0.09, not called) to 10.19 (FDR 2e-15) in condition mode, because the unconverged fit put the planted gene’s dispersion at 3.09 where its own optimum is 0.28 and the inflated standard error buried the effect. In niche mode a SPURIOUS competing association that outranked the true one (|t| 7.82 against 5.63) disappears, and the true one goes to 14.27. Two unit tests had encoded those artefacts and were updated.
Changes
nicheDesign()gainsre.celltype, defaulting toFALSE– the same deliberate asymmetry asrandom, since a design returned with penalty-identified columns is rank-deficient..toySPE()gainscomposition, which plants a between-sample composition confound with zero within-sample niche slope.polishSpiDE(): the per-gene convergence stage as a post-hoc adjustment on an existing fit — a fit made withconverge = FALSE, one serialised by an older version, or one to re-converge with other settings — without refitting. It shares one implementation and one penalty rule withfitSpiDE(converge = TRUE), so the two routes give the same fit (tested to 1e-8), and it clears every inference slot derived from the old coefficients sotestSpiDE()recomputes them.compositionTest(): the between-sample composition association that the nested intercept absorbs, tested on its own terms at the patient level — pseudobulk per (sample, index type), the sample’s mean niche density around those cells, alimmamoderated t across samples, with"niche"(pooled) and"condition:niche"(the patient-level counterpart of the three-way term) reported per (gene, index, niche). It is real signal, and it is not neighbourhood-dependent DE; the two now have two tests..toySPE(composition = )now plants a confound that reaches the tested slope: each sample’s A cells are shifted toward or away from the B-rich region, so samples differ in mean niche density the way tissue does, and G2’s baseline in Responders’ A cells shifts with that mean. The earlier version tied B-cell prevalence to condition, which moved the between-sample mean too little to matter next to the within-sample spread.
Documentation
- The vignettes are reorganised into four, written top-down rather than in the order the work happened: the quickstart (calls and result layers, no justification), The spiDE model (the model stated once, organised around the Frisch–Waugh–Lovell theorem, with the per-gene convergence — damped Newton, profile-ML dispersion, the Schur-complement absorption of the nested block — and the combination and FDR steps explained for the first time), The two-stage estimator (split out), and Calibration: reading lambda, and what the benchmarks measured, now the only place in the package that quotes benchmark numbers. Research history has been removed from all four.
Bug Fixes
Non-integer counts are refused when
converge = TRUE.dnbinom()is-Infoff the integers, so the convergence stage rejected every step, left the coefficients atfitNB’s values and returned the dispersion optimiser’s upper bound for every gene (measured 999.96) while its own diagnostics reported success.checkCounts()now refuses such an assay before the fit.The convergence stage no longer densifies the whole counts matrix. Its automatic block size only applied with more than one worker, so under the default
SerialParam()a single block realised every gene at once (8.3 GB on a real cohort), breaking the package’s never-densify invariant.A polished fit is scaled by the Pearson working dispersion on both paths. The fixed-effects path scaled its standard errors by
psi, which is safe only whilepsiis edgeR’s moderated value; after convergence it is not. Measured on the toy fixture, converging tooksd(t)from 2.11 to 2.45 before this fix and to 1.92 after it.Fit and inference now use one mean function. The convergence stage maximises the unclamped likelihood, so inference evaluates it unclamped rather than at
calculateMu()’s winsorised mean; the planted toy effect’s statistic goes 3.98 to 5.78. Withconverge = TRUE,winsortherefore sets only the Newton starting point.The Satterthwaite fallback no longer aborts the fit. Its documented degrade-to-
"between"path returned a bare number from.fitNBmixed(), so the caller died with$ operator is invalid for atomic vectors.A dispersion optimum sitting on its search bound, and a gene whose Newton cannot run, both now keep
fitNB’s estimate and are flagged in@polish(psi_bound,polished) rather than storing a bound or a zeroed fit.@polishis keyed by gene rather than by position, and a wrongly sizedlambda.ais refused with a message namingre.celltype.An underflowed fitted mean no longer becomes a
NaNstatistic. An unwinsorised linear predictor can sendexp()to exactly 0 for a few cells, and the Pearson working dispersion is then0/0; it propagated to the standard error, the statistic andspiGSEA(), which failed outright. The fitted mean is floored atexp(-30)in the convergence stage and at inference, at the same value.A covariate with missing or non-finite values is refused by name.
model.matrix()drops those rows, so the design no longer matched the random-effect block and the run died incbind()withnumber of rows of matrices must match.log()of a QC column that is zero for some cells is the usual way in.Inference absorbs the nested indicator block. The (sample x cell type) columns are a 0/1 partition of the cells, so the per-gene covariance comes from a Schur complement over the dense columns alone rather than a gram over the full design. Measured at the real cohort shape (77,454 cells, 345 dense + 660 nested columns): 2.16 s -> 0.35 s per gene, or 8.0 h -> 1.3 h for a 13,348-gene transcriptome, agreeing with the dense path to 7e-21. The saving is size-dependent – at a smaller shape the two are within 20% – so it is kept unconditional only because both are cheap there.
Examples now use a single bandwidth, and the fast fixed-effects path where their subject is an accessor rather than the mixed correction, taking
R CMD check’s example time from ~23 minutes to ~10.updateObject()could not repair an object serialised before a slot whose prototype isNULLwas added (polish, and latentlyre_group,tau2,penalty,df)..fillSlots()usedattr(object, s) <- value, andattr(x, "s") <- NULLremoves an attribute rather than setting it, so the slot stayed absent: the object remained invalid,show()errored, andupdateObject()– the documented repair path – failed on exactly the objects it exists to repair. It now assigns throughmethods::slot<-.The per-gene convergence stage ran as a single block regardless of
BPPARAM, so a caller requesting several workers got no parallelism. Absent an explicitblock.sizeit now splits one block per worker; blocking is exact, so the result is unchanged.
spiDE 0.99.14
New Features
twoStageSpiDE(): a second, structurally different estimator of condition-dependent niche effects that removes cell-level pseudo-replication by construction rather than by correction. Stage 1 estimates each patient’s niche slopes with a joint weighted fit per (sample, index cell type) – anchored on a storedSpaNormfit by default (stage1 = "spanorm"), with"ols"and"nb"alternatives – pools cores within patients by precision, and stage 2 contrasts the patient slopes with alimmamoderated t-test weighted by1/(v + tau2)(DerSimonian-Lairdtau2, pooled over genes). Returns the familiar tidy results schema plus three diagnostics (r2,inclusionwith an informative-dropout warning,tau2). In the paired benchmark it is the best-calibrated method in the study, trading power for that calibration; restrictindex/niche/genes to a pre-specified hypothesis. Documented in full, with equations, in the model vignette.
Changes
- The vignette suite is trimmed to the three user-facing documents (quickstart, model, calibration): the eight built vignettes totalled 10.85 MB of HTML, over Bioconductor’s 10 MB tarball cap on their own. The five validation studies and the benchmark tables they read moved to the research repository and are published at https://bhuvalab.github.io/spiDE-research/ – pkgdown’s “Statistical validation” menu links there.
- The shared palette’s two-stage colour is now CVD-safe (
#C51B7Dwith a triangle marker as a secondary channel; the previous#E7298Awas indistinguishable from the intercept teal under deuteranopia).
spiDE 0.99.12
New Features
updateObject()methods forSpiDEFitandSpiDEResults, filling any slots that did not exist when an object was serialised.Slots added to a class do not appear in objects pickled before them. Reading such an object still works, but anything that triggers validity fails – including
initialize(), which is the documented way to re-combine single-bandwidth fits across bandwidths:invalid class "SpiDEResults" object: slots in class definition but not in object: "results.celltype", "results.patient"This bit a real analysis: fits stored before the
CellType:conditionresult layers existed could no longer be combined, and those objects represent hours of cluster time.updateObject()fills absent slots from the class prototype and leaves every present slot untouched; it also descends into the containedSpiDEFitobjects. Verified on a four-bandwidth production result: validity goes from failing to passing with all fits and result rows preserved.
spiDE 0.99.11
Behaviour change
spiGSEA()now defaults totest = "competitive"(was"self-contained"). The simulation benchmark (research/, scenariogsea) measured both on ground truth and the self-contained form does not control error on correlated data: with nothing planted, at realistic inter-gene correlation and a nominal FDR of 0.05, it called 20.6 of 208 sets per replicate, every one of them false – a realised FDP of 1.00 – against 0.05 sets for the competitive test.The cause is not the inter-gene correlation term. Setting
rho = 0roughly quadruples the damage, so that term is doing real work, but the self-contained test stays badly anti-conservative with the correctrho. It assumes the gene-level statistics being averaged have unit spread, which holds under the null and fails under signal (measured spread 1.0 null, 1.8 at the largest effect tested). The competitive form divides by the observed spread and is immune.Two further results from the same benchmark. Under a global shift affecting most genes, the self-contained test called 96.3% of random sets against 0.7% for competitive. And because the per-gene statistics track expression, the self-contained test’s false calls concentrate in abundant sets: called sets sat at the 86th expression percentile against the 51st for random sets – closely reproducing what is seen on real data.
test = "self-contained"is retained for comparison with the flat-script pipeline it replaces, and is documented as unsuitable for inference.Power for the competitive test, at FDR 0.05, is usable from about 25 genes per set at moderate effects, and saturates by 50 genes at larger ones.
New Features
- A gene-set benchmark in
research/: scenariosgseaandgseacal, with gene-gene correlation induced at the rate observed in real spatial data and calibrated against the realised residual correlation rather than assumed.
spiDE 0.99.10
Improvements
testSpiDE()now stores that correlation on eachSpiDEFit(newrhoslot), computed as a by-product of the gene blocks inference already loads – so it costs no extra pass over the counts. Read it asfits(res)[[i]]$rho, like any other slot. It is a useful diagnostic in its own right: it says how much residual variation is shared across genes, and so how far a set ofmgenes falls short of carryingmgenes’ worth of independent evidence.spiGSEA()uses it by default and therefore needs no counts pass at all, which makes itsspeargument optional: a fitted result can be shared and queried against many gene-set collections without the counts matrix travelling with it.Fits serialised before this slot existed still load and still work; they fall back to estimating the correlation from the counts, exactly as they did when they were written.
Bug Fixes
spiGSEA()errored on a single gene set. The per-bandwidthvapply()returned a length-k vector rather than a 1 x k matrix, which the Cauchy combination then transposed against the weight matrix. Testing one set is ordinary usage.
spiDE 0.99.9
New Features
spiGSEA()adds a gene-set layer over a fitted model. The per-gene niche tests are individually under-powered on sparse spatial data; averaging a statistic over the genes of a pathway recovers power, with the average inter-gene correlation carried explicitly so that co-regulation is not mistaken for evidence.Two nulls are available and they answer different questions.
test = "self-contained"(default) asks whether the set’s mean statistic differs from zero;test = "competitive"asks whether it differs from the genes outside the set, aslimma::cameradoes. The default is the more permissive of the two: under a global shift it will call most sets, correctly but uninformatively.Works on either result layer via
type = "niche"(the three-way celltype:condition:niche statistics) ortype = "celltype"(the CellType:condition statistics).Three details differ from the flat-script gene-set code this replaces, all of them corrections:
- statistics become z before averaging (as
cameradoes), because the set statistic assumes unit variance while a t with v df has variance v/(v-2) – immaterial for the niche coefficients, but not for the CellType:condition ones where v is the between-patient S-2; - bandwidths combine on two-sided p-values, so a set shifting up at one scale and down at another no longer cancels to nothing;
- the inter-gene correlation is estimated per bandwidth rather than once, since each bandwidth is a different design and leaves different residuals.
The correlation is estimated without ever forming the gene x gene matrix (1.4 GB at 13,000 genes): standardised residuals are streamed one gene block at a time and accumulated through an identity that is exact, not an approximation, in memory linear in the number of cells. That pass carries the same
backend/BPPARAM/block.sizecontrols as the inference stage, and is verified to give identical answers across block sizes, worker counts and the CPU and GPU backends.- statistics become z before averaging (as
spiDE 0.99.8
New Features
The niche design now carries a
CellType:conditionblock, so a response that is cell-type-specific but niche-independent has a term of its own instead of being forced into the three-way niche interaction. Benchmarked over 2,840 simulated design points (research/): the previous niche-only design produces 6.8-9.5x more spurious niche calls on such genes. Calibration is unchanged (null type-I0.0450vs0.0449).This is a trade, not a free win. On truth that really is niche-only, power is lower (TPR
0.183vs0.319), because the new block absorbs part of the signal the three-way term used to carry alone.vignettes/spiDE-mixed-benchmark.Rmdreports both sides.Note the cell-means coding: there is no bare
conditionmain effect. The condition contrast is carried by theCellType:condition(ResponseCellType) columns, one per cell type. Code matching a single"Response"column finds nothing and should matchResponseCellType.results()gainstype = "celltype"andtype = "patient"alongside the default"niche": cell-type-specific response calls keyed by(gene, ct_index), and one abundance-weighted response contrast per gene.
Improvements
All three result layers now combine evidence across every bandwidth with the log-likelihood-weighted Cauchy combination. The cell-type and patient layers previously reported the last bandwidth’s fit alone. On a four-bandwidth production fit the cell-type layer returns 107 genes combined against 93 for the best single bandwidth, and 0 under the old behaviour – the widest bandwidth was the least informative of the four. If you compared these layers across analyses with different
sigmasets, those results change.Cauchy combination now consumes two-sided p-values. Under one-sided input
tan((0.5 - p)pi)diverges to-Infasp -> 1, so a gene up in one niche and down in another cancelled exactly. Brown’s method stays one-sided, where-2log(p)is bounded at 0 and safe.Consistent FDR scale across the three layers:
.dirBH()tested each direction atq/2but returned the per-directionqunscaled, understating cell-type and patient q-values roughly two-fold relative to the niche layer.The toy data generators no longer reseed the caller’s global RNG. They take a
seedargument for reproducibility but previously calledset.seed()directly, perturbing every random draw the caller made afterwards.
spiDE 0.99.7
Behaviour change:
fitSpiDE()/spiDE()now default todf.method = "satterthwaite"(previously"between"). Consequently a mixed-effects fit’sSpiDEFit@dfis a named per-tested-coefficient vector by default, where it used to be the scalarS - 2; code that assumed a scalar (e.g.stats::pt(t, fit@df)across genes) must now index the column it is testing, or passdf.method = "between"explicitly to restore the old behaviour. Nothing else about the fit changes — only the reference df used at inference time.The change follows the completed benchmark study (
research/), which measured both arms on identically seeded data."between"is severely over-conservative when samples are few — null type-I error~0.001atS = 4against a nominal0.05, with correspondingly near-zero power — and only approaches nominal at the largest sample sizes studied."satterthwaite"holds type-I error within0.042–0.065across the whole sampled range and gains~0.10mean TPR (paired, FDR 0.05), with observed FDP no worse. The trade is a mild liberal drift at largerS(worst measured~0.065against nominal0.05); a Kenward–Roger bias correction of the variance-parameter covariance is the indicated next step for closing it. Usedf.method = "between"where strict conservatism matters more than power, or for back-compatibility.
spiDE 0.99.6
fitSpiDE()/spiDE()gain adf.methodargument for the mixed-effects fit (random != "none"). The default"between"keeps the single between-patientS - 2reference df (back-compatible). The new"satterthwaite"computes a per-coefficient Satterthwaite reference df, so the response-niche interactions — which carry within-patient information and were over-conservative under a flatS - 2— get their larger effective df, while the response main effect stays at~ S - 2by construction.SpiDEFit@dfis accordingly a scalar under"between"or a named per-tested-coefficient vector under"satterthwaite". The df is invariant to the per-gene dispersion, so it is computed once per bandwidth and adds nothing to the per-gene inference loop. See the spiDE model and Speeding up the mixed-effects fit vignettes.
spiDE 0.99.5
mergeNiches()now records the merged niche group membership on the object, and the design matrix built byfitSpiDE()/nicheDesign()uses it to drop every covariate whose index cell type is a member of the merged niche it is tested against (previously only the exact index-equals-niche self interaction was dropped). This matches the neighbourhood self-density exclusion of the original analysis scripts.
spiDE 0.99.4
- Changed
fitSpiDE()’s defaultre.propfrom0.2to1(no cell subsampling in the random-effect variance-component loop). A replicate study on real data (vignettes/spiDE-mixed-benchmark.Rmd) found thatre.prop < 1doesn’t just add noise to the fittedtau2— the noise itself stays roughly flat fromre.prop = 0.2to0.8(never shrinking below the genuine between-patient signal it’s confounded with), and the meantau2is systematically biased downward at everyre.prop < 1tested, an attenuation that more replicates cannot average away. GPU backends makere.prop = 1affordable in practice (see the benchmark vignette), so it is now the default; lowering it remains possible but is rarely advised (see?fitSpiDE). - Bumped the
SpaNormrequirement to>= 1.7.7, which fixesgetGPUMemoryBudget()reporting a MIG-partitioned GPU’s whole physical card instead of the process’s assigned slice — spiDE’s GPU inference path (R/inference.R) calls this directly, so on MIG hardware the bug could feed a many-fold-too-large budget into.covBatchSize()and cause an out-of-memory failure instead of the intended blocked, bounded-memory fit. - Fixed a memory blowup that made
combine = "cauchy"(the default) unusable with random effects on realistically-sized data, on both backends. The batched Wald covariance introduced in 0.99.3 precomputed a Khatri-Rao cross term of the design (ncells x p^2) once per bandwidth. Being built outside the gene loop, its size could not be bounded byblock.size, and it scales quadratically in the design width: a 602-column random-intercept design over 21,843 cells needs 63 GB, and a 4,906-column random-slope design 4.2 TB. The per-gene Gram matrices are now built with a batched matmul over a bounded sub-batch of genes instead — memory is linear inpand capped automatically (options(spiDE.cov.mem.budget = <bytes>)on the CPU path, the GPU budget otherwise), independently ofblock.size. Results are unchanged (bit-identical on the CPU path) and invariant to the sub-batch size.
spiDE 0.99.3
testSpiDE()/spiDE()gain abackendargument ("auto","cpu","gpu") that GPU-accelerates the inference stage: the per-gene Wald covariance and negative-binomial working weights are batched across each gene-block and computed on the accelerator via SpaNorm’s tensor engine, rather than one gene at a time. On the GPU backend the block size is auto-selected to keep peak device memory within the detected budget (override via the newgpu.mem.budget), and a serialBPPARAMis used automatically to avoid multiple processes contending for one device. The batched Wald covariance also speeds up the CPU path. GPU results match the CPU path to a small tolerance (single precision on Metal/MPS); the CPU path stays exactly reproducible and block-size invariant. RequiresSpaNorm (>= 1.7.6).
spiDE 0.99.0
- Initial development version.
buildNiches(),mergeNiches(),computeSizeFactors()for constructing niche covariates and per-cell-type size factors.fitSpiDE()fits a per-gene negative binomial GLM over the neighbourhood interaction design using the SpaNormfitNB()engine.testSpiDE()combines Wald statistics across correlated covariates (Brown’s method) and bandwidths (Cauchy combination), with hierarchical FDR control.spiDE()convenience wrapper for the full workflow.