Machine learning can help advance theory formation in developmental psychology

• Replication crisis (Scheel 2022; Lavelle 2021)
• Solutions improve deductive (theory-testing) research practices
  • questionable research practices (QRPs)
  • replication
  • preregistration

Many preregistered hypotheses are not supported (Scheel, Schijen, and Lakens 2021)

• Lack of “good theories” another explanation for replication crisis

Theory: A model that describes the nature of relations between several phenomena in sufficient detail that it allows for the derivation of quantifiable hypotheses that can be subject to (severe) testing.

• most psychological theories fall short of this definition
  • insufficiently precise
  • too flexible
• Compare to Walasek, Frankenhuis, and Panchanathan (2022)

Why should statisticians care about theory?

• It dictates what models are sensible
• Useful as an instrument of cumulative knowledge acquisition
• Communicating social scientific research to its consumers

• Lack of theory cannot be overcome by improving deductive research
• Theory formation requires inductive (exploratory) research (Creswell and Clark 2017)

De Groot’s empirical cycle: a model of knowledge production through scientific research (de Groot 1961)

How can we improve exploratory research?

Role of flexibility:

• Confirmatory methods poorly suited to exploratory research
  • p-values: Type I error is 5%, but false discovery rate (FDR) is 14-50% (Vidgen and Yasseri 2016).
  • Model fit indices: manually specify models, no guarantee that best model is included

“unguided exploration” is effortful and inflates the risk of spurious results

\[ FDR = \frac{FP}{TP + FP} \]

Machine learning:

• automated model building
• checks and balances to prevent overfitting
• maximize predictive performance and generalizability of results

Psychology can benefit from its superior predictive performance (Yarkoni and Westfall 2017)

• E.g., personalized (mental) health care, automated assessment aids

NEW: Implications for the theory crisis have not yet been discussed

First step in Theory Construction Methods is identifying relevant phenomena (Borsboom et al. 2020)

• Phenomena: stable and general features of the world
• Few quantitative methods for phenomena detection (aside from expert opinion)

Text mining may be suitable for phenomena detection (van Lissa 2021)

• Unsupervised learning method
• Clusters in keywords and abstracts

Every study examines only a piece of the puzzle; we never see the complete picture

Machine learning accommodates more predictors than classical methods

• Regularization
• Variable selection

Including all relevant predictors is important:

• Assumption for causal interpretation
  • Multicollinearity
• Good theory incorporates most important causes
  • Cannot assess the relative utility across studies
• Include potential predictors from various theories, and undertheorized factors

Many machine learning methods accommodate:

• non-linear effects
• higher-order interactions, without having to specify the nature of these effects a-priori.
• Automatic: tree-based methods
• Manual: penalized methods

Psychological theories rarely account for complex effects

• Machine learning can provide nuance by revealing them

Some machine learning methods incorporate theoretical elements

• E.g.: assumption that development follows a latent growth curve (LGC).
• SEM forests (Brandmaier et al. 2016)
• regularized SEM (Jacobucci, Grimm, and McArdle 2016)

When the theory is a (nomological) network:

• LASSO-penalized Gaussian graphical model (GGM) (Epskamp, Rhemtulla, and Borsboom 2017)
• E.g.: network theory of major depression (Cramer et al. 2016)

All of these methods allow for theory guided exploration using machine learning

Explain heterogeneity at a more fine-grained level than the whole sample

• E.g., latent class analyses
  • “Which individuals are similar?”
  • Other unsupervised learning methods exist
• RI-CLPM and DSEM
  • Rarely explain heterogeneity in within-person effects
  • Regularization
• Tree-based models
  • Group individuals based on predictors to maximize homogeneity of the outcome
  • “Why are these individuals similar?”

When using data models to guide theory formation, generalizability is essential

Checks and balances to curtail overfitting

• cross-validation to estimate predictive accuracy

Naive interpretative approach

• Variable importance metrics
  • congruent with theoretical assumptions about important predictors?
  • any theoretically important predictors rank low?
  • any undertheorized factors rank high?
• Marginal associations
  • non-linear effects?
  • high importance but flat marginal association?

Comparing predictive performance of simpler parametric model that represents these ‘insights’ to machine learning model

Data models -> formal theory (Haslbeck et al. 2021)

• abductive formal theory construction (AFTC) framework
  • construct formal theory, generate data, mathematically compare to empirical data
  • Discrepancies: amend formal theory

• There is a paucity of good theory
• Need for exploratory research for theory formation
• Machine learning for rigorous exploration
  • automates model building
  • incorporates checks and balances for generalizable results
• Unsupervised learning can assist in phenomenon detection
• Supervised learning to identify important predictors
• Some algorithms incorporate basic theoretical elements

Developmentally sensitive period (Zimmermann and Iwanski 2014)

20% develop psychopathology (Lee et al. 2014)

Potentially lifelong implications for mental health and well-being

Substantial empirical research, but no overarching theoretical framework (Buss, Cole, and Zhou 2019)

First step: identifying relevant phenomena (Borsboom et al. 2020)

We conducted a text mining systematic review (TMSR) (van Lissa 2021)

• Narrative reviews: small samples, confirmation bias, emphasize positive results (Littell 2008)
• TMSR: Unlimited sample size, transparent, objective, reproducible

6653 papers on Addresses emotion regulation in population overlapping with adolescence [10-24]

<doi.org/10.1007/s40894-021-00160-7>

All code and data available at https://github.com/cjvanlissa/veni_sysrev

Workflow for Open Reproducible Code in Science (WORCS) used to make analyses reproducible (Van Lissa et al. 2020)

Phenomena relevant to adolescents’ emotion regulation according to theory (a) and narrative reviews (b; transparent nodes indicate constructs also present in the theory).

1. Theory (b) narrative reviews; transparent nodes indicate constructs also present in theory.

Author keywords (a) and abstracts (b)

• Both analyses reflected some constructs from theoretical literature
  • Particularly related to neurodevelopment and socialization
• Mental health-related outcomes feature prominently
  • Emotion regulation implicated in mental health problems (Lee et al. 2014),
  • This underlines the importance of research in this area
• Substantial correspondence between keywords and abstracts, suggesting validity of method
• Networks are sparse; few connections among constructs: fragmented literature

• Developmental disorders
• Physical health
• External stressors
• Structural disadvantage
• Addictive behavior
• Identity and moral development
• Sexual development

• Predictors of latent growth model?
  • Only linear differences, few predictors (power)
• Multigroup latent growth model?
  • Non-linear differences, but only one moderator
• Latent class growth model with auxilliary variables?
  • Groups with regard to trajectory, not predictors

• Allows non-linear differences between trajectories
• Performs variable selection
  • Cast wide net among potentially relevant predictors
• Checks & balances ensure generalizability
• Exploratory: No hypothesis

SEM-forest (Brandmaier et al. 2016)

1. Bootstrap sample
2. On each sample, estimate a SEM-model
3. Consider \(k\) candidate predictors
4. Identify predictor and value that maximizes \(LR\) of post-split multi-group model
5. Average predictions across bootstrap samples

We used 1000 bootstrap samples, \(k = 9\)

RQ1: What are the most important predictors of adolescents’ trajectories of emotion regulation development?

RQ2: What is the nature of the association of the predictors with the trajectories?

Participants

• RADAR data (https://www.uu.nl/en/research/radar)
• 497 Dutch adolescents (283 boys; age at T1: M =13.03, SD = 0.46)
• Collected between 2006-2011
• Most families were medium- to high-SES (10% low-SES)
• 87 candidate predictors
• DV: quadratic growth model of Difficulties in emotion regulation (Gratz and Roemer 2004)

Workflow for Open Reproducible Code in Science (WORCS) used to make analyses reproducible (Van Lissa et al. 2020)

All code and synthetic data available at

https://github.com/cjvanlissa/veni_forest

Important predictors:

• Personality and related constructs
• Best parental predictor is autonomy support (bal. rel.)
  • Negative parenting more predictive than positive parenting
• Conflict frequency and behavior
• Empathy, particularly personal distress

Less important than expected:

• SES
• Bullying/victimization
• Delinquency
• Substance use
• Monitoring

• Most important predictors are routinely assessed (big 5) or overt (conflict)
  • Prime candidates for early risk assessment
  • Conflict resolution behavior can be taught
    • Target for intervention if association is causal
• Emphasis on parenting (in literature and theory) might not be justified
  • Congruent with RI-CLPM showing few parenting effects in adolescence (Van Lissa et al. 2019)
• Order of predictors mirrors bioecological model

• Many predictors show non-linear effects: Emotion dysregulation only for +/-1SD
• Some predictors show almost no marginal effect (e.g., father’s age, drug use)
  • This suggests they might be important in interactions

• Proximal factors more predictive of ER development than distal predictors
• Parenting may be less relevant than previously thought
• Personality and conflict behavior are candidates for early diagnosis
• Conflict skills training might be avenue for intervention