Skip to contents

FAIRifying the Dunning-Kruger Effect

Source: vignettes/dunning-kruger.Rmd
dunning-kruger.Rmd

Introduction

In this example, we formalize the Dunning-Kruger (DK) effect according to (Feld, Sauermann, and De Grip 2017). The paper defines the DK effect as “low performers vastly overestimate their performance while high performers more accurately assess their performance”. It further contains a discussion of the DK effect and then restates it in terms of skill and overconfidence to show that measurement error can cause significant bias in the relationship between performance and overestimation. Statistical methods that can be used to correct for this bias are also discussed. Since this theory contains definitions of abstract concepts, relationships between the concepts, mathematical derivations as well as commonly used statistical models and experimental paradigms, it serves as a nice illustration on how to FAIRify all these different aspects.1

Learning Goals

Once you have completed this tutorial, you will know how to:

Definitions

Let’s start with collecting all definitions the theory makes use of:

  • performance as a test score
  • performance estimation as the difference between the expected and the actual test score
  • skill as the ability to perform well on a given test
  • overconfidence as the difference between self-assessed and actual skill
  • measurement error as luck on a test

Since these are verbal definitions, we can track them as a markdown file:

definitions <- 
"
## Definitions

- **performance** as a test score
- **performance estimation** as the difference between the expected and
the actual test score
- **skill** as the ability to perform well on a given test
- **overconfidence** as the difference between self-assessed and actual skill
- **measurement error** as luck on a test
"

cat(definitions, file="definitions.md")

Relationships

We can visualise the originally proposed relationships between the concepts as a graph:

As well as the reformulation of Feld, Sauermann, and De Grip (2017):

With −- signifying a negative association, ≃\simeq signifying “measured by” and :=:= signifying “defined as”.

To FAIRify this graph, we can use a graph specification library such as igraph (Csárdi et al. 2025):

library(igraph, warn.conflicts=FALSE)

g <- graph_from_literal(
    skill -- overconfidence,
    skill -- performance,
    overconfidence -- overestimation,
    performance -- overestimation,
    "skill + error" -- "overconfidence - error",
    "skill + error" -- performance,
    "expected performance - performance" -- overestimation,
    "expected performance - performance" -- "overconfidence - error"
)

E(g)$relationship <- c(
 "negative association",
 "≃",
 "≃",
 "negative association",
 ":=",
 ":=",
 "negative association",
 "= (Theorem 1)"
)

We can visualize this graph with

plot(
  g,
  vertex.size = 20,
  vertex.color = "white",
  edge.label = E(g)$relationship,
)

Finally, we save the graph in a standardized format such as GraphML:

write_graph(
  g,
  "relationship_graph.txt",
  format = "graphml"
)

Mathematical formulation

Definitions

We define the random variables

  • s*s^* denoting skill
  • ε\varepsilon denoting measurement error, with 𝔼[ε]=0\mathbb{E}[\varepsilon] = 0, ε\varepsilon independent of all other random variables included in the model
  • ss*s^*_s denoting self-assessed skill

And further performance pp as (#eq:p)p=s*+ϵ\begin{equation} (\#eq:p) p = s^* + \epsilon \end{equation} overconfidence oc*oc^* as (#eq:oc)oc*=ss*−s*\begin{equation} (\#eq:oc) oc^* = s^*_s-s^* \end{equation} expected performance pep_e as (#eq:ep)pe=s*+oc*\begin{equation} (\#eq:ep) p_e = s^* + oc^* \end{equation} Overconfidence oc*oc^* is measured by overestimation oeoe defined as (#eq:oc)oe=pe−p\begin{equation} (\#eq:oc) oe = p_e - p \end{equation}

Theorems

Theorem 1: oe=oc*−ϵ\begin{equation} oe = oc^* - \epsilon \end{equation}

Proof:

From eq. @ref(eq:oc) and @ref(eq:ep) it follows that pe=ss*p_e = s^*_s and further from eq. @ref(eq:ep) and @ref(eq:p) we see (#eq:dd)oe=pe−p=(s*+oc*)−(s*+ϵ)=oc*−ϵ\begin{align} (\#eq:dd) oe &= p_e - p \\ &= (s^* + oc^*) - (s^* + \epsilon) \\ &= oc^* - \epsilon \end{align}

Since there is no accepted standard on how to represent mathematical knowledge as a digital object (see also this whitepaper), there are many possible routes to FAIRify equations. Here we opt for a representation as latex code as a widely used and known way of typesetting equations. First, we create a file “equations.tex” containing the actual derivations:

\section{Definitions}
Define random variables
\begin{itemize}
 \item $s^*$ denoting skill
 \item $\epsilon$ denoting measurement error, with $\Exp[\epsilon] = 0$, $\epsilon$ independent of all other random variables included in the model
 \item $s^*_s$ denoting self-assessed skill
\end{itemize}

\noindent Then we define performance $p$ as
\begin{equation} \label{p}
  p \coloneq s^* + \epsilon
\end{equation}
and overconfidence $oc^*$ as
\begin{equation} \label{oc}
  oc^* \coloneq s^*_s-s^*
\end{equation}
and expected performance $p_e$ as
\begin{equation} \label{ep}
  p_e \coloneq s^* + oc^*
\end{equation}
Overconfidence $oc^*$ is measured by overestimation $oe$ defined as
\begin{equation}
  oe \coloneq p_e - p
\end{equation}

\section{Theorems}

Theorem 1:

\begin{equation}
  oe = oc^* - \epsilon
\end{equation}

Proof 1:
\noindent From eq. \ref{oc} and \ref{ep} it follows that $p_e = s^*_s$ and further from eq. \ref{ep} and \ref{p} we see
\begin{align} \label{dd}
  oe &= p_e - p \\
  &= (s^* + oc^*) - (s^* + \epsilon) \\
  &= oc^* - \epsilon
\end{align}

Then, we create a file “render.tex” containing the necessary information (document format, packages, commands) that can be used to render the equations:

\documentclass[a4paper,11pt]{article}

% load packages
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{parskip}

% Statistics
\newcommand{\Var}{\mathbb{V}}
\newcommand{\Exp}{\mathbb{E}}

% commands
\renewcommand*{\epsilon}{\varepsilon}

% operators
\DeclareMathOperator{\cov}{cov}

\begin{document}

\input{equations.tex}

\end{document}

As you can see, we use \input{equations.tex} to insert the equations into the document. This way, the mathematical theory is version controlled separately from the LaTex code required to render it. This way, it is clear when changes are made to the theory (i.e., equations.tex is edited), and when changes are made to the formatting of the theory (i.e., render.tex is edited).

Statistical Models

Using a linear regression model, the Dunning-Kruger effect can be stated as oc*=α+β1s*+u\begin{equation} oc^* = \alpha + \beta_1 s^* + u \end{equation} with β1<0\beta_1 < 0. Substituting the observable variables and rearranging according to eq. @ref(eq:p) and @ref(eq:dd): oe=α+β1p+u−ϵ(1+β1)\begin{equation} oe = \alpha + \beta_1 p + u - \epsilon(1 + \beta_1) \end{equation}

Correction

There are different ways to correct for the bias introduced by measurement error:

  • Bias correction: use a bias correction formula that takes into account the correlation between performance and the error term
  • IV approach: measure performance on a second test (p2p_2) and compute β1=cov(oe,p2)cov(p,p2)\beta_1 = \frac{\mathrm{cov}(oe, p_2)}{\mathrm{cov}(p, p_2)}.

Let’s add this model again as latex code by adding a new file “linear_model.tex”:

\subsection{Linear Model}
Using a linear regression model, the Dunning-Kruger effect can be stated as
\begin{equation}
 oc^* = \alpha + \beta_1 s^* + u
\end{equation}
with $\beta_1 < 0$.
Substituting the observable variables and rearranging according to eq. \ref{p} and \ref{dd}:
\begin{equation}
  oe = \alpha + \beta_1 p + u - \epsilon(1 + \beta_1)
\end{equation}

\subsubsection{Correction}
There are different ways to correct for the bias introduced by measurement error:
\begin{itemize}
 \item Bias correction: use a bias correction formula that takes into account the correlation between performance and the error term
 \item IV approach: measure performance on a second test ($p_2$) and compute $\beta_1 = \frac{\cov(oe, p_2)}{\cov(p, p_2)}$.
\end{itemize}

and adding to render.tex:

...
\section{Statistical Models}
\input{linear_model.tex}
...

If we now render render.tex, the resulting document looks like this:

Rendered Document

References

Csárdi, Gábor, Tamás Nepusz, Vincent Traag, Szabolcs Horvát, Fabio Zanini, Daniel Noom, Kirill Müller, David Schoch, and Maëlle Salmon. 2025. igraph: Network Analysis and Visualization in r. https://doi.org/10.5281/zenodo.7682609.
Feld, Jan, Jan Sauermann, and Andries De Grip. 2017. “Estimating the Relationship Between Skill and Overconfidence.” Journal of Behavioral and Experimental Economics 68: 18–24.