10  GLM-VI: Nested models

Nested models refer to models that are identical, except for the fact that some parameters are constrained to zero in one of them, while all parameters are free in the other. The smaller or constrained model model is said to be “nested in” the larger or unconstrained model, meaning that all predictors in the smaller model are also present in the larger model. By definition, the unconstrained model always provides a better fit than the constrained model.

Incremental F-tests are used to determine whether the increase in \(R^2\) between two nested models is statistically significant. Recall that \(R^2\) represents the proportion of variance in the dependent variable explained by the predictors in the model. The incremental F-test compares the variation in the dependent variable explained by an unconstrained model with the variation explained by a constrained model. The F-test assesses whether the increase in R-squared is greater than what would be expected by chance alone, indicating the significance of adding additional predictors to the model.

Hierarchical regression refers to an approach where predictors are added to a regression model in blocks, allowing us to assess the additional variance explained by each block of predictors with an incremental F-test. Each block represents a set of predictors, and they are added to the model in a stepwise manner. This approach is useful when we want to determine the significance of adding a single categorical predictor which is represented by multiple dummy variables, or when we want to test whether adding some predictors (e.g., theoretically relevant variables) explain significant variance above and beyond other variables (e.g., demographic covariates). By conducting incremental F-tests at each step, we can assess the significance of adding each block of predictors and their contribution to the overall model.

In summary, nested models allow us to compare models with different levels of complexity, incremental F-tests help us determine the significance of adding predictors to a model, and hierarchical regression enables the examination of additional variance explained by different blocks of predictors. These concepts are valuable tools in statistical analysis and can provide insights into the relationships between variables and the predictive power of a model.

11 Lecture

12 Formative Test

A formative test helps you assess your progress in the course, and helps you address any blind spots in your understanding of the material. If you get a question wrong, you will receive a hint on how to improve your understanding of the material.

Complete the formative test ideally after you’ve seen the lecture, but before the lecture meeting in which we can discuss any topics that need more attention

Question 1

Which of the following statements is true about nested models?

Nested models always have the same number of predictors.Model A is nested in Model B if they are identical, except Model A uses a subsample of the sample used to estimate Model B.Model A is nested in Model B if they are identical, except some parameters in model A are constrained.Model A is nested in Model B if they are identical, except some parameters in model B are constrained.

Question 2

What can we say about the relative fit of Model A and Model B if Model A is nested in Model B?

Model A will always have better fit than Model B.Nested models always have equal fit.In nested models, one model always has better fit than the other.Model B will always have better fit than Model A.

Question 3

In the context of nested models, what does ‘constrained’ mean?

Some variables are fixed to 0.The R-squared value of the smaller model is constrained to be lower than that of the larger model.The set of predictors is constrained.Some parameters are fixed to 0.

Question 4

What is the purpose of performing a nested model test?

To compare two different models.To determine if the models are independent of each other.To determine whether adding specific predictors significantly improves the model's fit.To check if the predictors are correlated with each other.

Question 5

Given the nested models: Y = a + b1X1 and Y = a + b1X1 + b2*X2, which model is the ‘constrained’ one?

Y = a + b1*X1Neither is constrained.It cannot be determined from the information given.Y = a + b1X1 + b2X2

Question 6

How does the F-test for the R-squared of a single model relate to the F-test for the delta R-squared of two nested models?

Both are F-tests; these are ratios comparing the explained variance to the unexplained variance.Both can be seen as R-squared tests; both compare the model of interest to a model with only an intercept, which is a nested model test.The similarity is only superficial; although both are F-tests, the calculation is different.Both can be seen as delta R-squared tests; the former compares the model of interest to a model with only an intercept, which is a nested model test.

Question 7

What can we conclude from an incremental F-test in nested models?

Whether the predictors of the nested models are significantly different.Whether the smaller model is significantly better than the larger model.Whether the larger set of predictors has significant multicollinearity.Whether adding specific predictors significantly improves the model's fit.

Question 8

When performing an incremental F-test, what does the numerator in the F ratio represent?

The increase in the number of degrees of freedom when adding predictors to the model.The reduction in the number of predictors from the larger to the smaller model.The reduction in residual mean square when adding predictors to the model.The increase in residual mean square when adding predictors to the model.

Question 9

Which scenario is suitable for using hierarchical regression?

To determine if theoretically relevant factors explain variance beyond demographic characteristics.To test whether each dummy of a categorical predictor with >2 categories explains more variance than the other dummies.To compare two unrelated models.To check the multicollinearity of the predictors.

Question 10

Given two nested models, one has regression sum of squares = 283.96 and residual sum of squares = 958.58; the other has regression sum of squares = 202.76 and residual sum of squares = 1039.78. What is the delta R-squared between these models?

Can't say with this information.085.078.065

Question 11

In a nested model test, numerator degrees of freedom for the F ratio is 4, how many parameters were added to the model?

435Depends on the model

Question 12

In a nested model test, one model has regression sum of squares = 202.76 and 3 parameters and residual sum of squares = 1039.78, the other model has regression sum of squares = 283.96 and 4 parameters and residual sum of squares = 958.58. There are 387 participants. What is the F-value for the nested model test?

28.29Can't say based on this information.32.4424.90

Question 1

Nested models are identical except for some constrained parameters.

Question 2

The bigger model always has a better fit than the smaller model in a nested pair.

Question 3

In the context of nested models, a ‘constrained’ model has some parameters fixed to 0.

Question 4

A nested model test is used to determine whether adding specific predictors significantly improves the model’s fit.

Question 5

The model Y = a + b1*X1 is the ‘smaller’ or ‘constrained’ model.

Question 6

Both F-tests can be seen as nested model tests; the F-test for the R-squared of a single model tests against the variance explained by a model including only the intercept.

Question 7

The incremental F-test determines if adding specific predictors significantly improves the model’s fit.

Question 8

The numerator in the F ratio of an incremental F-test represents the reduction in residual mean square when adding predictors to the model.

Question 9

Hierarchical regression is suitable for determining if theoretically relevant factors explain variance beyond demographic characteristics.

Question 10

The delta R-squared is the difference in regression sums of squares, divided by the total sum of squares. The latter can be calculated by adding the error sum of squares to the regression sum of squares for either model.

Question 11

The numerator degrees of freedom is equal to the difference in number of parameters.

Question 12

The F-value is obtained by first calculating the regression mean square: dividing the difference in regression sum of squares by the difference in model DF: (283.957-202.758)/1, and calculating the residual mean square for the larger model: 958.58/(387-4). The F-value is then ((283.957-202.758)/1)/(958.58/(387-4)).

13 In SPSS

13.1 Multiple Regression

14 Tutorial

14.1 Multiple Regression

This assignment revolves around multiple regression with more than two predictors.

Use the data file called Work.sav. These data were about work characteristics.

Open the date file in SPSS to get started!

Consider the following research question(s):

“To what extent do variety at work, learning possibilities, and independence at work explain pleasure at work, and which of these explanatory characteristics is most important?”

What are the independent variables in this research question?

pleasurevariety, learning possibilities, independence, pleasurevarietyvariety, learning possibilities, independence

Let’s now run a multiple regression analysis to get the output we need to answer our research question.

Navigate to Analyze > Regression > Linear. Enter the dependent variable (pleasure at work). Enter the independent variables (variety at work, learning possibilities, independence at work). Paste and run the syntax.

In the next steps we will go over the output from this analysis.

What percentage of the total variance in work pleasure is explained by variety at work, learning possibilities, and independence at work altogether?

Which of the following statements correctly summarizes the F-test for the R2 in this analysis?

R-square is significant, F(3, 121) = 14.455, p < .001R-square is not significant, F(3, 118) = 14.455, p < .001R-square is not significant, F(3, 121) = 14.455, p < .001R-square is significant, F(3, 118) = 14.455, p < .001

Consider the “Coefficients” table.

What is the unstandardized regression coefficient for the effect of variety at work on work pleasure?

Describe the effect of variety at work on work pleasure as precise as possible.

In other words, how should we interpret the unstandardized regression coefficient for variety at work?

When variety at work increases with one unit and the other two variables stay constant, pleasure at work increases with .272 units.

Now, consider the effect of independence at work on work pleasure.

What is the standardized regression coefficient of the effect of independence at work on work pleasure?

Describe the effect the effect of independence at work on work pleasure as precise as possible using the standardized regression coefficient.

In other words, how should we interpret the standardized coefficient for this predictor?

When independence at work increases with one SD and the other two variables stay constant, pleasure at work increases with .107 SDs.

Which of the predictors has a significant partial effect on pleasure at work when using α =.10? Independence at work Variety at work and Learning possibilities Learning possibilities Variety at work None

In the final couple of steps we went through the output displayed in the “Coefficients” table. Before you proceed with the next assignment, please review the following aspects in the answers your gave:

• When you wrote down the effect of variety at work on pleasure at work in unstandardized form, did you include that it is the effect of the IV on the DV controlled for the other two variables? If not, please keep in mind that this is very important! In multiple regression, all effects are partial (unique) effects, controlling for the other predictors.
• When you wrote down the effect of independence at work on pleasure at work in standardized form, did you indicate that it is the effect in standard deviations, and, again, did you include that it is the effect of IV on the DV controlled for the other two variables (i.e., 1 SD change in independent variable independence at work leads to a .107 SDs change in the predicted score for pleasure at work, controlled for the other variables)?

14.2 Unique Contributions

Use the data file called Work.sav. These data were about work characteristics.

In this assignment, we will look more closely at how to evaluate the “added value” of one predictor in addition to the other predictors in an analysis explaining the dependent variable. Or, put the other way around, how much we lose if we would remove a certain predictor from the model.

We will again consider the analysis in which we predicted pleasure at work by the three predictors learning possibilities, independence at work, and variety at work.

We want to know how much the variable variety at work adds to predicting work pleasure.

Run the two analyses specified below:

Run the regression analysis using all three predictors.

Run the regression analysis analysis with only learning possibilities and independence at work as predictors.

Inspect the R2 of both regression analyses.

What happened with the R2 when variety was removed?

With all three predictors included, R-square equals .269 (see slide 3) With Variety at Work removed, R-square equals .248. So, R-square decreased with .021, indicating that Variety at Work uniquely explains 2.1% of the total variance in Pleasure at Work.

Let’s do the same for the other two predictors; that is, check how much the R2 changes if you would remove the predictor from the model.

Write down the changes below.

With Learning Possibilities removed, R square equals .211. R-square decreased with .058, which means that Learning Possibilities uniquely explains 5.8% of the total variance in Pleasure at Work. With Independence at Work removed, R square equals .260. R-square decreased with .009, which means that Independence at Work uniquely explains 0.9% of the total variance in Pleasure at Work.

Which predictor explained the least unique variance, controlling for the other two? Independence at work Variety at work Learning possibilities

14.3 Hierarchical Regression Analysis

In this assignment we will carry out a hierarchical regression analysis, which is a structured way to compare nested models in linear regression analysis.

With hierarchical regression analysis we are able to compare two (or more) nested models. Consider the model below:

\(Y′ = b0 + b1X1 + b2X2 + b4X4 + b5X5 + b7X7\)

Which of the following models is/are nested within the larger model presented above?

\(Y′=b0+b1X1+b3X3+b4X4\)\(Y′=b0+b1X1+b2X2+b4X4\)\(Y′=b0+b1X1+b2X2+b6X6\)

We will now get back to our data on work characteristics using the data file Work.sav.

Consider the following situation in which researchers are interested in clusters of variables:

Researchers are interested in the effect of “Health” (i.e., mental pressure, physical demands, and emotional pressure) and “Social Environment” (i.e., relationship with coworkers, and relationship with supervisors) on the Need for recovery.

We will address this research question in a number of steps.

First, we will look at the full model (i.e., the model that has all health and social environment predictors).

Run the regression analysis in which you regress Need for recovery on the three health variables (i.e., mental pressure, physical demands, and emotional pressure) and the two social environment variables (i.e., relationship with coworkers, and relationship with supervisors).

What is the R2 of the full model?

Now we want to test the significance of the clusters of variables. We will first look at the Health variables (emotional pressure, physical demands, & mental pressure). We want to test whether Health has a direct effect on Need for recovery controlled for the social environment predictors.

To accomplish this, we have to do a hierarchical regression analysis, in which we compare two nested models: a small one, and a larger one.

Write down what these two models look like.

How many predictors does the smaller model contain?

How many predictors does the larger model contain?

• Small model: \(Recover_i = b0 + b1 * sccowork_i + b2 * scsuperv_i + e_i\)
• Large model: \(Recover_i = b0 + b1 * sccowork_i + b2 * scsuperv_i + b3 *scmental_i + b4* scphys_i + b5* scemoti_i + e_i\)

The small model has 2 predictors and the large model has 5 predictors. The large model has three predictors more than the small model.

Now run the hierarchical regression analysis in SPSS!

Navigate to Analyze > Regression > Linear

Click on “Reset” to start from scratch.

Select Need for Recovery as the dependent variable.

Select only the Social Environment variables (i.e., Relationship with coworkers and Relationship with supervisors) as the independent variables (this is the small model). This is block 1 of 1.

Click on “Next”. Select the three Health variables (i.e., emotional pressure, physical demands, and mental pressure) and add those to this second block.

Important: Now click on the button “Statistics” to request the R2 change.

Paste and run the syntax. Note the following new elements:

  /STATISTICS COEFF OUTS R ANOVA CHANGE
  /METHOD=ENTER sccowork scsuperv
  /METHOD=ENTER scmental scphys scemoti.

The CHANGE command asks for \(R^2\) change statistics, and the two rows of /METHOD=ENTER sequentially add blocks of predictors to the model (enter them into the model).

Let’s inspect the table labeled “Model Summary” first.

What percentage of the total variance in Need for recovery is explained by the small model?

How much of the total variance in Need for recovery is explained by the large model?

What is the R2-change from Model 1 to Model 2?

The Model Summary table also reports the results of the F-tests, which tests whether the change in R2 is significant.

Write down the null and alternative hypothesis that are evaluated by these F-tests, then check your answer?

\(H_0: \Delta R^2 = 0\)

\(H_0: \Delta R^2 \ne 0\)

Suppose three researchers use the output to see whether the R2-change is significant. Researcher I tests at the 10% level, Researcher II tests at the 5% level, and Researcher III tests at the 1% level.

Which of the researchers will conclude that there is a significant result? All three researchers Only researcher I Only researcher III Only researcher II

Suppose the researchers summarized the result as follows:

“Health variables explain 11.3% of the variance in need for recovery.” Yes No

The correct interpretation is:

“The health variables explain an additional 11.3% of the total variance in Need to recover, on top of what is already explained by the social environment variables.”

Alternatively, you may summarize the result as:

“The health variables uniquely explain 11.3% of the total variance in Need to recover, while controlling for the social environment variables.”

Since the R2-change was significant, we have evidence that “Health” had a direct effect on the Need to recover.

Now test whether the social environment variables have an effect on need to recover (while controlling for the health variables). Summarize the results (include all relevant statistics: test statistics, degrees of freedom, p-values).

Include in your answer:

The R2 of the Small and Full model.

The R2-change and whether or not it is significant (use a significance level of .05).

A substantive interpretation of the R2-change (within the context of this assignment).

The first step is to run the hierarchical regression analysis, with the health variables in Block 1 and the social environment variables in Block 2.

The model with social environment variables explains 14.2% of the total variance in Need for Recovery. Health variables explain an additional 7.8% of variance in Need for Recovery, and this difference significantly differs from zero, \(\Delta R^2 = .08, F(2,116) = 5.800, p = .004\). This means that the social environment variables uniquely explain 7.8% of the total variance in Need for Recovery, while controlling for the health variables. The total explained variance in Need for Recovery is 22.0%, which is significantly different from zero, \(F(5, 116) = 6.536, p < .001\).

14.4 Dummies and Continuous Predictors

In this assignment we will predict a continuous outcome variable from a dichotomous predictor (i.e. gender).

The data file that we will use is PublicParticipation.sav.

This data file contains data on the following variables:

• income (higher score = higher income)
• public participation (i.e. being a member of school boards, municipal councilor, etc.)
• education
• age
• gender (0 = females; 1 = males)

We want to test the following research question:

“Are there gender differences in Public Participation?”

To answer this research questions, we can use an independent samples t-test.

Run an independent samples t-test to answer the research question (consult the information button in case you forgot how to run an independent samples t-test.)

Consult the output to answer the questions in the next steps.

Analyze > Compare means > Independent samples t-test.

Choose Public Participation as the dependent variable.

The grouping variable is Gender. Define the groups: group 1 = 0 (i.e., women) and group 2 = 1 (i.e., men).

Paste and run the syntax.

What is the mean difference between men and women?

Consult the table Group Statistics.

“In the sample, men score on average points higher on public participation than women.”

Which of the following statements correctly summarizes the results of Levene’s test (use α=.05)?

Levene's test is not significant, no evidence against the assumption of homoscedasticityLevene's test is significant, evidence against assumption of homoscedasticity.

The researchers conclude:

“We have convincing evidence that the population means of public participation differs between males and females.”

Is this a valid conclusion? Yes No

Report the test results, then check your answer.

The mean level of public participation differs between males (\(M = 16.54\)) and females (\(M = 9.95\)), \(t(41) = -4.942, p <.001\).

What is the (absolute) value of the t-statistic?

From the results of the t-test the researchers would conclude that males on average have higher levels of public participation than females.

But, of course, we don’t know for sure, because we did not test the entire population. Which error type could the researchers have made? Type I error Type II error Type I or Type II error

14.4.1 Examining Income

In the previous steps, we used the independent sample’s t-test to test for mean differences in public participation and found a significant result. Results suggested that males show higher levels of public participation. However, males and females may differ not only in gender, but also in other characteristics that explain differences in public participation. In particular, the researchers hypothesize that income differences may play a role as well. Research has shown that income is positively associated with public participation. So if males and females differ in income (on average), the gender differences in public participation may be partly due to income differences.

In other words, according to the researchers income may be a collider moderator confounder predictor of the effect of gender.

Before we proceed, let’s first check whether the groups differ on income. Request the means for the variable income for both males and females separately.

Do the males and females in the sample differ in average income?

Take your existing syntax for gender differences in Participation, and change it to test for income differences instead:

T-TEST GROUPS=Gender(0 1)
  /MISSING=ANALYSIS
  /VARIABLES=Income
  /ES DISPLAY(TRUE)
  /CRITERIA=CI(.95).

Is there a difference in income? Yes No

The next step is to study gender differences controlled for income differences. This is not possible via the independent samples t-test interface, but it is possible when specifying our model using regression analyses. Regression analysis allows us to study gender differences controlling for other variables.

14.4.2 Add continuous predictor

Before we proceed with the regression analysis including income (we will do this in the next assignment), we will first see what the regression model looks like if we would only use gender as the only predictor in the model.

Run a regression analysis using Public Participation as the dependent variable, and gender as the independent variable (Analyze > regression > linear). Note that the variable gender is already dummy coded.

Inspect the table with Coefficients.

What is the regression slope of gender?

Compare the value to the mean difference from ; what do you see?

Recall that the independent samples t-test is the exact same test as the t-test for the regression slope of a dummy variable.

Compare the value of the t-statistic in the regression with the value of t-statistic in the independent samples t-test (you may ignore the minus signs). What do you see?

The regression coefficient of gender is 6.502. This is the same as the difference in means of Public Participation between males and females. The t-statistic of the regression coefficient is 4.942. The t-statistic in the independent samples t-test was -4.942. As we can ignore the minus sign (just make sure to check which group is higher than the other), we can conclude they are equal.

What percentage of the total variance in public participation is explained by gender?

Recall that the independent samples t-test and regression analysis with a dummy variable predictor are completely equivalent; SPSS just offers different interfaces to the same linear model.

We need to use the regression interface when we want to add other predictors (e.g., income) to see if the gender differences are confounded by other characteristics, and we can calculate the R2 value to evaluate the size of the effect of gender.

Next, we want to know whether there are differences in public participation between men and women with the same income level.

In other words: let’s look at gender differences in public participation controlled for income.

Run a regression analysis of public participation on income and gender (analyze -> regression -> linear). Consult the output and answer the following questions:

Gender and income jointly explain % of the variance in public participation.

Which of the following alternatives correctly report the F-statistic testing the significance of the R2? F(2,40)=26.845, p < 0.001 F(2,42)=26.845, p < 0.001

The researchers conclude:

“On average, women score 4.818 points higher on public participation than men.”

Is this a valid conclusion? No Yes

I hope you concluded that the researchers’ conclusion is incorrect.

They should have said:

“On average, women score 4.818 points less on public participation than men of an equal income level”

So, if we take two groups of men and women with the same income (= “controlling for income”), then we expect a mean difference of 4.818 in public participation.

Is there evidence of gender differences, while controlling for income differences when testing at \(\alpha = .01\)? Yes No

Write down the complete regression equation, then check your answer below.

The regression equation is as follows:

\(\text{Participation}_i = 5.762 + 3.475*\text{Income}_i+ 4.818 * \text{Gender}_i+e_i\)

Suppose we have a man with an income of 3.0 and a woman with an income of 2.0, what is the predicted difference in public participation?

Man with an income of 3 has a predicted score of:

Public Participation′=5.762+3.475∗3+4.818 ∗1

Public Participation′=21.005

Woman with an income of 2 has a predicted score of:

Public Participation′=5.762+3.475∗2+4.818 ∗ 0

Public Participation′=12.712

Difference = 21,005 – 12,712

Difference = 8.293

We previously computed the mean difference in public participation for men and women, which was 6.50.

Why is this previous mean difference not the same as the effect of gender that we find in this most recent regression analysis?

The mean difference as observed in the sample is the mean difference without controlling for income differences.

However, as we have seen, men and women also differ in the average income, and income differences between males and females may also explain differences in public participation. In the regression analyses, we controlled for income differences, which means that we “filtered out” the effect of income (the correct way to say it: we partialed out income effects). After controlling for income the effect of gender is somewhat smaller. Hence, the difference in public participation in the sample means is partly due to income differences.

One researcher argues that there are gender differences in public participation, and gender differences in income, and that income additionally has an effect on public participation.

In this theoretical model, income is a Moderator Common cause Confounder Mediator

14.5 Nested Models

Finally, we might want to know how much unique variance is explained by gender, after controlling for income. To this end, we would perform a nested model test.

Conduct this test as you have done before, using the graphical interface or syntax. Remember: Enter Income first, then Gender - and ask for the \(R^2\) change statistics.

Let’s inspect the Model Summary table.

What percentage of the total variance in Participation is explained by Income (only)?

How much of the total variance in Participation is explained by both Income and Gender?

What is the R2-change (in percentage) from Model 1 to Model 2?

The Model Summary table also reports the results of the F-tests, which tests whether the change in R2 is significant.

Suppose three researchers use the output to see whether the R2-change is significant. Researcher I tests at the 10% level, Researcher II tests at the 5% level, and Researcher III tests at the 1% level.

Which of the researchers will conclude that there is a significant result? All three researchers Only researcher I Only researcher II Only researcher III

Report the results of your analysis (include all relevant statistics: test statistics, degrees of freedom, p-values).

Include in your answer:

• The R2 of the Small and Full model.
• The R2-change and whether or not it is significant (use a significance level of .05).
• A substantive interpretation of the R2-change (within the context of this assignment).

Income explains 39.1% of variance in Participation, which is significantly greater than zero, \(F(1,41) = 26.32, p < .001\). Gender explains an additional 18.2% of variance in Participation, and this difference significantly differs from zero, \(\Delta R^2 = .18, F(1,40) = 17.06, p < .001\). The total explained variance in Participation by Income and Gender is 55.2%, which is significantly different from zero, \(F(2, 40) = 26.85, p < .001\).

14.6 One more Categorical Variable

Let’s finish this assignment by adding one more categorical variable, and seeing whether it has a significant unique effect after controlling for Income and Gender. We will use Education level, which has three categories. We thus have to create two dummy variables to account for this variable.

First, make the dummy variables, using low Education status as reference category.

Remember that these dummies together code for one variable, so you must add them to the model in the same step together.

What is the F test statistic for the \(\Delta R^2\) test for adding Education to a model that already contains Income and Gender?

What is the difference in expected Participation between someone with Low education status and High education status?