Mdthbs

Is it advisable to inform the CEO about his brother accessing his office?

How can I get a file's size in C++ in 2019 with C++ 17?

Emphasize numbers in tables

What was the ASCII end of medium (EM) character intended to be used for?

Available snapshots for main net?

Non-inverting amplifier ; Single supply ; Bipolar input

How to extract coefficients of a generating function like this one, using a computer?

GFCI versus circuit breaker

Can you run PoE Cat6 alongside standard Cat6 cables?

German idiomatic equivalents of 能骗就骗 (if you can cheat, then cheat)

How might boat designs change in order to allow them to be pulled by dragons?

Can I deep fry food in butter instead of vegetable oil?

what does "$@" mean inside a find command

Can I hire several veteran soldiers to accompany me?

What is the point of using the kunai?

Why would a propellor have blades of different lengths?

Was Wolfgang Unziker the last Amateur GM?

What is the meaning of "it" in "as luck would have it"?

What is this white film on slides from late 1950s?

Why is quantum gravity non-renormalizable?

Why can't i use !(single pattern) in zsh even after i turn on kshglob?

Confusion in understanding the behavior of inductor in RL circuit with DC source

Are there advantages in writing by hand over typing out a story?

Why are symbols not written in words?

Family-wise Type I error OLS regression

Overall type I error with dependent testsControlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVACorrecting for family-wise error rate with series of repeated measures ANOVA?A situation where ignoring clustering optimises the Type I and Type II error rates?Does the logic of “family-wise error” apply to effect size estimation?Type I/II errorPair-wise comparisons in non-parametric ANCOVA in R/SPSSIn using backward elimination procedure how to control for type I error?How to measure risk of a Type 2 error in A/B testsType 1 Error correction for multiple comparisons: ANOVA vs multiple regression

4

$begingroup$

Why is it advised to control the Type I error rate (e.g. Turkey's HSD) when conducting several pairwise comparisons, but not when assessing the significance of several coefficient estimates when conducting say OLS regression?

regression multiple-comparisons type-i-and-ii-errors

share|cite|improve this question

asked 8 hours ago

ChernoffChernoff

15388 bronze badges

$endgroup$

add a comment | 

4

$begingroup$

Why is it advised to control the Type I error rate (e.g. Turkey's HSD) when conducting several pairwise comparisons, but not when assessing the significance of several coefficient estimates when conducting say OLS regression?

regression multiple-comparisons type-i-and-ii-errors

share|cite|improve this question

asked 8 hours ago

ChernoffChernoff

15388 bronze badges

$endgroup$

add a comment | 

$begingroup$

Why is it advised to control the Type I error rate (e.g. Turkey's HSD) when conducting several pairwise comparisons, but not when assessing the significance of several coefficient estimates when conducting say OLS regression?

regression multiple-comparisons type-i-and-ii-errors

share|cite|improve this question

asked 8 hours ago

ChernoffChernoff

15388 bronze badges

$endgroup$

share|cite|improve this question

asked 8 hours ago

ChernoffChernoff

15388 bronze badges

share|cite|improve this question

asked 8 hours ago

ChernoffChernoff

15388 bronze badges

asked 8 hours ago

ChernoffChernoff

15388 bronze badges

asked 8 hours ago

ChernoffChernoff

15388 bronze badges

1 Answer
1

active

oldest

votes

3

$begingroup$

If your goal is confirmation through hypothesis tests, you should correct for the FWER (or FDR), regardless of the type of model used. If you have a source for the claim to the contrary, please include it in your question.

However, confirmation isn't the only reason someone would use linear regression. You may want to simply predict the outcome variable, or you might just be interested in the magnitude of the effects that the explanatory variables have on the outcome. Personally, I am rarely interested in the $p$-values of my linear models.

Even if you are interested in the $p$-values of a linear regression, which $p$-values you should correct for multiple testing depends on what you are doing, for example:

• Whether the intercept differs significantly from $0$ is rarely interesting. Including this $p$-value in the correction can inflate the type II error rate by increasing the number of tests, or even increase the type I error rate by including a nonsense significant result (in case of FDR correction);


• If your research question revolves around the effect of a single explanatory variable, but you want to include potential confounders, there is no need to even look at those other variables' $p$-values;


• Similarly, if your research questions concerns the presence of a (significant) interaction effect, the significance of the marginal effects may be irrelevant.

For this reason, there is no standard multiple testing correction applied to most of the default summaries of linear models, but you can of course apply your own after deciding which $p$-values matter.

Contrast this with Tukey's honest significant difference: You are comparing every group with every group⁠. Not only is this the maximum number of hypothesis tests you can perform⁠—increasing the risk of poor inference without some standard correction applied⁠—but it also exists exclusively to perform comparisons, whereas linear regression in general can be used for all kinds of purposes.

share|cite|improve this answer

edited 4 hours ago

answered 8 hours ago

Frans RodenburgFrans Rodenburg

4,46811 gold badge55 silver badges3030 bronze badges

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f415451%2ffamily-wise-type-i-error-ols-regression%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

3

$begingroup$

If your goal is confirmation through hypothesis tests, you should correct for the FWER (or FDR), regardless of the type of model used. If you have a source for the claim to the contrary, please include it in your question.

However, confirmation isn't the only reason someone would use linear regression. You may want to simply predict the outcome variable, or you might just be interested in the magnitude of the effects that the explanatory variables have on the outcome. Personally, I am rarely interested in the $p$-values of my linear models.

Even if you are interested in the $p$-values of a linear regression, which $p$-values you should correct for multiple testing depends on what you are doing, for example:

• Whether the intercept differs significantly from $0$ is rarely interesting. Including this $p$-value in the correction can inflate the type II error rate by increasing the number of tests, or even increase the type I error rate by including a nonsense significant result (in case of FDR correction);


• If your research question revolves around the effect of a single explanatory variable, but you want to include potential confounders, there is no need to even look at those other variables' $p$-values;


• Similarly, if your research questions concerns the presence of a (significant) interaction effect, the significance of the marginal effects may be irrelevant.

For this reason, there is no standard multiple testing correction applied to most of the default summaries of linear models, but you can of course apply your own after deciding which $p$-values matter.

Contrast this with Tukey's honest significant difference: You are comparing every group with every group⁠. Not only is this the maximum number of hypothesis tests you can perform⁠—increasing the risk of poor inference without some standard correction applied⁠—but it also exists exclusively to perform comparisons, whereas linear regression in general can be used for all kinds of purposes.

share|cite|improve this answer

edited 4 hours ago

answered 8 hours ago

Frans RodenburgFrans Rodenburg

4,46811 gold badge55 silver badges3030 bronze badges

$endgroup$

add a comment | 

3

$begingroup$

If your goal is confirmation through hypothesis tests, you should correct for the FWER (or FDR), regardless of the type of model used. If you have a source for the claim to the contrary, please include it in your question.

However, confirmation isn't the only reason someone would use linear regression. You may want to simply predict the outcome variable, or you might just be interested in the magnitude of the effects that the explanatory variables have on the outcome. Personally, I am rarely interested in the $p$-values of my linear models.

Even if you are interested in the $p$-values of a linear regression, which $p$-values you should correct for multiple testing depends on what you are doing, for example:

• Whether the intercept differs significantly from $0$ is rarely interesting. Including this $p$-value in the correction can inflate the type II error rate by increasing the number of tests, or even increase the type I error rate by including a nonsense significant result (in case of FDR correction);


• If your research question revolves around the effect of a single explanatory variable, but you want to include potential confounders, there is no need to even look at those other variables' $p$-values;


• Similarly, if your research questions concerns the presence of a (significant) interaction effect, the significance of the marginal effects may be irrelevant.

For this reason, there is no standard multiple testing correction applied to most of the default summaries of linear models, but you can of course apply your own after deciding which $p$-values matter.

Contrast this with Tukey's honest significant difference: You are comparing every group with every group⁠. Not only is this the maximum number of hypothesis tests you can perform⁠—increasing the risk of poor inference without some standard correction applied⁠—but it also exists exclusively to perform comparisons, whereas linear regression in general can be used for all kinds of purposes.

share|cite|improve this answer

edited 4 hours ago

answered 8 hours ago

Frans RodenburgFrans Rodenburg

4,46811 gold badge55 silver badges3030 bronze badges

$endgroup$

add a comment | 

$begingroup$

If your goal is confirmation through hypothesis tests, you should correct for the FWER (or FDR), regardless of the type of model used. If you have a source for the claim to the contrary, please include it in your question.

However, confirmation isn't the only reason someone would use linear regression. You may want to simply predict the outcome variable, or you might just be interested in the magnitude of the effects that the explanatory variables have on the outcome. Personally, I am rarely interested in the $p$-values of my linear models.

Even if you are interested in the $p$-values of a linear regression, which $p$-values you should correct for multiple testing depends on what you are doing, for example:

• Whether the intercept differs significantly from $0$ is rarely interesting. Including this $p$-value in the correction can inflate the type II error rate by increasing the number of tests, or even increase the type I error rate by including a nonsense significant result (in case of FDR correction);


• If your research question revolves around the effect of a single explanatory variable, but you want to include potential confounders, there is no need to even look at those other variables' $p$-values;


• Similarly, if your research questions concerns the presence of a (significant) interaction effect, the significance of the marginal effects may be irrelevant.

For this reason, there is no standard multiple testing correction applied to most of the default summaries of linear models, but you can of course apply your own after deciding which $p$-values matter.

Contrast this with Tukey's honest significant difference: You are comparing every group with every group⁠. Not only is this the maximum number of hypothesis tests you can perform⁠—increasing the risk of poor inference without some standard correction applied⁠—but it also exists exclusively to perform comparisons, whereas linear regression in general can be used for all kinds of purposes.

share|cite|improve this answer

edited 4 hours ago

answered 8 hours ago

Frans RodenburgFrans Rodenburg

4,46811 gold badge55 silver badges3030 bronze badges

$endgroup$

share|cite|improve this answer

edited 4 hours ago

answered 8 hours ago

Frans RodenburgFrans Rodenburg

4,46811 gold badge55 silver badges3030 bronze badges

answered 8 hours ago

Frans RodenburgFrans Rodenburg

4,46811 gold badge55 silver badges3030 bronze badges

answered 8 hours ago

Frans RodenburgFrans Rodenburg

4,46811 gold badge55 silver badges3030 bronze badges