Term
| Short answer tests contain ____ that MC tests do not. |
|
Definition
|
|
Term
| Effects of guessing are usually _____. |
|
Definition
| underestimated BC can usually eliminate at least one distractor |
|
|
Term
| Short answer tests _____ more reliable than MC tests. |
|
Definition
|
|
Term
| Reliabilities are influenced by _____. |
|
Definition
|
|
Term
| Increase reliability by ____ not ____. |
|
Definition
| increasing # items; NOT by increasing alternative responses |
|
|
Term
| With power tests, correlation between ___ and __ approaches 1.0, even if two means are different. |
|
Definition
| correlation between scores obtained with the time limit; hypothetical scores |
|
|
Term
| A test is considered a power test when ____. |
|
Definition
| scores obtained with and without time limits are highly correlated |
|
|
Term
| Is it ever advised to replace a power test with a speed test? |
|
Definition
| Only if underlying trait obviously involves speed |
|
|
Term
| Reliability and construction of power tests relies heavily on ______. |
|
Definition
| the sizes and patterns of correlations among items (has to do with internal structure of speed tests) |
|
|
Term
| A time limit affects ____, _____, and _____. |
|
Definition
1)pattern of correlations among items
2)pattern of item-total correlations
3)average correlations |
|
|
Term
| What are 2 problems associated with instructing people not to guess? |
|
Definition
1)difficult to frame instructions clearly
2)effects of the instructions vary over students and create an irrelevant source of individual differences |
|
|
Term
| How does guessing affect parameters? |
|
Definition
1)causes estimated mean score to be larger than it would have been if left blank
2)people who know the least gain the most from guessing
--> variance of errors due to guessing adds to the measurement error from the previously considered sources
3)makes scores less reliable for high ability |
|
|
Term
| What is the blind guessing model? |
|
Definition
Abbott's formula
Assumes the probabilities of correctly guessing on different items are independent
1/k = probability of guessing correct
Q = 1-p = (L-1)/K = probability of guessing incorrect
R-W/(k-1) = score a subject receives on a MC test - # wrong responses /(# alternative responses - 1) = Abbott's formula = Correction for guessing |
|
|
Term
|
Definition
| Measurement artifact that emerges from specific situations |
|
|
Term
|
Definition
| Characteristic of an individual that is consistent across situations |
|
|
Term
| What is reverse regression? |
|
Definition
| Artifact of the regression model in which the same data may simultaneously appear to demonstrate that of a focal group is both underpaid and underqualified |
|
|
Term
| WRT the impact a test has, which approach is oriented toward individuals? groups? |
|
Definition
Linear or moderated multiple regression
Quoteas |
|
|
Term
| Speed tests have ___ which ____. |
|
Definition
time limits
Maximize the time differences and therefore reliability |
|
|
Term
| What is and is not appropriate with speed test methods? |
|
Definition
Appropriate: Test-retest
NOT appropriate: Coefficient alpha |
|
|
Term
|
Definition
| a test in which the presence of a time limit does NOT contribute to individual differences |
|
|
Term
| What is Abbott's formula? |
|
Definition
correction for guessing
assumes subjects either:
(a) know the correct answer or
(b) guess blindly |
|
|
Term
| guessing lowers reliability BC ____. |
|
Definition
| two individuals with the same underlying knowledge may get different scroes BC of differences in luck |
|
|
Term
| Oblique rotation in component solution will be more nearly ____ than oblique rotation produced by Common Factor solution BC _______. |
|
Definition
orthogonal;
Unique error that is part of the component structure will attenuate correlation |
|
|
Term
| Common factor solutions will be (more/less) biased, so the mean value will be closer to ___. |
|
Definition
|
|
Term
| Component solution will ___ actutal magnitude of correlations BC ___. |
|
Definition
overestimate
bias is present |
|
|
Term
| In Principle Components, residuals ___ when number of factors _____. |
|
Definition
|
|
Term
| Absolute magnitude will always ___ in Component solution. |
|
Definition
|
|
Term
| With Common Factor solution, magnitude will ____. |
|
Definition
sometimes increase
this implies too many factors have been extracted |
|
|
Term
| Residuals will be ____ in absolute magnitude in the Common factor solution for a given number of components. |
|
Definition
|
|
Term
| Component Eigenvalues will ____ Common Factor Eigenvalues BC ____. |
|
Definition
exceed
Accounts for more error |
|
|
Term
| Common factors will fit the model ___ than a component solution BC ____. |
|
Definition
|
|
Term
|
Definition
maximize spread of variance of pattern elements on a factor
starts with orthogonal structure and then determines an ideal pattern having greater spread than the orthogonal structure |
|
|
Term
|
Definition
Cleans up items
Useful when wish to stress a general factor with which all variables correlate |
|
|
Term
| What does it mean when pattern and structure matrices are similar? |
|
Definition
|
|
Term
| What is a varimax rotation? |
|
Definition
Designed to eliminate general factors
Captures meaning of simple structure within the confines of an orthogonal framework |
|
|
Term
| If correlations are low in a factor correlation, what type of rotation should you use? |
|
Definition
|
|
Term
| Pattern elements ____. They are always ____ and may be ____. |
|
Definition
-Describe one variable per unit changes in a factor, holding all other variables constant
-They are always regression weights
-They may be standardized into beta weights |
|
|
Term
| h2 of ____ is the sum of the squared structure AND pattern elements. |
|
Definition
|
|
Term
| h2 of ____ is the sum of the squared structure OR pattern elements. |
|
Definition
|
|
Term
| In an orthogonal solution, the S matrix is ____ the pattern matrix. |
|
Definition
|
|
Term
| The more _____ factors are, the more different structure and pattern matrices become. |
|
Definition
different
However, in orthogonal solution, the correlation matrix plays NO role BC assumes items are not correlated. |
|
|
Term
| All relevant properties are included in the ___ matrix. |
|
Definition
|
|
Term
| Oblique factors generally represent ___ variables better than orthogonal factors. |
|
Definition
|
|
Term
| In a nutshell, what is the difference between varimax, quartamax, and promax? |
|
Definition
Varimax: cleans up constructs
Quartamax: Cleans up items
Promax: Starts orthogonal, then goes oblique |
|
|
Term
| What is the difference between pattern and structure matrices? |
|
Definition
Pattern: relationship b/n item and components,HOLDING THE ITEMS CONSTANT
--> Contains correlations b/n the variables (rows) and factors (columns)
Structure: Relationship of the items and components, IGNORING THE OTHER ITEMS
--> Contains the regression weights used to predict the variables (rows) from the factors (columns) |
|
|
Term
| What will a proper factor rotation do? |
|
Definition
Strengthen the relationship b/n variables and factors
Factors will better represent variables that belong to it, rather than those that do not belong
Concentrates the variation shared by 2 variables that correlate highly on a single factor, rather than on several factors |
|
|
Term
| Will structure or pattern elements be higher? Why? |
|
Definition
Structure
BC it includes unique variance |
|
|
Term
|
Definition
| How many times you have to work the model to get a solution. |
|
|
Term
| What does the factor correlation matrix contain? |
|
Definition
| Correlations among factors. |
|
|
Term
| Which guessing model states that you have a higher probability of guessing correctly on a second try? |
|
Definition
|
|
Term
| What is the mean and standard deviation of K-alternative forced choice tasks WRT incorrect alternatives? How about mean value correct alternatives? |
|
Definition
Set at 0, SD of 1
Set at d', SD of 1 |
|
|
Term
| True/False: Imposing a time limit makes a measure a speed test. |
|
Definition
|
|
Term
| True/False. Test items for a speed test are of trivial difficulty. |
|
Definition
| True. (example = addition) |
|
|
Term
| With the sophistocated guessing model, ___ is a sufficient estimator of d'. |
|
Definition
| # correct responses obtained when subjects respond to all trials |
|
|
Term
| Factor analysis can be used to determine: |
|
Definition
1) groupings or clusterings of variables
2) which variables belong to which group and how strongly they belong
3) how many dimensions are needed to explain the relaitons among the variables
4) a frame of reference (coordinate axis) to describe the relations among the variables more conveniently
5) scores of individuals on such groupings |
|
|
Term
| What is an "effect indicators"? |
|
Definition
Observable variables that are regarded as outcomes of an underlying latent variable
e.g. Common factor analysis |
|
|
Term
|
Definition
| variables are simply transformed to other variables for convenience |
|
|
Term
| What are "Causal indicators"? |
|
Definition
| latent variables that are regarded as the outcomeof the observables |
|
|
Term
| What is the key to successful factor analysis? |
|
Definition
1) careful choice of variables
2) selection of subjects to ensure that all variables of interest correlated highly iwth other variables |
|
|
Term
| Factor analysis is a general method of ____. |
|
Definition
| Decomposing the variance of a measure into 1+ common factors reflecting what variables share + additional unique factors which normally describe variance in a mesuare that CANNOT be shared among other individuals |
|
|
Term
| In Factor Analysis, variables are expressed as ___. |
|
Definition
| weighted linear combinations of factors where the weightings are termed "pattern elements" |
|
|
Term
| How does the component model differ from the commmon factor model? |
|
Definition
| It IGNORES unique factors |
|
|
Term
| Unique variance is broken down into (2 parts): |
|
Definition
1) Measurement error (unreliability)
2) Specific variance which is systematic but not shared with other variables in the analysis |
|
|
Term
| What is the difference between a general and group factor? |
|
Definition
General factors relate to all variables
Group factors relate to some variables |
|
|
Term
| What are the 2 stages of factor analysis? |
|
Definition
1) the direct solution condenses the variance shared among the variables & defines the # of factors
2) Second stage of rotation makes final result more interpretable |
|
|
Term
| Direct solutions are nearly always _____ (correlated/uncorrelated). |
|
Definition
|
|
Term
| What are the 3 approaches to condensation? |
|
Definition
1) Defining a factor's content in advance (e.g. as the sum of variables in the analysis --> Centroid analysis)
2) Maximizing a property of the sample (e.g. by accounting for the most possible variance --> Principle component and Principle axis analysis)
3) Estimating population parameters (e.g. choosing the most probable outcome given the data ML analysis) |
|
|
Term
| What is the general rule about Eigenvalues WRT principle components? |
|
Definition
| Eigenvalues greater that 1.0 are misleadding |
|
|
Term
|
Definition
| Combinations of observable variables (aka measures, tests, indicators, observables) |
|
|
Term
| List 3 ways linear combinations are used in factor analysis. |
|
Definition
1) Effect indicators
2) Components
3) Causal indicators |
|
|
Term
| What are "effect indicators"? |
|
Definition
Linear combinations in which the observables are the results (effects, outcomes) of the factor
-Observables = DV --> Contain error
-Factor = IV --> Error free |
|
|
Term
|
Definition
Simple linear combinations of observables (therefore are observables in their own right)
Knowing one pair implies knowing the other through a simple transformation (if only one term in a given pair is unknown, theother pair is indeterminate) |
|
|
Term
| What are "causal indicators"? |
|
Definition
Linear combinations in which the factor depends upon the observables
--> factor becomes the criterion in a regression analysis sense
-Identifies error soley with the factor, however BC the observables can be observables, observables may also contain error
(e.g. people have high socioeconomic status BC they are wealthy and well-educated; they do not become wealthy or well-educated BC they are of high socioeconomic status |
|
|
Term
| Exploratory factor analysis defines factors in terms of ____. |
|
Definition
Best fit
--> "Most variance accounted for"
It is data-driven |
|
|
Term
| A rotated factor is a ____. |
|
Definition
linear combination of the initial factors
Only divide up variance more EQUALLY --> They will explain EXACTLY THE SAME total variance as the initial factors, even though the variabhles will related to the rotated factors differently than relate to the initial factors |
|
|
Term
| Factors are defined directly in ____. |
|
Definition
| confirmatory factor analysis |
|
|
Term
| What is one similarity and one difference b/n multiple correlation and factor analysis? |
|
Definition
They relate a linear combination of variables to a criterion
Different: in MR, the predictors and criterion are separate entities; in FA, the predictors (factors) are at least partially defined by the criteria (variables) |
|
|
Term
| If a subtest correlates substantially with other measures given the same or similar names, it should possess ____ validity. |
|
Definition
|
|
Term
| What is MOST important for FA? |
|
Definition
|
|
Term
| What are criteria for defining variables? |
|
Definition
1) More variables in a set that a given variable correlates with and hte higher the general level of correlation, the better
--> Higher the level of intercorrelation, the easier it is to determine patterning of correlations
2) Variables should be reliable
--> problem of attenuation ue to unreliability BC affects measures of relationship like r
3) Analysis should contain variables with known properties called marker variables
4) Large sample sizes should be used to ensure that groupings are not simply effects of sampling error |
|
|
Term
| All the variance is considered ____ in component models. They estimate the unique variance to be ___ for every variable. |
|
Definition
|
|
Term
| ___ is highly desirable in component analysis. |
|
Definition
|
|
Term
| Weightings are called ____, which may be viewed as ____. |
|
Definition
| pattern (b) elements; may be viewe as regression weights |
|
|
Term
| Each variable has ___ in factor analysis. |
|
Definition
|
|
Term
| What is the average communality? |
|
Definition
| Proportion of variance accounted for |
|
|
Term
| What is a limitation to communality? |
|
Definition
| The communality of each variable and therefore the proportion of variance accounted for generally increases as the number of factors increases for the same reason that all multiple correlations are biased with respect to number of predictors (EVEN WHEN THE ADDITIONAL FACTORS ARE MEANINGLESS) |
|
|
Term
| What is the multiple correlation for components and common factors? |
|
Definition
Components: 1.0 BC components are linear combinations of the other variables
Common factors: LESS THAN 1.0 BC they are broader than the variables that define them
**Multiple Correlations reflect the proportions of variance in the factors that are explained by the variables |
|
|
Term
| True/False. Adding a variable that is highly correlated with other variables in a grouping does NOT add appreciably to the factor definition. |
|
Definition
|
|
Term
| What should you do if the factors have low multiple correlations? |
|
Definition
| Add additional variables. |
|
|
Term
| What are 2 things that are UNAFFECTED by rotation? |
|
Definition
1) Individual h2 values
2) Overall indices of fit (e.g. proportion of variance accounted for) |
|
|
Term
| Name 4 general effects of alternative ways of defining data. |
|
Definition
1) Standardizing measures eliminates the effects of differences in the unit of measurement
2) Expressing the measures as deviation scores eliminates the effects of differences in location of the variables (means) from the analysis but allows differences in variance to play a role
3) Expressing the measures as raw scores allows difference in both location and variance to affect the results
4) Dividing raw scores by the variance of the particular variable over subjects would eliminate differences in variance but allow differences in location to remain |
|
|
Term
|
Definition
|
|
Term
| Initial solutions in factor analysis are nearly always ___. |
|
Definition
|
|
Term
| If deviation scores are used in the analysis INSTEAD of standardized scores, _______. If raw scores are used, then _____. |
|
Definition
covariances are estimated instead of correlations
mean sums of products are estimated |
|
|
Term
| What is contained in the diagonal elements of R? |
|
Definition
communality estimates
Component analysis: 1.0
Common factor analysis: < 1.0 |
|
|
Term
| True/False. Communality estimates ARE NOT THE SAME as communalities. |
|
Definition
True.
Communality estimates: rij in R
Communalities: h2 |
|
|
Term
| What does the beta weight for a predictor equal? |
|
Definition
| its correlation with the criterion (validity) when the predictors are UNCORRELATED |
|
|
Term
| The number of factors to be retained is suggested by ____. |
|
Definition
the first set of structure elements
--> if highly correlated with 1st factor, may not need other factors
--> if moderately correlated, may need several factors
--> if correlations are near 0, may not be any factors in the data |
|
|
Term
| What is the method of successive extraction used for? |
|
Definition
| the obtaining the uncorrelated (orthogonal) factors of exploratory factor analysis |
|
|
Term
| What is the method of extracting simultaneous factors? |
|
Definition
>1 factor may be extracted at each step, but the factors extracted at that step will usually be correlated
May be orthogonal or oblique
Characteristic of confirmatory approaches |
|
|
Term
|
Definition
1) A line segment having both:
(a) direction (orientation) and
(b) length (magnitude) in geometry
2) A set of numbers, such as test scores, in algebra |
|
|
Term
| The more different the vectors, the ___ the angle and the ___ the cosine. |
|
Definition
larger (up to 90o) angle
smaller the cosine |
|
|
Term
| Two unrelated vectors form ___ angle. Their cosine is ___. |
|
Definition
|
|
Term
| The cosine of the angle between the two vectors viewed geometrically is totally equivalent to their ______ if they are defined algebraically as 2 sets of numbers. |
|
Definition
|
|
Term
| The method of successive extraction is the basis for obtaining the ______. |
|
Definition
| Uncorrelated (orthogonal) factors of exploratory factor analysis. |
|
|
Term
| Angles between 90o and 180o have ___ and therefore ____. |
|
Definition
negative cosines
negative correlations |
|
|
Term
| How do you calculate the length of the common factor? |
|
Definition
| Square root of its communality |
|
|
Term
| When vectors are all of unit length, _____. When they are different lengths less than 1 they are _____. |
|
Definition
Component analysis
Common factor analysis |
|
|
Term
| If there are more than two factors, the vectors (would/would not) have unit lenght because they would also project into other dimensions. |
|
Definition
|
|
Term
| What is a common goal of factor analysis? |
|
Definition
| To find the minimum rank of a matrix of correlations by a suitable choice of diagnoal elements (communality estimates) |
|
|
Term
| If the factor model fit perfectly, it would partition the total variance of any variable (1.0 in standard form) into 3 terms: __, __, and ___. |
|
Definition
Variance due to measurement error +
Specific variance +
Common variance |
|
|
Term
| What is the measurement error of variable Xi? |
|
Definition
Squared standard error of measurement from reliability theory
= 1 - coefficient alpha |
|
|
Term
| If the reliability of variable X1 is .80, then the measurement error variance would be ___ and the systematic variance is ___. |
|
Definition
|
|
Term
| What is systematic variance? |
|
Definition
| Systematic variance is the sum of common variance and specific variance |
|
|
Term
| What is specific variance? |
|
Definition
| Specific variance is nonrandom but CANNOT be explaind by relationships with other varaibles in the model |
|
|
Term
What is unique variance?
What is a standardized variable's unique variance? |
|
Definition
h2 + u2
Specific variance + error variance
1-h2 |
|
|
Term
| What is the difference b/n Factor analysis and Component analysis WRT Components of variance? |
|
Definition
Factor: partitioning variables into common and unique variance
Component analysis: explains common variance, with linear combinations of the variables
--> Unique variance becomes residual not explained by obtained factors |
|
|
Term
| Salients are generally referred to as ____. |
|
Definition
|
|
Term
| What are the 7 types of factors? |
|
Definition
1) General: all measures are salients
2) Group: Some, but not all are salients
3) Common: General and Group are called common factors BC what they measure is common to more than one variable
4) Unipolar: when all salients have the same sign
5) Bipolar: Some salients are positive, some are negative
6) Singlet: Factor with only one salient (e.g. masculinity/feminity on MMPI)
7) Null: No salients |
|
|
Term
| Name 3 general approaches to Condensing Variance in Exploratory Factor Analysis. |
|
Definition
1) Factors are defined before analyzing the data.
--> Centroids: when a factor (usually a component) is the equally weighted sum of the variables
2) Optimize some property of the sample data.
--> PrC maximizes the amt of variance that can possibly be explained
3) Use sample data to predict the results in a population
--> Maximum likelihood EFA stresses statistical inference, rather than assuming an indefinitely large sample |
|
|
Term
| What are 4 ways you use a correlation matrix WRT factoring? |
|
Definition
1) If correlations are high enough to warrant factoring
2) Common groupings in the data
3) Signs and sizes within groupings
--> Sizes of the correlations defines how strongly the factor (grouping) is defined
4) Correlation b/n groupings to decide about the type of rotation to use
--> If low (<.3), use orthogonal solution, however, if not, use oblique
4) |
|
|
Term
| Pattern elements are to ____ as structure elements are to ____. |
|
Definition
Regression weights
Correlations |
|
|
Term
| h2 in orthogonal solution equals ___. |
|
Definition
| sum of squared structure (or pattern) elements |
|
|
Term
| Are absolute residual correlations bigger in a Component Solution or a Common Factor Solution? |
|
Definition
|
|
Term
| In PrC, what are Eigenvalues? |
|
Definition
| Proportion of variance accounted for by each PrC |
|
|
Term
| PrC must account for _______ than the number of centroids. |
|
Definition
|
|
Term
| Each ___ defines the total variance explained by the PrC. |
|
Definition
|
|
Term
| All component Eigenvalues are either ___ or ____ because they are interpretable as variances. |
|
Definition
|
|
Term
| The number of ___ represents the # of PrC factors neeed to explain all the variance in a correlation matrix. |
|
Definition
| positive nonzero eigenvalues |
|
|
Term
| The sum of the diagonal elements in R are called the ____. This equals ___. |
|
Definition
trace
the sum of the eigenvalues |
|
|
Term
| The product of all eigenvalues in R equals its ___. |
|
Definition
Determinant
used in FA as a multivariate measure of variance |
|
|
Term
| When each variable in the population is perfectly correlated with every other variable, each element in R is ___. |
|
Definition
|
|
Term
| Measurements are reliable to the extent that they are ____. |
|
Definition
|
|
Term
| Any random influence which causes different measurements of the same variable to vary is a source of ____. |
|
Definition
|
|
Term
| A long test with a _____ among items is ALWAYS a highly reliable test. |
|
Definition
| positive average correlation |
|
|
Term
| Correction for Attenuation |
|
Definition
| Used to examine the correlation between two variables as the reliability of each is changed to a designated level and not simply made perfect as was previously assumed |
|
|
Term
| Increase test reliability by using: |
|
Definition
1) more items
2) using Spearman-Brown prophecy formula |
|
|
Term
| What does variation within a test do? |
|
Definition
| lowers average correlation among items, but average correlation is still sufficient to estimate reliability |
|
|
Term
| What are 9 sources of variation in a test? |
|
Definition
1. Item sampling: Individuals have probability of correctly answering each item, depending on their a) true score & b) difficulty of the item
2. More items less error
3. Error due to sampling is predicable from average correlation
4. Coefficient alpha is appropriate measure of relaitbility for any type of item
5. Guessing: lowers correlations b/n items and overall test reliability
6. Accidentally marking one answer instead of another
7. Misreading a question, due to confusing wording
8. Fatigue on long tests
9. Random (not systematic) grader errors |
|
|
Term
|
Definition
Individuals have probability of correctly answering each item, depending on their
a) true score &
b) difficulty of the item |
|
|
Term
| Three sources of error that cause domain-sampling model to overestimate actual correlation b/n forms: |
|
Definition
1. Systematic differences in content of the 2 tests
a. Items composed, not randomly sampled
2. Systematic Effects: from subjectivity of scoring, due to different standards among judges
3. Change in the subject in the attribute being measured
a. More important with mood-related measures than ability mreasures |
|
|
Term
| Alternative form correlations may be higher than ____. |
|
Definition
| within-test estimates of reliability |
|
|
Term
| Defined Internal Consistency. |
|
Definition
| Estimates of reliability based on the average correlation among items within a test |
|
|
Term
| What is Coefficient alpha? |
|
Definition
represents both number of items and their average correlation
1. Usually provides good estimate of reliability BC sampling of content is usually major sources of measurement error for static constructs
2. Sets upper limit for reliability of tests constructed in terms of domain-sampling model based on observed correlations
a. If a is low: testis too short or items do not have much in common
b. Measurement problem choose different items |
|
|
Term
| Alternative Forms: If correlation b/n AF is markedly lower than coefficient a (e.g. .20 or more), measurement error is present (due to 3 sources of error): |
|
Definition
1) systematic differences in content,
2) subjectivity of scoring,
3) variation in trait over time |
|
|
Term
| i. If average correltion within 2 test formats is substantial (e.g. .20), but average cross correlation b/n items on two forms is low (e.g. .10), then _____. |
|
Definition
| tests reliably differ in content and thus measure different traits |
|
|
Term
| If correlation b/n tests over a 2-wk interval is less than correlation for tests taken on same day, ____. |
|
Definition
| scoring is probably reliable but trait is temporally unstable (desired) |
|
|
Term
| What does unreliability of scoring mean? |
|
Definition
trait does not exist in a manner consistent over judges;
Raters are inconsistent |
|
|
Term
| WRT Alternative forms, Coefficient a is good estimate of reliability on measures that ____. |
|
Definition
| domain of content is easily specified and ppl are stable over time (e.g. aptitude and achievement tests) |
|
|
Term
| If alternative form cannot be constructed, then ___. |
|
Definition
Domain of content cannot be defined;
Cannot accurately communicate what is being measured |
|
|
Term
| What are 4 other estimates of reliability? |
|
Definition
i. Coefficient alpha
ii. Correlations
iii. Split-half approach
iv. Test-Retest |
|
|
Term
| What is the split-half approach WRT reliability? What is a problem with this approach? |
|
Definition
test items divided in half; scores on both are correlated
Misleading estimates when items ordered in terms of difficulty |
|
|
Term
| What is Test-Retest WRT reliability? What are the problems of this approach? |
|
Definition
Same people are retested by the same test after a period of time
- Used instead of alternative-from method to determine reliability
- Problems: (lead to spuriously high correlations b/n tests) BC:
a. Remember answers
b. Repeated work habits
c. Similar guesses
d. Only partly dependent on inter-item correlations reliability of test is function of average correlation among items
--> Makes it possible to have a measure with no internal consistency stable over time (High retest correlation with low internal consistency) |
|
|
Term
| ____ reduces validity of decisions when temporally unstable measure is used to make practical decisions about people and work. |
|
Definition
|
|
Term
| Two types of reliability coefficients to be computed and reported: |
|
Definition
1) Coefficient a: all forms of a test
2) Correlations among alternative forms (reveal sources of measurement error not detectable by coefficient
2) |
|
|
Term
| What are 3 uses of the reliability coefficient? |
|
Definition
i. Correction for attenuation
ii. Confidence Intervals
iii. Effect of Dispersion on Reliability |
|
|
Term
| WRT Correction for attenuation, ii. If relevant measures are modestly reliable, observed correlations will ___ correlations among traits. |
|
Definition
|
|
Term
| Correction for attenuation formula: |
|
Definition
r’xy = rxy / (√(r’ xxr’yy) / (√r xxryy)
r’xy = estimated correlation b/n variables x and y if their reliabilities are changed
r’xx = changed reliability for variable x
rxx = obtained reliability for variable x |
|
|
Term
| WRT Corrections for Attenuation, the Reliability coefficient: |
|
Definition
| Estimates extent to which obtained correlations b/n variables are attenuated by measurement error |
|
|
Term
| Corrected correlations are seldom dramatically different from observed correlations, therefore (3 things): |
|
Definition
1. Low correlations have nothing to do with reliability
2. Increase # items to make a test more reliable
3. Tests usually correlate poorly BC measure different things NOT due to measurement error |
|
|
Term
| What is the formula for confidence intervals for obtained score? |
|
Definition
|
|
Term
| CI should be obtained around a person’s ______ not their _____. |
|
Definition
| true score, not their test scores |
|
|
Term
| WRT CIs, high obtained scores are biased ___, low obtained scores biased _____. |
|
Definition
|
|
Term
| What are Unbiased scores? |
|
Definition
Average scores people would obtain if they were administered all possible tests with a aonstand # of items from a domain
T’ = rxxx
X = deviation score
Rxx = reliability
T’ = estimated true scores |
|
|
Term
| CI should be centered around the ___. |
|
Definition
| estimated (regressed) observed score |
|
|
Term
| WRT CIs, Less error in estimating ____ from an ___ than the converse. |
|
Definition
| true scores; observed score |
|
|
Term
| _____ may be used to measure change over time. |
|
Definition
|
|
Term
| Standard error of measurement = |
|
Definition
-estimated standard deviation of obtained scores if any individual is given a large # of tests from a domain
-stable across populations which differ in variability BC the changes in RC and SD are partially offsetting |
|
|
Term
| Reliability coefficient is directly related to _____ for any sample. |
|
Definition
|
|
Term
| Reliability coefficient will be ___ in more variable samples. |
|
Definition
|
|
Term
|
Definition
| ratio of true-score variance to obtained-score variance |
|
|
Term
| 5 ways to reduce measurement error |
|
Definition
1. Writing items clearly
2. Making test instructions easily understood
3. Adhering closely to the prescribed conditions for administering an instrument
4. Making subjective scoring rules as explicit as possible
5. Training raters to do their jobs |
|
|
Term
| If # items know, use ______ to estimate how much reliability will increase if the # of items were increased by any factor k : |
|
Definition
Spearman-Brown prophecy
Rkk = kr11 / (1+k-1)r11
k = # items on shorter test / # items on longer test
rkk = estimated reliability of the shortened test
r11 = reliability of longer test |
|
|
Term
| Reliability approaches ____ as test length increases, so long as average correlation of items in a domain is _____. |
|
Definition
|
|
Term
| What formula would you use to determine how many times you have to increase the length of a test? |
|
Definition
K = rkk(1-r11) / r11 (1-rkk)
Rkk = desired reliability
R11 = reliability of existing test
K = number of times test would have to be lengthened to obtain a reliability of rkk |
|
|
Term
| If average correlation among items in a domain is _____, the correlations b/n samples of items will be ____ and # of items needed to achieve acceptable reliability will be ____. |
|
Definition
| very low; small; very large |
|
|
Term
| If sig correlations are found, ____ will estimate how much the correlations will increase when reliabilities of measures are increased. |
|
Definition
| corrections for attenuation |
|
|
Term
| 4 Limitations on the Reliability Coefficient’s Utility |
|
Definition
i. Relibility estimates based upon observed correlations affected by similarities of the item distributions
ii. Look at item’s distributional properties (e.g. p values) and correlation with total test score to avoid eliminating items spuriously
iii. Hetergeneities in item distributions undersestimate worth of an item
iv. Coefficient a = lower bound of population reliability (true variance: total variance) |
|
|
Term
| If tests had an acceptable reliability, but they were mutually uncorrelated, Coefficient a would be ___. |
|
Definition
| 0; however linear combination NOT necessarily 0 |
|
|
Term
| Reliability of variable in linear combination = |
|
Definition
true-score variance/obtained variance
True-score variance = Obtained score variance x reliability |
|
|
Term
| What do negative elements in a linear combination effect? What do they not effect? |
|
Definition
ii. Does not affect logic BC still 3 items
iii. DOES affect Denominator (denominator is variance of linear combination) Minus sign reverses signs of correlations
iv. IF S3 correlated negatively with X1 and X2 minus sign in the linear combination would increase denominator over what it would have been had it been added |
|
|
Term
| Larger variance of linear combination --> |
|
Definition
|
|
Term
| WRT linear combinations, Reliability of a sum (CAN/CANNOT) be estimated by coefficient alpha. |
|
Definition
|
|
Term
Within an ANOVA design:
1. True variance is estimated from:
2. Error variance is estimated from:
3. Ratio is identical to: |
|
Definition
1. Difference in MSs
2. MS within groups
3. Alpha |
|
|
Term
| What does Generalizability Theory allow you to do? |
|
Definition
| To evaluate both random sampling error that arises within a domain and systematic error that might arise BC different judges evaluate different attributes |
|
|
Term
| Tests may be evaluated by standards of (3 things) |
|
Definition
1. Content
2. Construct
3. Predictive validity |
|
|
Term
| How are Predictive and construct validation similar? |
|
Definition
| Both involve correlating measure with a criterion |
|
|
Term
| What is content validation? |
|
Definition
Begin with domain of content that defines what is to be measured (who the test is applicable, test plan that defines how it is to be measured)
Administer --> item-analysis to define each item’s difficulty; discrimination --> how highly each relates to total test score |
|
|
Term
| What is Construct Validation? |
|
Definition
• Begin with hypothesis that implies domain of content
Scales content should be homogeneous; difficulty BC methods used to infer trait are heterogeneous |
|
|
Term
| Difference b/n one-time use and repeated use tests: |
|
Definition
1. Greater potential legal scrutiny
2. Various non-psychometric considerations (greater test security)
3. Several cycles of refinement before use |
|
|
Term
| Achievement tests: created through |
|
Definition
|
|
Term
| Ability tests: created through |
|
Definition
|
|
Term
Domain of Content and Test Plan should describe (6 things):
**Should be done BEFORE creating test
**Define an appropriate domain of content is ESSENTIAL |
|
Definition
1. Types of items to be employed with examples
2. Approximate # of items to be employed in each section and subsection
3. How long the test will take to administer
4. How it will be administered
5. How it will be scored
6. Types of norms or other referencing that will be obtained |
|
|
Term
|
Definition
| measure divergent thought and creativity |
|
|
Term
| Construct good items and have clarity by (4 things): |
|
Definition
1. Phrasing of items
2. Relate to domain
3. Points the knowledgeable student toward what is demanded
4. Avoid trivial Qs |
|
|
Term
| Thorndike’s construction for ALL items: |
|
Definition
1. Make complexity of items appropriate to students’ level
2. Define task & directions as clearly as possible
3. Inform students about grading standards
4. Write itmes simply & straightforward
5. Know mental processes to be used
6. Use novel material or organization to prevent reproduction of lectures
7. Vary complexity and difficulty of items improves ability to discriminate
8. Make questions Independent
9. Avoid negatively phrased items
10. Never use double negatives |
|
|
Term
| What are 6 recommendations for short answer tests? |
|
Definition
1. Omit only key words
2. Do NOT leave too many blanks
3. Put blanks near end of Q
4. Avoid specific determiners such as “all” and “none”
5. Avoide ambiguous determiners such as “frequently” and “sometimes”
6. Have each item express a single ideal |
|
|
Term
| What are 10 recommendations for MC items? |
|
Definition
1. Be sure stem clearly formulates problem
2. Include as must of item’s content in stem as possible
3. Include only necessary material
4. Use novel material and examples
5. Be sure distracters are plausible
6. Use “none of the above” or “all of the above” sparingly
7. Make alternatives of approximately equal length & parallel grammatical construction
8. Randomize location of correct alternative
9. Make sure each alternative agrees with stem
10. Try to eliminate any factor that makes the correct alternative stand out
11. Formulate incorrect alternatives so that they detect common ways in which students may be misinformed |
|
|
Term
| If you have low coefficient alpha, what are 3 things you can do? |
|
Definition
a. Construct more items
b. Get responses from larger group of subjects
c. Perform complete item analysis |
|
|
Term
| What does the Spearman-Brown prophecy indicate? |
|
Definition
| indicates adding items to increase reliability obeys a law of diminishing returns; difficult to improve reliability of a test that is already reliable substantially by adding more items |
|
|
Term
| WRT Test length, what are 3 things to consider? |
|
Definition
1. Reflect time available
2. Desired reliability
3. More variable the population, smaller # of items needed to achieve a given reliability |
|
|
Term
| What are 3 things to consider WRT pilot sample of subjects? |
|
Definition
1. Pilot sample needs to be similar to target population
2. Conditions of pilot study should resemble eventual use
3. Rule: at least 200 normative subjects |
|
|
Term
| WRT item analysis, content validity relies on _____ grounds. |
|
Definition
| on rational, not empirical |
|
|
Term
|
Definition
| statistical data regarding how subjects responded to each item and how each item relates to overall performance |
|
|
Term
|
Definition
a. Be suspicious of items if distracter is chosen more often than correct alternative instructions or item are misleading
b. Distracters hardly ever chose are too transparently incorrect
c. Proportion choosing correct alternative or item p value is the classical index of item difficulty (can also apply to sentiments)
i. Items with extreme p values should be excluded BC do NOT discriminate among individuals |
|
|
Term
| What is the classical index of item difficulty? |
|
Definition
p-value
Items with extreme p-values should be excluded BC do NOT discriminate among individuals |
|
|
Term
| Driver’s test: example of test designed for _____. |
|
Definition
|
|
Term
| Tests designed for mastery learning have ______ when instruction has the desired effect |
|
Definition
|
|
Term
General achievement tests: _____ BC difficult to teach enough in a short period of time.
Classroom tests: _____. |
|
Definition
temporally stable
temporally unstable |
|
|
Term
| Content-validated tests: (DO/DO NOT) need to correlate with any other measure nor have very high internal consistency. |
|
Definition
|
|
Term
|
Definition
1. Describe how each item relates to overall test performance
2. Provide discrimination among discrimination indices |
|
|
Term
|
Definition
|
|
Term
| Less ambiguous, NOT extremely difficult make individual scores on final test ___. |
|
Definition
|
|
Term
| Item response theory uses ___ and correlates items with this estimate. |
|
Definition
| a estimate of trait magnitude (theta θ) |
|
|
Term
| What are discrimination indices (3 things)? |
|
Definition
a. Covariance b/n an item and total score
b. Average correlation b/n a given item and all other items
c. Proportion of people passing the item in the top half of the class – the proportion of people passing the item in the bottom half of the class |
|
|
Term
| WRT IA, negative items may mean (3 things): |
|
Definition
1. Bad wording
2. Sampling error
3. Miskeying |
|
|
Term
| Content validation uses ___ as a final decision-making about whether to keep an item. |
|
Definition
| Use human judgment as final decision to reject or include an item |
|
|
Term
| Construct and predictive validity use __ to make final decision about whether to keep an item. |
|
Definition
|
|
Term
| WRT item selection, ___ is the primary criterion. |
|
Definition
Discrimination index (E.g. corrected item total r)
-a. Items with high item-total r values have more variance relating to what the items have in common & add more to test’s reliability than low values |
|
|
Term
| ____ are used as a secondary criterion for item inclusion. |
|
Definition
P values
-Corrected and uncorrected item-total r values are biased toward items with intermediate p values |
|
|
Term
| If correlation is low, but intended reliability is high, what can you do? |
|
Definition
-Add sets of 5 – 10 items until desired reliability is obtained
-Key to successful item selection is re-administration of test in a new sample |
|
|
Term
| # items needed to be added depends on: |
|
Definition
1. item-total r values
2. the reliability of the 1st set of items |
|
|
Term
| WRT adding items, Obtained reliabilities are ____ than predicted reliabilities. |
|
Definition
|
|
Term
|
Definition
a. Items may be from a poorly defined domain where correlations among items are uniformly low
i. Reliability grows slowly as the # of items increases
b. Items may be factorially complex (multidimensional) so that clusters of items have relatively ihg correlations with one another, but low correlations with members of other clusers
i. Range of inter-item correlations will be large
c. Some items may have high correlations with one another, but other items may have near zero correlation with all other items adding items will do no good in this case
i. Detect by noticing marked decrease in item-total correlations at some point |
|
|
Term
| Most likely to find a ____ of items in the initial set that correlate well with total score, _____ that do not correlate at all, _____ in the middle. |
|
Definition
small to modest #
large number
moderate block |
|
|
Term
|
Definition
Statistical data that provide a frame of reference to interpret an individual’s scores relative to scores of others
1. Less essential when measure is intended for use in group research, rather than individual decisions
2. Expressed in percentiles (percentage of persons in the normative sample at or below a particular score) and z-scores
3. MORE meaningful to think of grades as reflecting JUDGMENTS based on the instructor’s conception of the various categories
4. Scores CRITERION-referenced when implications for relevant bxs (e.g. score of 50 on admission test .75 chance of completing the program)
5. Scores DOMAIN-referenced when relate to the domain being measured |
|
|
Term
| True/False. Measures designed through content validation need NOT correlate with any external criterion to be valid |
|
Definition
|
|
Term
| Tests developed for construct validity CANNOT be developed without a _____. |
|
Definition
| theory that dictates the properties of that measure |
|
|
Term
Personality traits involve _____.
Abilities tests involve ____. |
|
Definition
|
|
Term
| Measurement properties for content homogeneity should include: |
|
Definition
1. Coefficient a reliability
2. Temporal stability
3. Homogeneity of content: content is homogeneous when it has little measurement error (high coefficient a) and measures ONLY one attribute
a. Implies measures are unidimensional and unifactorial |
|
|
Term
| How do you know when content is homogeneous? |
|
Definition
When it has little measurement error (high coefficient a) and measures ONLY one attribute
-Implies measures are unidimensional and unifactorial |
|
|
Term
| Heterogeneous Content if ___. |
|
Definition
| if average correlation among items and average item-total correlation is low |
|
|
Term
| WRT content validation, you want: |
|
Definition
1. Average correlation with total scores is high
2. Spread of correlations about this average is small |
|
|
Term
| WRT Factor Analysis, correlations and variance among _____ items are usually higher than _____. |
|
Definition
multicategory (e.g. Likert)
dichotomous items |
|
|
Term
| Mean and standard deviation for multicategory items provides info about ____ and ______. |
|
Definition
Item difficulty for judgements
Endorsement level for sentiments |
|
|
Term
| ______ is preferred discrimination index for content-validated tests. |
|
Definition
| Corrected item-total correlation |
|
|
Term
| To improve item distributional characteristiscs(WRT IA and Selection): |
|
Definition
a. Use multicategory format (Likert scale)
b. Change modifier (eg. Dropping “very”) |
|
|
Term
| What are 2 problems iwth Empirical (criterion-oriented) approaches to test construction? |
|
Definition
i. Difficult to determine which items fall on which scales
ii. Items may correlate with the criterion because they correlate with each other |
|
|
Term
| A construct is less likely to require norms than a content-validated measure BC of ___. |
|
Definition
Differences in their probable used
--> Items derived from content validation involve ad hoc requirements that are of pragmatic, rather than theoretical import |
|
|
Term
|
Definition
Interested in discovering factors by exploration
i. Defines factors in terms of “best fit” “most variance accounted for”
ii. Step-wise: 1st reduce items into # of factors, next rotate |
|
|
Term
| Name and describe Three uses of Linear combinations: |
|
Definition
1. Effect Indicators: linear combinations in which the observables (DVs) are the results of the factor (IV)
a. observables (DVs) = error free
b. factor (IV) = contain error
2. Components: Simple linear combinations of observables and observables in their own right
a. IF one term is known, the other is indeterminate
3. Causal Indicators: Linear combinations in which the factor depends upon the observables
a. Factor becomes the criterion (SES & wealth)
b. Identifies error solely with the factor
c. Observables and predictors can contain error |
|
|
Term
| Common factor analysis uses which type of linear combination? |
|
Definition
|
|
Term
| S model is equivilant to S curve when ___. |
|
Definition
|
|
Term
| WRT correlations and validity, when should it be high, and when should it be low? |
|
Definition
| Correlation should be low if you are measuring two different things; high if you are measuring similar things |
|
|
Term
| What is the formula for Linear Combination of Variables? |
|
Definition
| simple linear regression: y =b0 + b (x) e |
|
|
Term
| WRT linear combinations, what are components? |
|
Definition
| Components : linear combinations of observables and therefore observables in their own right |
|
|
Term
| Can you ever know a true score? |
|
Definition
| No, because it is a theoretical idea. |
|
|
Term
| WRT FA, if it is reliabile, then ____ and ____. |
|
Definition
-Reliabile --> DECREASE MAGNITUDE
- ATTENUATION: R XY = RXY / SQRT (R XX R YY )
- AS RELIABILTIY GOES DOWN, MORE ATTENUATION |
|
|
Term
| Numbers in the matrix are ___. |
|
Definition
| factor loadings: correlation of item with unknown factor!! |
|
|
Term
| variance can be decomponsed into ___. |
|
Definition
| Variance due to measurement error + specific variance + common variance |
|
|
Term
|
Definition
| Measurement error + Specific variances = u2 |
|
|
Term
|
Definition
h2 aka communality
- means variance that is shared across the factors (BC may share variance across different constructs)
- reflects correlation with factors |
|
|
Term
| Factor analysis is frequently described as partitioning variables into ___. |
|
Definition
| common and unique varianace |
|
|
Term
| Principle Components is a __ procedure which ____ thereby _____. |
|
Definition
- Variance MAXIMIZING PROCEDURE
- Analyzes all the variance: h2 + u2
- MAXIMIZES EXPLAINED VARIANCE BC OF THE DIAGNOAL CORRELATION MATRIX (1.0 IS IN THE DIAGNOAL
- USES ALL THE VARIANCE); |
|
|
Term
| ___________ maximizes the variance. |
|
Definition
|
|
Term
| Variance of a standard score = __. |
|
Definition
|
|
Term
| Communality is the sum of ______. |
|
Definition
| sum of the squared factor loadings across the factors |
|
|
Term
|
Definition
| sum of squared factor loadings across the factors |
|
|
Term
| There will ___eigenvalues to number of items. |
|
Definition
|
|
Term
| Principle axis analyzes ___. |
|
Definition
| Only variance shared across the items |
|
|
Term
| Factor Transformation Matrix is a function of ___ and ___. |
|
Definition
|
|
Term
| Initial eigenvalues are based on ___ |
|
Definition
| unrotated matrix (initial item values) are used in extraction |
|
|
Term
|
Definition
| Total Extraction sums of squared loadings /# components |
|
|
Term
|
Definition
|
|
Term
| What are 3 approaches to condensation? |
|
Definition
o Centroid analysis: Defining a factor’s content in advance as sum of variables in analysis
o Principle Components & Principle axis Analysis: Maximizing a property of the sample data by accounting for the most possible variance
o Maximum likelihood Analysis: Estimating population parameters |
|
|
Term
| Eigenvalues that exceed 1.0 may be ___. |
|
Definition
|
|
Term
|
Definition
|
|
Term
| What is a varimax rotation? |
|
Definition
Makes vectors interpretable by cleaning up factors/constructs
-->High/low on different components |
|
|
Term
| Varimax assumes the items are ___. |
|
Definition
|
|
Term
| Validation is an iterative process, therefore you should ____. |
|
Definition
| re-validate on a separate sample |
|
|
Term
| Pattern matrix = rel of ___. |
|
Definition
item and component holding constant all other items
-like standardized regression coefficient BC partials out contribution of other variables |
|
|
Term
| Structure is like a ____. |
|
Definition
| regular bivariate correlation. |
|
|
Term
Varimax assumes factors are ___.
Oblique rotation: assumes factors are ____. |
|
Definition
|
|
Term
| Stability of a factor solution is better when ____. |
|
Definition
| done across groups and industries |
|
|
Term
| You don't have convergence when _____. If this happens, you know ____. |
|
Definition
there are too many iterations
something is wrong with the model (e.g. bad items, small sample size, etc.) |
|
|
Term
| # variables you have in the variance/covariance matrix is given by what formula? |
|
Definition
|
|
Term
| How is Coefficeint alpha calculated? |
|
Definition
proporetion of true score/observed score
True score/error = observed score!! |
|
|
Term
| 4 reasons for differences in intercepts |
|
Definition
• Possible reasons for differences in intercepts:
• bias in test
• Bias in criterion
• Reliability of test
• Omitted variables (when you omit a relevant 3rd or 4th variable SPECIFICATION ERROR!!) |
|
|
Term
| Diffrence in validity coefficient (rxy) may be cited as evidence of ___. |
|
Definition
|
|
Term
|
Definition
| DIFFERENTIAL VALIDITY means the correlation of the selection tool and performance will vary across groups; if steeper slope, better --? Means validitiy foefficient is different between groups |
|
|
Term
| Intercept bias is heavily influenced by ____. |
|
Definition
|
|
Term
| What are sources of bias? |
|
Definition
1. Real mean differences in attributes
2. Differences are function of the test
3. Content of test is familiear to some groups, but not others
4. Method of Presentation (e.g. written v. video, etc.)
5. Interaction with administrator |
|
|
Term
| What are remedies for bias? |
|
Definition
1. Change content of test
2. Employ multiple methods of assessment
3. Change method |
|
|
Term
| What are 3 ethical positions WRT bias? |
|
Definition
o Unqualified Individualism: use tests to select MOST qualified individuals that can be found Indifferent to race or gender of applicants
o Quotas: using non-psychometrically; explicitly recognize race and gender differences (e.g. if 20% African Americans in the population, then select 20% African Americans for company)
o Qualified individualism: compromise b/n unqualified individualism and quotas |
|
|
Term
| When is adverse impact legal? |
|
Definition
o 4/5s of selection ratio (20 whites hired/100 white applicants)
o IF there is an adverse impact, validation study may be necessary to prove the test is jjob-related; IF SO, ADVERSE IMPACT IS LEGAL |
|
|
Term
What are times power tests designed to measure?
Who uses them? |
|
Definition
measure designed to assess power but administered with a time limit, normally imposed for administrative pruposes (e.g. classroom availability)
o THIS IS TYPICALLY WHAT IS USED AT UNIVERSITY – LEVEL (combination of speed and power, but MAINLY POWER) |
|
|
