Nominal

Ordinal: ranking, no indication of magnitude

Interval: equal intervals, unit is arbitrary

Ration: absolute measure, equal intervals (absolute 0)

Reliability- consistency and stability

Validity- measurement fits purpose

[image]

[image]

[image]

Y = a + b1X1 + b2X2 + b3X3….

SEM=SD*√(1-r_xy)

if reliability increases then standard error of measurement decreases

-studies mechanism underlying effect

-looks at experimental design & proximal outcomes

-eg theory of reasoned action:attitudes-intention-behavior

when the mediator has a different effect on the DV depending on the level of a moderator

-mediation differs for subgroups of participants

• realism/utility (can you make factual statements abou unobservable things such as theories)
• scientific reasoning
• demarcation of falsification (falsifiable=scientific)
•  socio-political paradigm (academic traditions, scientific fads, funding, rewards/incentives)

-nomological network

-specificity

-level of analysis

-situation/context

-over time

• Alpha change: difference in presense of construct between time 1 and time 2 (pre/post) 
• Beta change:change in scale measurement(interpretation of performance, score might be good before taking test and bad after)
• Gamma change: change in meaning of construct itself

Unit of measure: should match theory of construct and level of analysis (individual, group, etc)

Composition models should be:

-direct

-additive or referent shift

-dispersion (separation, variety, disparity)

-interactive/process

-consistency and stability of test scores

-tests measure stable attributes, consistency reflects stability of an attribute

• Test-retest 
• alternative/equivalent forms
• internal consistency
• inter-rater 

Coefficient of reliability: dependent measures t-test, pearson's correlation

Limitations

-reactivity 

-memory effects

-more than 2 observations

-construct overlap in stability

Coefficient of equivalence: pearson for interval ratio, kappa for nominal/categorical

limitations

-difficult to create parallel forms that are exactly alike

-construct overlap

-more than 2 observations

-assesses internal consistency

-widely used (easier bc you don't need 2 measures or 2 observers)

Kuder-Richardson 20 (KR-20): looks at relation of each item w/ every other item for dichotomous items only

Cronbach's Alpha: continuous data, average of all possible splits

Agreement

(models)

-Sensitive to extremes and levels

Consensus models (degree of shared view)

-Direct consensus: what percent agrees?

-reference-shift consensus: do group members share common view?

Dispersion model (degree of variability in view)

-measuring a construct that is representative of the group

Types of Construction:

-global (e.g. grp function, size, location)

-shared (latent perception/attitudes, norm, climate)

-configural (pattern of dispersion/diversity in group e.g. group consensus)

• Inter-rater reliability: most common when objectifying a qualitiative measure
•  kappa (nominal/ordinal) or correlation coefficients (interval/ratio): shows consistency but not agreement
•  Interclass correlation coefficient (ICC)
•  Rwg (within rater agreement)

Rwg=within rater agreemetn 

• degree all judges agree
• different from how consistent each judge is
• judges can respond differently but very consistently (one always low one always high) which creates high kappa or coefficients but low Rwg

ICC looks across raters and between groups of raters

EG...6 olympic judges, 3 US & 3 Canadian

-calculate overall agreement

-calculate b/ agreement (between US & Canadian)

-raters within country more likely to agree if rating athlete from same country

-raters judging athletes from other counteries rated lower and had less agreement

-high is better but must be balanced with practicality

-high necessary for important final decisions or finding small individual differences

-low is okay for preliminary decisions distinguishing large factors

• Bigger is always better: don't ask the same ?, subscales should be more homogenous than FS
• Alpha measures only internal consistency of scale: influenced by N of items (more than 20)
• Alpha is a fixed property of a scale
• Alpha ranges between 0 & 1: can be negative when items need to be reverse coded

.70 early stages

.80 basic research

.90 minimally tolerable estimate

.95 clinical purposes

(Nunnally, 1978)

if alpha is too high it shows redundency 

-composite test scores typically more reliable with the true score than individual tests alone

-less variance attributed to systematic error

-scale length increases reliability

-each subpart provides its own limitation but corrects for others so composite is more reliable

Methods of estimating

-random/double random designs; complete/partial

-causal steps

-goodman; sobel tests of standard error

Limitations

-function of number of items

-function of average inter-item correlations (lots of really similar items aritificially inflates alpha)

-measures correlation between items, not really consistency

• characterstic of test scores not test itself
• depends on total score var
• more heterogenous populations have higher reliability
• reliability is an estimate interpreted considering sample, context, purpose