Characteristics of survey research:

·      Interdisciplinary

·      Goal to have representative/probability sample

·      Purpose to generate group-level summary stats

·      Research topic clearly defined

·      Desire to measure some unknown characteristics of a pop

·      Collect info about well-defined pop

·      Collection of standardized data

·      Info gathered by asking individuals

·      Questions are usually closed format

·      Numerical/quantitative data are entered into a data file for stats

·      Data often used for both descriptive and analytic use

Purpose of surveys:

·      Exploration

·      Description

·      Hypothesis/theory development

·      Hypothesis/theory testing

·      Explanation of causal model

·      Prediction

·      Evaluation

Considerations for primary data collection

·      What is research problem/question?

·      Which variables should be measured?

·      How should the key variables be measured?

·      What is the pop?

·      How will data be analyzed?

·      How will the results be used?

·      What resources are available to conduct study?

Survey:

systematically collect info on a topic by asking individuals questions with a goal to generate statistics on the group or groups that those individuals represent

Questionnaire:

instrument typically contains a series of related questions for respondents to answer; form of recording data(paper, telephone, web-based)

Sampling frame

list of pop #s from which you draw eligible ppl to participate

doesn’t contain everyone to be in sampling frame; inexpensive

Methods of data collection:

·      Mailed/self administered survey

·      Telephone survey

·      Face-to-face survey/personal interview

=(#eligible sample who complete questionnaire)/(total number of eligible sample)

·      Sense of how many eligible ppl you got to ask about your study

·      About generalizability

Sampling frame bias:

does not include all members of the defined pop; or does not include correct info about sample members

Response bias: 

one subgroup of the pop is more or less likely to cooperate than others

·      Often related to SES

Selecting a method of data collection:

·      What is the coverage and contact info for sampling frame?

·      Who are the respondents?

·      Is research question more amendable to one method than to others?

·      Time and money available?

·      Need to think about pop you are attending to

Things to include in mail survey cover letter:

·      What the study is about, why is it important, how results will be used

·      Why respondent is important to study

·      How respondent was selected

·      Promise of confidentiality

·      Phone number to call with questions and who will be reached at that number

·      Instructions to return questionnaire

·      Sign-in blue ink to avoid impression that they are impersonally mass-produced

·      Should fit onto single page

Phrases to avoid in first line of cover letter:

"enclosed is a questionnaire"

"this is survey"

"I am a researcher

"your help is needed"

·      “to complete a degree at my university, I am required to write a thesis”

Front Cover of Paper Questionnaire

·      Study title

·      Graphic illustration

·      Any needed directions

·      Name and address of study sponsor

Back Cover of Paper Questionnaire 

·      Invitation to make additional comments

·      Thank you

·      Plenty of white space

·      Return mail address

Envelopes:

·      Use 1^st class mail

·      Stamps rather than metered- bc bills are metered

·      Can request address correction

·      Self addressed return envelopes

Order recommended for mailed survey:

·      Advance letter

·      Questionnaire and cover letter

·      “postcard” reminder

·      questionnaire with different cover letter

·      telephone numbers

·      final questionnaire and cover letter

what if rate of return is less than expected?

·      Pattern to nonresponse?

·      Reason for nonresponse?

·      Prob with mailing procedures?

introducing study for telephone survey:

·      organization/agency conducting study

o   identify where you are calling from

o   do not give full name

·      objectives or goals

·      basis of sample selection

·      characteristics respondent possesses that led to his/her inclusion in the sample

·      assurance that participation is valued

·      assurance that answers are neither correct nor incorrect

·      assurance of confidentiality

·      estimate time required and permission to proceed

questions respondents often want to know:

·      who is sponsoring

·      what is purpose?

·      Who is person responsible for survey?

·      How did you get my name?

·      How can I be sure its authentic?

·      Why do you need to know about ppl livng in my household?

·      Why don’t you interview spouse/son/daughter?

·      Is this confidential?

·      Can I get a copy of results?

Reasons for refusal?

·      Too busy

·      Bad health

·      Too old

·      Feel inadequate

·      Not interested

·      No one else’s business what I think

·      Objects to surveys in general

·      Objects to telephone surveys

Best times to call partipants

·      Sat 9am-noon

·      Sun-thurs 6-9pm

o   Privacy issues

o   Is the mode effective?

o   Diff in response to items not bc of participant characteristics but the way questions are asked

o   May end up with mode effects in dataset that you would have to figure out what to do with.

Considerations of web-based survey:

·      Cost

·      Flexibility in question type

·      One question per page vs necessity to scroll

·      Control of sequence of responses

·      Type of data file received how much cleaning and editing is needed

·      Authentication

Unauthenticated surveys

·      Completed by anyone with URl

·      No associated participant list

·      Cant control who completes it or how many times a person completes it

·      Must be used when do not have direct access to email addresses

o   Can send participants personal email

o   Track survey progress on participant level and send reminders

o   Custom data points can be piped into questions or calculations

o   Duplicate surveys are prevented

o   Useful for personalization or verifying info

o   All former responses are preserved unless the participant indicates desire to start over from beginning

o   Must know who you want to invite and reliable means of contact

o   If emailing not available,  very time consuming to locate and confirm info

o   If sending paper invites, must take into acct having to print, sign, stuff and mail

• multimedia capabilities
• required fields
• data validation
• conditional logic

Data validation: 

·      rules can be programmed for each question so there cant be invalid data

o   Time to clean data reduced

o   Less time for searching for outliers

o   No need to recode missing data

·      Conditional logic

o   Skip programs

o   “show if” option

§  navigation is simplified for participant

§  branching errors can be eliminated

§  progression through survey is streamlined

§  completion time is reduced since participants don’t have to pay attention to skip patterns

§  custom error messages

§  piping: customized text into questions, response options. Etc.

§  drop down lists

§  customized end pages

§  progress bar option

§  internal variables

o   same as for interviews using landline phones

o   potentially increased coverage

o   convenience for participants

 §  can randomly call participants

o   portable

o   privacy concerns

o   costs

o   question visibility

o   distractions and safety concerns

o   cannot use automatic dialers

ecological momentary assessment(EMA)

·      repeated measures of current behaviors and experiences in real time, in natural environments

·      aims to minimize recall bias, maximize ecological validity, and allow study of microprocesses that influence behavior in realworld context

·      assessments focus on current states

·      intended to reduce problems with autobiographical memory

o   characterizing individual differences

o   describing natural history

o   assessing contextual association

o   documenting temporal sequence

§  Examination of events in natural spontaneous context

§  Provide info complementary to that obtained by other methods

§  Reduction of likelihood of retrospection

§  User training

§  Participant burden

§  Literacy issues

§  Can be lost, misplaced or forgotten

§  Reactivity

§  Compliance issues

§  Under-reporting

ú  Signaling user

ú  Time and data stamps

ú  Randomization of questions

ú  Hypertextuality(skip patterns)

ú  Data management

Device maintence, visibility, open-ended responses, training, comp based literacy and selection bias

pilot study:

·    draft questions

·      test questions

review and revise(every step not at last step)

expert panels

focus groups

§  Ppl published in your area

§  Opposing viewpoints

§  Interdisciplinary fields

§  Odd number

§  Ppl usually with similar characteristics assembled for a guided discussion of a topic

§  Heterogeneous

§  6-8 ppl

§  warm up activity

ú  good sense of what ppl consider important about your topic

ú  high efficiency

ú  how participants think and talk about topic- terminology

ú  can involve illiterate

o   demonstrate significance/relevance of problem

o   develop operational definitions and measurement methods

o   obtain experience/ demonstrate competence

o   identify real and/or potential problems

o   develop time schedule

• questions standards
• cognitive interviews
• pretest: practice run
• interviewer debriefings
• post-interview interview
• respondent debriefing
• interaction coding

§  content(asking right things)

§  cognitive(understandable, correct info to answer, willing to respond)

§  usability(easily completed)

§  evaluate measurement instrument(s)

§  evaluate data collection procedure

§  evaluate admin procedures

§  evaluate time schedule

§  evaluate budget estimates

§  refine data reduction and analysis plans

§  identify and resolve unanticipated problems

ú  overview of pretest

ú  question by question problem identification

ú  question by question suggestions for revisions

ú  summary comments

ú  have them talk about experiences with pretest

§  tell participant that question portion will be “x” minutes then you’ll go back and speak about their experience

ú  non-verbatim reading of question

ú  requests for clarification

ú  requests to reread the question

ú  inadequate answers

ú  most accurate measurement of probs with particular question that would happen in final test

·      valid measure of the factors of interest

·      convinces respondents to cooperate

·      elicits acceptability accurate info

amounts of error in the answers to questions that are due to factors such as faulty memory, respondent misunderstanding, or method of data collection

factors that interfere with respondent memory

·      passage of time

·      intervening events

·      low salience of the event

·      uniformity of events

·      attitudes

·      beliefs

·      behavior

·      attributes

o   alternative answers are uniform

o   tend to make questions clearer

o   less burdensome to respondent

o   limit extraneous and irrelevant responses

o   aid in asking sensitive questions bc implicitly “acceptable” responses

o   can offer respondent more privacy

o   respondent may select fixed responses randomly rather than in thoughtful fashion

o   require respondent to choose a “closest representation” of actual response

o   subtle distinctions among respondents cannot be detected

o   may lead to inadvertent errors

o   good for exploratory studies

o   commonly used to elicit precise piece of info that respondents can recall easily when large number of possible answers are possible

o   useful for eliciting frequency of sensitive behaviors

o   elicit certain amount of irrelevant and repetitious info

o   requires greater degree of communication skills by respondent

o   may take more time

o   stat analysis require interpretive, subjective, and time-consuming categorization of responses

·      revelant to topic

·      easy to answer

·      should elicit factual answer or uncomplicated opinion

·      interesting

·      applicable to, and answerable by all respondents

·      closed format

·      should not be overly sensitive

used to establish respondents qualifications to answer subsequent questions

·      social desirability

·      adjectives conveying + or – qualities

·      links to authorities

·      cognitively complex

·      acquiescence response set: tendency toward agreement regardless of question content

·      bias related to education: less educated  tend to agree more frequently than well educated

·      must determine if middle category will be included

tendency of respondents to agree rather than disagree with statements as a whole or with what are perceived to be socially desirable responses to questions

o   religious affiliation

o   income

o   race

o   social security numbers

o   sexual practices

·      use broad spectrum options

·      provide context for question

·      use a series of questions to lead up to threatening question

·      placed late in questionnaire

·      clearly delineate choices

·      generally one response per line

·      use numerical codes for close-ended responses

number of responses categories:

·      self admin: <10 response options per question

·      telephone: <5-6 response options per question

·      face to face: can have as many as 15-20 response options if you can show a respondent a card with options listed on it

order of response alternatives

·      randomly list nominal data categories to avoid potential biasing effects

·      list least socially desirable response option first

organization of questionnaire

·      intro

·      respondent selection

·      substantive questions

·      background questions

·      post-interview questions

·      assign numbers to each question

·      use consistent numerical codes and formats(0=no, 1=yes)

·      provide clear instructions

·      provide clear skip instructions

·      phrase full and complete questions

·      use column format for series with:

o   same response categories

o   comparable skip patterns

·      put all parts of a question on the same page

·      allow plenty of space on the questionnaire

end questionnaire with thank you

1. nominal

2. ordinal

3. interval

data in categories can only be counted with regard to frequency of occurrence; no ordering implied(marital status)

·      rank ordering of categories in terms of the extent to which they possess the characteristic of the variable

o   underlying continuum along which respondents can be ranked

o   no assumption about distances between categories

o   ex: s, m, l, xl

·      labels, orders, and uses constant units of measurement to indicate exact value of each category of response

o   age with magnitude between #s: 39-40, 40-41, etc

o   measure what it is supposed to measure

§  threatened by systematic error

o   consistent in producing the same results every time the measure is used

§  threatened by nonsystematic error(random error)

random

systematic

occurs bc of natural and uncontrollable variance; reduced by repeating measurements

occurs when the sample respondents are systematically different from the target pop responses; is not reduced by repeating measurements

·      extent to which a data collection procedure successfully measures a variable of interest

o   extent to which there is a systematic departure in answers from true values

§  true value unknown so we cannot compare the observed value to it

1. face

2. content

3. criterion

4. construct

§  validity of a survey at face value; whether a measurement is logical

ú  shows questions to trained and untrained individuals to assess meaning; very qualitative

ú  simplest validity

§  the extent to which a measure represents all relevant dimensions

ú  evaluation

·      lit review

·      review by experts in field

ú  still qualitative; no stat test

§  extent to which the measure agrees with or predicts some criterion of the “true” value(or gold standard)

ú  evaluation: corr coeff; sens/spec analysis

concurrent

predictive

the extent to which the measure agrees with other criterion measure assessed simultaneously

the extent to which a measure can predict future events; based on prior knowledge; not the outcome of interest of the current study

the extent to which relationships between measures agree with relationships predicted by theories or hypotheses

ú  evaluation: corr coeff(r), sens/spec analysis, factor analyses

convergent

discriminant

agreement among equivalent but independent simultaneous measures

·      lack of agreement with measures of unrelated variables

o   measures ability to distinguish among groups that theory claims ought to be distinguished; assumes your theory is good

1. test-retest reliability

2. inter-rater reliability

3. alternate form reliability

4. split-half reliability

§  measurements taken by the same observer for the same group of subjects with the same instrument, under equivalent conditions, but at diff time points

ú  evaluation by corr coeff, chi-square

ú  problem: ppl become familiar with some of the questions 2^nd time administered

ú  wait 4-8 wks max

ú  only good for conditions that don’t change over a short period of time

§  measurements are taken by diff observers for the same group of subjects, with the same instrument, under equivalent conditions, at the same time; reveals how well the raters agreed when rating the same phenomena

ú  evaluation: rank order or kappa

ú  way to determine how good your coding or instrument is in whatever you’re trying to measure.

§  measurements are taken by same observer for the same or equivalent groups of subjects, with alternate forms of the instrument under equivalent conditions at the same time

ú  evaluation: cronbach alpha

must make sure that items have the same level of difficulty, difficult to design equivalent forms

§  a single form of an instrument is administered once to a group of subjects; items are then divided randomly into halves, which are tested as alternate forms of the important

ú  useful when impractical to assess reliability with 2 tests or 2 administrations

ú  evaluation: score for one half of items is compared to the score for the remaining half of the items

o   mechanical

o   lack of clarity

o   changes in personal factors of observers

o   variation in administration

o   situational factors

o   reactive measures

o   use of tape recorders/video cameras

scale

assignment of individuals to points along an ordered continuum

·      reduces complexity of data from multiple measurements into a single value

·      enable ordinal comparisons of individuals

·      reduce random error by drawing upon consensus among several measurements

you have to think about what is best/worst for you and put yourself somewhere; basis of hospital pain scale

§  easy to read and code

§  easy to use

§  familiar scale for most ppl

§  usually good for global rating

 no known differences between each number and

may miss important components

o   Study purpose

o   Procedures

o   Risks

o   Benefits

o   Freedom to withdraw

o   Alternative treatments

o   Voluntariness

o   Confidentiality

voluntary agreement given by a human participant in a research study following comprehension and consideration of all relevant info

o   Physical disabilities

o   Seeing/visual disabilities

o   Hearing disabilities

o   Cognitive disabilities

Ethical considerations related to publication

·      Plagiarism

·      Data falsification or fabrication

·      Image manipulation

·      Publication redundancy or duplicate submission

·      Gift or ghost authorship

·      Failure to disclose conflicts of interest

Biased peer-review

·      Less expensive

·      Less time-consuming

·      Can be more accurate

·      Some samples can lead to a statistical inference about the entire pop

·      More accurate than census bc you could undercount in a census

(required number of completed instruments)/[(rate of expected eligible)*(expected response rate)]

generalize to the entire pop; unbiased

·      selecting sample at random guards against selection bias

·      more effiecient than a complete enumeration

·      stat theory provides valid estimates for quantiles of interest and an estimate of sampling error

·      sample size can be determined with pre-specified accuracy

1. simple random sample

2. systematic sample

3. stratified sample

4. cluster sample

o   every element has the same probability of selection and every combination of elements has the same probability of selection

o   EPSEM= equal probability selection method

o   Probability of selection= N/n

§  Most basic selection process; easy to understand; properties well-understood

§  Self weighting; standard formulas apply

§  Often cumbersome to carry out in practice; not always feasible to get sampling frame

§  Important subpops may be missed in the sample

o   Every element has the same probability of selection but not every combination can be selected

o   Sampling interval: (N/n)

o   Probability of selection: n/N

o   Every combination does not have same probability

o   Have to determine sampling interval and probability of selection

o   Pros:

§  Easy to implement; only one random number is needed

o   Cons: any hidden pattern or periodic structure in the data will introduce serious bias

o   Done when there are groups of interest for purposes of analysis

o   Ensure sample resembles some aspect of the pop

o   As much as possible, elements within groups should be homogeneous and the groups should be heterogeneous

prop

disprop

§  Proportionate to pop

§  Probability of selection is same even though we drew different numbers bc we drew proportionate to pop

§  Used when we want to ensure sufficient sample sizes for specific stratum are large enough to make further influence

§  Sample size is not necessarily proportional to pop size of the stratum

§  Estimators have to be weighted appropriately

§  Determine ahead of time how many ppl you want in your strata

§  More control over the units in each stratum

§  May increase precision of sampling estimates

§  Some subpops may be of special interest, need to ensure sufficient representation

§  Strata need to be clearly defined, mutually exclusive and exhaustive

§  Pop proportion in each stratum must be known to attach appropriate weight to the statistical formula

§  Has to be possible to draw samples from each stratum; need at least 2 units per stratum

§  Reduction in cost

§  No satisfactory sampling frame available

§  Other sampling  methods not feasible

primary sampling unit

first unit that is sampled in the design