Calibration Studies Testing
Overall Sample | Scale Setting Sub-Sample
| Additional Chronic Pain Sample | Sleep
and Wake Disturbance Sample | PROMIS Standards for Measure Development |
PROMIS Standards slide deck
From July 2006 to March 2007, data were collected from the U.S. general population
and multiple disease populations. The sampling plan was developed for collecting
responses to the candidate items from the targeted PROMIS domains and was designed
to accommodate multiple objectives:
- create item calibrations for each domain;
- estimate profile scores for various disease populations;
- create linking metrics to legacy questionnaires (e.g., SF–36);
- confirm the factor structure of the domains; and
- conduct item and bank analyses. Because of the large total number of items (>
1000), it was not possible for participants to respond to the entire pool. Based
on an estimate of 4 questions per minute, the number of items administered to any
respondent was limited to about 150 (37 minutes).
Two data collection designs ("full bank" and "block administration")
were used. Some individuals were administered full banks of 56 items for a subset
of the PROMIS Calibration Studies domains while others were administered 14 blocks
of 7 items selected from the 14 item banks (3 physical functioning banks, anxiety,
depression, anger, substance abuse, fatigue impact, fatigue experience, social–role
performance, social–role satisfaction, pain interference, pain quality, pain
behavior). In addition to the PROMIS items and appropriate "legacy" items
(items from widely used fixed length measures such as the SF–36) completed
by those administered full banks of items, participants were administered approximately
21 auxiliary items consisting of global health rating items and sociodemographic
variables including age, income, number of hospitalizations, disability days, whether
they take prescribed medicines, body mass index, gender, race/ethnicity, whether
the person was married or living with someone, educational attainment, and whether
the person was working full–time, a full time student, or not working full
time or going to school full–time. There were also a series of questions about
the presence and degree of limitations related to 25 chronic medical conditions:
hypertension, angina, CAD, heart failure, heart attack, stroke or TIA, liver disease,
kidney disease, arthritis or rheumatism, osteoarthritis, migraines, asthma, COPD,
diabetes, cancer, depression, anxiety, alcohol or drug problems, sleep disorder,
HIV/AIDS, spinal cord injury, multiple sclerosis, Parkinson's disease, epilepsy,
and amyotrophic lateral sclerosis.
The full–bank administration allows for evaluation of dimensionality and calibration
within item banks (domains). The block administration permits an evaluation of associations
between domains. The ability to calibrate items on a common metric (item linking)
is possible because of the administration of blocks of items from the full bank
to general population and clinical samples. Each item was administered to at least
900 respondents from the general population and 500 respondents with a chronic medical
Data were collected primarily by YouGovPolimetrix (www.polimetrix.com, also see www.pollingpoint.com), a polling firm based in Palo
Alto, CA. YouGovPolimetrix operates PollingPoint.com, a centralized portal that
allows interested individuals to provide their views about public policy and other
current issues. The respondents for a typical YouGovPolimetrix Internet survey are
selected from the PollingPoint panel, a panel of over one million respondents who
have provided YouGovPolimetrix with their names, physical addresses, email addresses,
and other information, and who regularly participate in online surveys. YouGovPolimetrix
uses a sample matching procedure to select a representative sample when a listing
of the population is available but it is difficult to draw a random sample from
the population frame. The sample matching methodology starts with a listing of all
respondents in the desired or target population. Next, a random sample of the desired
size is selected from the population listing (the "target sample"). Third,
for each element of the target sample, the closest match is selected from the PollingPoint
panel. This method has been shown to give accurate results in a wide variety of
applications, even for groups significantly underrepresented on the Internet. Its
validity depends upon the panel of available respondents being sufficiently large
and diverse, not upon Internet usage or other types of behavior. For PROMIS, we
specified targets in terms of gender (50% female), age (20% in each of 5 age groups:
18–29, 30–44, 45–59, 60–74, 75+), race/ethnicity (10% black
and Hispanic), and education (10% less than high school graduate).
Panelists have been recruited by a variety of methods, such as e-random digit dialing,
invitations via web newsletters, and Internet poll-based recruitment where panelists
have opted to participate in a survey advertised on the World Wide Web. The topics
of these surveys vary greatly – from politics to polls on popular entertainment.
All of the PollingPoint panelists have provided their e-mail so that they may receive
survey invitations to participate in future surveys. Panel members receive minimal
This methodology was developed by Stanford Political Science Professor Douglas Rivers
and was recently evaluated using the 2005 California Special Election. The matched
samples performed as well or better than conventional random-digit-dial (RDD) telephone
surveys, correctly predicting the outcomes for all of the propositions and having
an average absolute error of 3 percent (consistent with normal sampling error).
In general, YouGovPolimetrix has reported a base response rate of 67% for general
population surveys (similar to the general population component of the PROMIS Calibration
Studies data collection) and rates at or exceeding 80% with more specialized, clinical
samples such as the ones in PROMIS. They also utilize sample matching methodologies
for selection of representative samples from non–randomly selected pools of
respondents, which is ideally suited for Web access panels.
The PROMIS Calibration Studies sample included 21,133 respondents, with 1,532 recruited
from primary research sites associated with PROMIS network sites and the vast majority
(19,601) from YouGovPolimetrix’s panel sample. YouGovPolimetrix sample data was
collected using their website on a secure server. The PROMIS network site data was
collected using the PROMIS Assessment System.
The PROMIS Statistical Center (PSC) received de–identified datasets from YouGovPolimetrix.
Each of the 7,005 individuals (6,676 from YouGovPolimetrix, 236 from UNC, and 93
from Stanford) assigned to administration of full item banks was administered a
pair of the 14 item banks. The 7–item blocks for each of the 14 banks were
administered to 14,128 individuals (6,245 general population, 7,883 clinical samples).
The clinical samples included persons with heart disease (n = 1,156), cancer (n
= 1,754), rheumatoid arthritis (n = 557), osteoarthritis (n = 918), psychiatric
illness (n = 1,193), chronic obstructive pulmonary disease (n = 1,214), spinal cord
injury (n = 531), and other conditions (n = 560).
The clinical population supplied by YouGovPolimetrix was identified through a pre–survey
of 250,000 YouGovPolimetrix panel members. These respondents were asked to complete
the PROMIS clinical form. Persons who self–reported currently being diagnosed
with a given condition were eligible for inclusion in the Calibration Studies clinical
sample associated with that condition. It should be noted that the general population
sample included people with and without diseases (general population respondents
were also administered the clinical form but their responses did not impact their
participation in the general population sample).
- Liu, H. H., Cella, D., Gershon, R., Shen, J., Morales, L. S., Riley, W., & Hays,
R. D. (2010).
Representativeness of the PROMIS Internet panel. Journal of Clinical Epidemiology.