Item Classification
Confronted with thousands of items, a method for sorting through the content and
deciding on the most representative and informative items was needed. We called
this method "binning and winnowing." Items were placed into common "bins,"
typically defined by the content of the stem. Next, items were "winnowed"
out if they were deemed redundant or inferior to alternative items in the same bin.
Binning
The PROMIS domain workgroups first selected those items from the item library that
they believed represented their domain. This process was done in teams so that at
least 2 people reviewed each item for inclusion. Upon completion of domain identification,
domain workgroups proceeded with the task of binning items. Binning refers to a
systematic process for grouping items according to meaning and specific latent construct.
For example, "walking" became a bin within the physical function domain.
The final goal was to have a bin from which a small number of items could be chosen
to adequately represent the bin. We did not predetermine the number of items that
would adequately represent a bin. Rather, the goal for this process was to identify
enough items to capture the meaning of the bin and to eliminate unnecessary redundancy
in the item pool. By grouping items systematically, the domain workgroups could
observe redundancy among items and identify the best potential items based on qualitative
characteristics.
PROMIS domain workgroups (including several investigators across the PROMIS research
sites and the SCC) began by creating a set of bins based on a review of that domain’s
literature, including previous factor analytic studies of domain items, and theory-based
studies of the domain. This "top-down" approach began with a conceptual
model of the facets of each domain. However, each domain workgroup approached the
process with the flexibility to add or subtract bins based on the content of items
themselves. By taking this approach, we retained the organizational structure put
forth by the domain experts, but took advantage of new ideas as expressed by the
items written in the clinical literature. This allowed for the most inclusive and
open approach at this stage of item evaluation.
The purpose of binning was to enable the identification of redundant and inferior
items. Thus, what is important is that the final set of items emerging from this
process adequately represents the facets of the latent trait.
Winnowing
The goal of winnowing was to reduce the large item pool down to a representative
set of items. The process of winnowing helped to identify item characteristics that
would include or exclude them from the PROMIS item banks based on domain definitions.
Ultimately, this process was based on the judgment of reviewers and was accomplished
by a consensus process of 2 or 3 reviewers for each domain. We adopted a set of
criteria for excluding entire bins or items within bins because they were not applicable
to the current domain activities for PROMIS. Many items excluded seem to measure
important domains or subdomains that are not currently the focus of PROMIS item
banks.
PROMIS investigators used the following criteria to remove items from consideration:
- item content was inconsistent with the domain definition;
- an item was semantically redundant with a previous item;
- the item content was too narrow to have universal applicability;
- the stem of the item was disease specific, reducing general applicability of the
item; and
- the item was confusing.
For example, items related to satisfaction with physical function were identified,
binned and removed from the physical function item bank consideration, because satisfaction
was not in the PROMIS definition of physical function.
Across all domains, approximately 30% of the items were eliminated due to redundancy,
and approximately 45% were eliminated because they did not fit within the domain
definitions adopted by PROMIS investigators.
By carefully analyzing each item and comparing them to other items within a given
bin, domain workgroups were better able to apply the several criteria to each item.
All decisions about item winnowing were reviewed by multiple members of the domain
workgroup and members of the SCC to ensure a high level of consensus and to impose
some standardization of processes across domain groups.
The process of binning and winnowing yielded a smaller set of items that were then
subjected to editing to match PROMIS stylistic conventions in a process of item
revision.
Publications Reporting Item Classification
- DeWalt, D.A., Rothrock, N., Yount S., Stone, A., on behalf of the PROMIS Cooperative
Group. Evaluation of Item Candidates –The PROMIS Qualitative Item Review.
Med Care 2007; 45: S12-S21.