From Trusting the Subject (2003), Anthony Jack and Andreas Roepstorff write:
“The unique challenge facing a science of consciousness is that that the best instrument available for measuring experience depends on cognitive processes internal to the subject. So just how much faith can we place in the capacity of the mind to understand itself? In principle, the construction of a maximally robust methodology for introspective evidence would require a detailed understanding of the operation of introspective processes — the processes that mediate the acquisition of introspective knowledge and underlie the production of introspective reports”
“It is important to realize that no principled problem stands in the way of the scientific assessment of various types of introspective evidence. The testing of the reliability, consistency and validity of various types of introspective report measures lies well within the orbit of currently available methods. A measure may be called ‘reliable’ if it yields the same results when tested in multiple sessions over time (‘test–retest reliability’) and across individuals (a cousin of ‘inter-rater’ and ‘inter-observer’ reliability). Of course, subjects’ reports may differ, and so appear to be unreliable, simply because their internal mental processes and states vary. Thus it is critical to establish well controlled experimental conditions for eliciting reports. The considerable advances in behavioural science since the time of the Introspectionists offers experimenters considerable advantages in this regard (see Ericsson, this volume). Not only do these advances make it much more probable that experimenters can establish conditions under which introspective measures can be shown to be reliable, they also provide much greater insight into the behavioural and neural correlates of experiential phenomena.
A measure may be called ‘consistent’ when it can be shown that the results are not due to specific features of the measurement technique. Tests of consistency provide a means of checking that the observed effect is not due to a methodological artefact. Thus we might test the consistency of introspective evidence by comparing immediate forced-choice button-press reports with retrospective and open-ended verbal reports. In this way we might establish, for instance: that the results of forced-choice button-press reports have not been influenced by variations in the criterion for response or by automatisation of response such that they no longer constitute true introspective reports; and that retrospective reports have not been distorted by forgetting or memory interference effects.
‘Validity’ is the most important factor to establish, yet it is also the most theoretically complex, and a particularly vexed issue in cognitive science. A measure is validated when it can be shown to accurately reflect the phenomenon it purports to measure. Validity is complex because scientific measures are often simultaneously interpreted as providing evidence for phenomena at a number of
different levels. A rough characterisation of three major sources of evidence in cognitive science might read as follows:
-Data from functional Magnetic Resonance Imaging (fMRI) serves most directly as evidence of cerebral blood flow (which has been validated), less directly as evidence for neural activity (which is in the process of being properly validated), and least directly as a means of identifying and localising specific cognitive functions (far from well validated).
– Behavioural measures (e.g. the averaging of reaction time measures over multiple trials) serve most directly as evidence for stable patterns of behaviour, less directly as a means of assessing information processing, and least directly as means of establishing the existence and operation of specific cognitive functions.
-Introspective reports serve most directly as evidence about the beliefs that subjects have about their own experience, less directly as evidence concerning the existence of experiential phenomena, and least directly as evidence concerning the operation of specific cognitive functions.