Uncertainty and predictiveness modulate attention in human predictive learning.

Theories of Attention in Associative Learning

Both the theories of Mackintosh (1975) and Pearce and Hall (1980) proposed that the salience of a cue is changed as a consequence of prediction error. Prediction error occurs when the outcome of a cue is not expected. The strength of an association between a cue and an outcome represents the extent to which a cue predicts the outcome. As associative strength increases prediction error decreases and learning ceases when the outcome is fully predicted. Although prediction error can be large for unexpected outcomes (e.g., at the start of training of a cue–outcome association), it can also be large when a fully expected outcome is omitted. The Mackintosh model proposes that on a given trial, the salience of the cue that is the best predictor of the outcome increases and the salience of other cues that are present decrease. Therefore, prediction error is calculated for each cue, independent of the associative strength of the other cues present, and the cue with the smallest prediction error gains attention. Changes in associative strength for individual cues on a trial are calculated by the following equation: Δ V A = α A · θ · ( λ – V A ) . 1

Prediction error is represented by the discrepancy between the current associative strength of Cue A (V A ) and the maximum associative strength that can be supported by the outcome (λ). The salience of the outcome (θ) and the salience of Cue A (α A ) determine the rate at which the current prediction error drives changes in learning on a given trial. Changes in α A on a given trial are governed by the following rule: Δ α A > 0 if | λ – V A | < | λ – V X | Δ α A < 0 if | λ – V A | ≥ | λ – V X | . 2

The prediction error for Cue A (λ − V A ) is compared with the prediction error for all the other cues present on a trial (λ − V X ). If the error is smaller for Cue A than all the other cues, then α A increases but if it is the same or greater then it decreases. The consequences of these rules for changes in associability are clear when considering a situation in which two cues with different initial values of alpha are conditioned in compound (e.g., AB; see Figure 1 ). If we assume that alpha is higher for Cue A than for Cue B, then—because the initial increase in associative strength on Trial 1 will be higher for Cue A than for Cue B—Cue A will be a better predictor of the outcome than Cue B. This results in alpha increasing for Cue A and decreasing for Cue B. The difference in associative strength between the cues drives the difference in alpha further over training and the difference in alpha determines the subsequent maximum associative strength that the cues eventually achieve.

The Pearce–Hall model, in contrast to the Mackintosh model, does not assume that changes in attention to a cue are governed by the individual prediction error for that cue, but the summed error for all cues present on a trial. The following equation determines changes in associative strength on a given trial: Δ V A = α A · S · λ . 3

S and lambda are determined by the intensity of the cue and the outcome, respectively. Here, alpha reflects specifically the associability of the cue, that is, how readily the cue is able to form associations with other stimuli. Alpha changes with experience determined by the extent of prediction error according to the following equation: α n = λ – Σ V n – 1 . 4

Prediction error is calculated as the discrepancy between lambda (the maximum associative strength supported by the unconditioned stimulus) and the combined associative strength of all the stimuli present on a trial (ΣV). Therefore, in contrast to the Mackintosh model that calculates prediction error for each cue (individual error term), the Pearce–Hall model assumes that prediction error is determined by the additive strength of all cues (summed error term). The associability of a cue on a given trial ( n ) is determined by the size of prediction error on the preceding trial ( n − 1). As prediction error decreases (i.e., as associative strength increases), alpha decreases. Importantly, alpha is determined by the size of prediction error regardless of whether it is positive (due to the surprising presence of an outcome) or negative (due to the surprising absence of an outcome). Therefore, partial reinforcement, in which a cue is sometimes paired with an outcome and sometimes not, leads to alpha remaining high over training. This scenario is modeled in Figure 2 . Cues A and B were paired with reinforcement on separate trials, but Cue A was reinforced on every trial, and Cue B was reinforced on 50% of trials. Although the associative strength of Cue A is higher than that of Cue B, the alpha for Cue A reduces in comparison to Cue B over training in line with decreases in the size of prediction error. Alpha for Cue B remains high over training.

Although both models assume that the initial level of alpha is determined by the salience of a cue and the amount of attention that it receives, the Mackintosh model and Pearce–Hall model propose that changes in alpha specifically determine how readily a cue will be learned (i.e., a cue’s associability). Therefore, the experiments that have been used as support for either the Mackintosh (1975) or Pearce and Hall (1980) models have assessed changes in attention by measuring how rapidly new learning is acquired with the cues (e.g., Hall & Pearce, 1979 ; Mackintosh & Little, 1969 ). This is done by pairing the previously experienced cues with new outcomes and measuring the extent of learning of the new cue–outcome associations. Although the two theories make opposing predictions, due to the wealth of evidence for both accounts, the results may be reconciled by assuming that individual prediction error and summed prediction error separately lead to predictiveness and uncertainty effects and under particular conditions one effect may outweigh the other ( Haselgrove, Esber, Pearce, & Jones, 2010 ; Le Pelley, 2004 ; Pearce & Mackintosh, 2010 ).

By describi

The Role of Predictiveness and Uncertainty in Human Associative Learning

In human associative learning, a role for predictiveness in determining attention has also been found (e.g., Le Pelley & McLaren, 2003 ), but a role for uncertainty has received less support. Similar to the procedures in the animal literature, changes in attention as a consequence of learning have been assessed by measuring the extent of new learning with cues. In a recent review Le Pelley, Mitchell, Beesley, George, and Wills (2016) cite 18 articles demonstrating that such procedures reveal an effect of predictiveness on attention. In contrast, a role for uncertainty, as measured by associability (how rapidly a cue is learned about), is less clear. Griffiths, Johnson, and Mitchell (2011) showed that prior learning of a predictive relationship between a cue and a moderate outcome reduced the ability of the cue to become predictive of a larger outcome compared with a cue whose consequences were uncertain due to receiving a number of extinction trials in which the moderate outcome was not presented. A similar study, however, failed to find support for a role of uncertainty in attention ( Packer, Siddle, & Tipp, 1989 ). Furthermore, Le Pelley et al. (2016) stated that unpublished attempts to replicate the findings of Griffiths et al. (2011) have not been successful. Recently, however, Easdale, Le Pelley, and Beesley (2017) have shown that sudden switches in the level of uncertainty may increase associability.

Rather than measuring changes in associability, some studies have examined the effect of uncertainty on overt attention, as measured by eye gaze. Hogarth, Dickinson, Austin, Brown, and Duka (2008) found that participants showed greater fixation of eye gaze for a cue that was uncertain cue due to leading to an outcome (an auditory stimulus) on only 50% trials compared with a cue that was predictive due to leading to the outcome on 100% of trials. This effect, however, is not always replicated ( Austin & Duka, 2010 ). Beesley et al. (2015) found that participants spend a greater proportion of time fixating on uncertain cues within a trial than on predictive cues. There was not, however, any advantage of uncertain cues over predictive cues in a subsequent test of associability. Indeed, in the test of associability, the opposite was found, with previously predictive cues learned about more than previously uncertain cues. This has led to the suggestion that uncertainty may affect levels of attention to all cues generally, rather than leading to stimulus-specific changes in attention and associability ( Beesley et al., 2015 ; Le Pelley et al., 2016 ).

The lack of behavioral evidence for a role of uncertainty in attention in human associative learning is at odds with research demonstrating neural correlates of uncertainty in the human brain. For example, Li, Schiller, Schoenbaum, Phelps, and Daw (2011) found that patterns of activity to cues were sensitive to the absolute prediction error associated with the cue (i.e., the discrepancy between the outcome and the anticipation of the outcome, independent of whether the discrepancy was positive or negative). This neural correlate of unsigned prediction error mimics the calculation of error for determining uncertainty as described by Pearce and Hall (1980) . It is possible that the behavioral procedures used to date have not been sensitive enough to detect an effect of uncertainty. Alternatively, the lack of behavioral evidence for uncertainty may suggest that although uncertainty is encoded at some level, it does not impact on attention. If this is the case it would suggest a divergence between humans and nonhuman animals in the learning mechanisms that affect attention.

In the present study, we report a series of experiments that demonstrate that nonpredictive, uncertain cues do receive more attention than predictive cues, under particular conditions, as measured by the extent to which the cues can enter into associations with new outcomes, in a human learning procedure. We examined changes in associability (the learning rate parameter for a cue) rather than more explicit measures of attention such as eye gaze due to the assumptions of the Mackintosh (1975) and Pearce and Hall (1980) models that alpha determines learning rate. Our starting point for this line of work was an experiment (Experiment 1) that was similar in design to an experiment reported by Livesey, Thorwart, De Fina, and Harris (2011) , in which cues that were irrelevant in a discrimination learning procedure, by virtue of being nonpredictive and presented simultaneously with predictive cues, were compared with cues that were nonpredictive, but were not presented simultaneously with predictive cues (uncertain cues; Experiment 1). Our design was somewhat simpler than that used by Livesey et al. and we were interested in establishing the conditions under which changes in associability occur. Thus, task difficulty and complexity of the design of the task have been suggested as factors that may influence whether an uncertainty effect is observed ( Le Pelley et al., 2016 ). In contrast to Livesey et al., who failed to find a difference between the two types of cues, we found that the irrelevant cues had lower associability than the uncertain cues. There are a number of potential explanations for this result, but the results of Experiments 2a and 2b demonstrated that the results of Experiment 1 reflected, at least in part, that uncertain cues increase in associability relative to other cues. Those results also contradicted the results of Livesey et al., who found the opposite effect: predictive cues were learned about more readily than uncertain cues. The subsequent experiments were devoted to identifying the key differences between our procedures and those used by Livesey et al. that determine whether predictiveness or uncertainty has the greatest effect on attention paid to a cue. Experiment 3 was a replication of the procedure used by Le Pelley and McLaren (2003) to determine whe

Experiment 1

The purpose of Experiment 1 was to assess whether uncertainty, as determined by the summed associative strength of a compound of stimuli, affects the associability of cues. The design of Experiment 1 is shown in Table 1 . In Stage 1, participants received trials with cues that were not predictive of outcomes by virtue of being paired equally often with two outcomes (i.e., Outcomes 1 and 2) across trials. Some of these nonpredictive cues were presented in compound with cues that were predictive across trials. Thus, Cues V through Y were nonpredictive, but were presented in compound with Cues A through D, which were predictive. For example, Cue V led to Outcome 1 when presented in compound with Cue A, but led to Outcome 2 when presented in compound with Cue B. In contrast, Cue A led to Outcome 1 regardless of the other cue in the compound (i.e., AV or AW). To differentiate between cues, we refer to the nonpredictive cues V through Y as irrelevant cues . Other nonpredictive cues were presented in compounds with cues that were also equally nonpredictive. Thus, Cues P through S were presented in the compounds PQ, RS, PS, and QR, and these compounds were paired with Outcomes 1 and 2 equally often. We refer to these nonpredictive cues as uncertain cues .

Although the uncertain and irrelevant cues have the same statistical relationship with Outcomes 1 and 2, the compounds in which they are presented differ in terms of their summed prediction error. The summed error reflects the discrepancy between the outcome and the combined predictive strength of all the cues present on a trial. The summed error of the compounds that consist of two uncertain cues will be high due to both cues being nonpredictive of the outcome. The summed error of the compounds that include the irrelevant cues will be lower, however, due to the presence of the predictive cues. Thus, as the associative strength of the predictive cue increases over the training, the summed error of the compound decreases. This is an important distinction given the assumptions of the Pearce–Hall model, which predicts that increases in attention to nonpredictive cues are driven by the summed error term rather than the individual error of cues. Therefore, the Pearce–Hall model anticipates that uncertain cues should receive more attention than irrelevant cues.

To assess whether uncertain cues gained more attention than irrelevant cues the associability of the stimuli was assessed in a second stage of training (see Table 1 ). Participants were presented with compounds consisting of one irrelevant cue and one uncertain cue. These new compounds were paired with new outcomes; either Outcome 3 or 4. Therefore, the irrelevant cues and uncertain cues were now equally predictive of these new outcomes. In the test phase participants were presented with novel compounds that consisted of either two irrelevant cues or two uncertain cues that had each led to the same outcome in Stage 2. Participants were asked to rate how likely Outcome 3 or 4 was given a particular compound. Greater attention to one type of cue over another would be indicated by more extreme ratings of the compounds for the correct outcome.

Twenty-four people (10 women, 14 men) participated in Experiment 1. The age range was 18 to 38 ( M = 25.36, SD = 4.26). All participants had normal or corrected-to-normal vision. Durham University psychology undergraduates received participant-pool credit and others were compensated for their time at a rate of £10/hr ($13.07). All procedures were approved by the Department of Psychology Ethics Sub-Committee (15–10), Durham University.

The sample sizes across all experiments (except Experiment 3, see respective Method section) ranged from 21 to 32 (Experiment 5 used a between-subjects procedure with n = 24 per group). Variation between experiments reflected the number of participants that were available for testing within a particular time frame. For each experiment or between-subjects condition within an experiment, we aimed to test in excess of 20 participants similar to the study by Livesey et al. (2011) that used sample sizes of between 23 and 31 participants.

Apparatus and stimuli

All experimental stimuli were presented on a standard desktop computer with a 19-in. CRT monitor. Presentation of stimuli was controlled by MATLAB with CRS (Cambridge Research Systems, Rochester, England) toolbox and Psychtoolbox ( Brainard, 1997 ). The distance between participants and the CRT monitor was 45 cm. Flags of the following countries were used as cues: United States, Brazil, Canada, China, United Kingdom, Spain, France, Germany, Israel, Japan, Korea, Mexico, Russia, Singapore, Sweden, Turkey, Benin, Guyana, Jamaica, The Republic of the Congo, Portugal, Cuba, Panama, and Uruguay. Each flag was 10° × 8° (Width × Length) in size. The outcomes (1 through 4) were represented by images depicting support (image of an apple), attack (image of a bomb), retreat (image of man running), and surrender (image of a man kneeling). Each outcome image was 4.6° × 4.3° in size. Participants made responses by clicking on a mouse.

Procedure

Participants were instructed that they would play the role of a soldier and were required to predict which outcome would be correct given the combination of flags presented. They were told that they would receive feedback for each choice, such that they could learn by trial and error as the procedure progressed. In Stage 1, each trial started with the presentation of two cues (flags) and two outcomes. Flags were presented in the top left and right corners of the screen. Outcomes were presented in the middle of the lower half of the screen. One outcome was presented above the other outcome. Participants had to choose to either select an upper outcome icon (e.g., bomb) or lower outcome icon (e.g., retreat) by using a left click of the mouse. Immediately after a response was made the word “Correct!” or “Incorrect” appeared in the center of the screen for one second. The next trial started immediately after the feedback screen. Participants received trials that belonged to one of two different conditions (see Table 1 ). In the predictive/irrelevant condition, participants were presented with pairs of flags. Across trials, individual cues would appear equally often with two other flags (e.g., on half the trials in which Cue A was presented, it would be presented with V and on the other half with W). The unique combination of flags on a particular trial always led to the same outcome in Stage 1, either Outcome 1 or 2. However, across trials, one flag in each compound was predictive in that it always led to the same outcome in Stage 1, independent of which flag it was paired with on a given trial (e.g., A was predictive of Outcome 1 when presented with other flags: AV→O1, AW→O1). The other flag in each compound was irrelevant in that it was not predictive by virtue of being paired with two different outcomes equally often (e.g., V was irrelevant when presented with other flags: AV→O1, BV→O2). Participants received eight trial types in the predictive/irrelevant condition: AV→O1, AW→O1, BV→O2, BW→O2, CX→O2, CY→O2, DX→O1, DY→O1. Cues A, B, C, and D were predictive, and V, W, X and Y were irrelevant. In the uncertain condition, participants were presented with pairs of flags that, across trials, led to two different outcomes equally often. Similar to the predictive/irrelevant condition, individual flags were each presented equally often with two other cues (e.g., PQ and PS), but independent of the particular compound that was presented, the probability of a particular outcome was 50%. Participants received four trial types in the uncertain condition: PQ→O1/O2, PS→ O1/O2, RQ→O1/O2, RS→O1/O2. Participants received 192 trials in total, with 16 trials of each trial type. The order of trial types across trials was random with the constraint that there was an equal number of each trial type every 48 trials. For every trial type, the spatial location of individual flags was balanced across every four trials of the same trial type so that each flag equally often occupied the left or right location (e.g., A on the left, V on the right; V on the left, A on the right). The spatial location (top or bottom) of Outcome 1 and 2 was random across trials.

In Stage 2, participants received eight trial types in which pairs of flags reliably led to either Outcome 3 or 4. Four of the eight trial types consisted of pairs of flags that included one irrelevant cue and one uncertain cue from Stage 1 (recombined cues: VP→O3, WQ→O4, XR→O3, YS→O4). For the remaining trial types, new flags that were not previously experienced in Stage 1 were used (EF→O3, GH→O4, IJ→O3, KL→O4). These trials with new flags were used as filler trials in order to increase the memory load of Stage 2, and replicated, in part, the procedure used by Le Pelley and McLaren (2003) . Participants received 64 trials consisting of eight trials of each trial type. The order of trial types across trials was random with the constraint that there were an equal number of each trial type every 16 trials. All other details were the same as Stage 1.

In the test phase, participants were presented with novel pairings of the flags previously presented in Stage 2. Flags were presented in the top left and right corners of the screen in a similar manner to the previous training stages. Participants were asked to rate how likely Outcome 3 or Outcome 4 was given the combination of flags on a scale ranging from 1 to 9, which ran horizontally on the screen, with one outcome at one end of the scale and the other outcome at the other end. Participants were instructed that choosing either 1 or 9 would indicate that the outcome corresponding to the respective numbers was likely, whereas the other outcome was not. There were eight trial types. Two of the trial types consisted of pairs of flags that were previously irrelevant in Stage 1. One pair consisted of flags that had both led to Outcome 3 in Stage 2 (VX) and the other Outcome 4 (WY). Two of trial types consisted of pairs of flags that were previously uncertain in Stage 1. One pair consisted of flags that had both led to Outcome 3 in Stage 2 (PR) and the other Outcome 4 (QS). The remaining trial types consisted of the new flags presented in Stage 2. One trial type consisted of flags that led to Outcome 3 in Stage 2 (IJ), and another with flags that led to Outcome 4 (KL). The remaining trial types consisted of pairs of flags that had led to different outcomes during Stage 2 (EH and FG). The purpose of the test trials with the filler cues from Stage 2 was to test whether participants were able to use the rating scale appropriately and to replicate the procedure used by ( Le Pelley & McLaren, 2003 ). Participants received two test trials with each trial type. The spatial location of each flag was balanced such that across trials each flag appeared equally often on the left and right. The location of Outcome 3 and 4 on the scale was random across trials.

The identity of each cue (A–D, P–S and V–Y) was random across participant

Data analysis

The accuracy of responding, as measured by the proportion of correct responses for the different conditions was recorded during Stage 1 and 2 training. Performance was assessed over blocks of trials (the number of trials per block is stated in the relevant analyses). In the test stage the ratings were coded such that scores of 1 indicated that Outcome 3 was likely and scores of 9 indicated that Outcome 4 was likely. The mean score for the two test trials of each trial type was calculated. For all experiments, data were analyzed using multifactorial analyses of variance (ANOVAs). Interactions were analyzed by simple main effects analysis using the pooled error term from the original ANOVA.

Analysis of the filler trials was conducted in Stage 2 to determine whether these new cues were learned in addition to the recombined cues that were previously presented in Stage 2. For the sake of brevity, we have omitted analyses of the test trials with the filler cues; but for all experiments, performance on the filler cue test trials was as expected with ratings being below 5 for IJ and above 5 for KL, indicating that participants that learned the cue–outcome associations. Ratings for EH and FG were close to 5, consistent with the fact that each compound included one cue associated with Outcome 3 and another with Outcome 4.

For all experiments, no exclusion criteria were used and the initial data analysis was carried out on all participants. Other studies, such as Livesey et al. (2011) , have used an exclusion criterion in order to eliminate participants that did not learn in the first stage of training. In order to aid comparison with studies that have employed an exclusion criterion, the number of subjects that failed to show performance of 60% and above in the last half of Stage 1 training on the soluble components of the learning task are reported. This criterion was used by Livesey et al. (2011) . In addition, we report in the online supplemental material statistical analyses of the test phase on the subset of participants that met the criterion.

Stage 2

Participants acquired the discrimination over training. Learning was superior for the new cues compared with the recombined cues from Stage 1. Mean performance on the last block (two trials of each trial type) was 81.25% ( SEM = 4.32) for the recombined condition and 92.71% ( SEM = 2.59) for the novel condition. There was a significant effect of block, F (3, 69) = 35.50, p < .001, η p 2 = .61, 90% CI [.47, .68], and cue condition, F (1, 23) = 18.50, p < .001, η p 2 = .45, 90% CI [.18, .61], but no significant interaction of factors, F (3, 69) = 1.71, p = .17, η p 2 = .07, 90% CI [.00, .15].

Test stage

The ratings for the test stage are shown in Figure 3a . Participants rated the likelihood that Outcome 3 or Outcome 4 would occur on a nine-point scale. Scores below 5 indicated that participants expected Outcome 3, and scores above 5 indicated that participants expected Outcome 4. The raw ratings on the 1 to 9 score were analyzed. The ratings for compounds consisting of cues paired with Outcome 4 (WYand QS) were higher than for those paired with Outcome 3 (VX and PR), indicating that participants learned the cue–outcome associations. The difference between cues paired with Outcomes 3 and 4 was greater for the uncertain condition than the irrelevant condition. A 2 (cue condition: irrelevant cues [VX, WY] vs. uncertain cues [PR, QS]) × 2 (Outcome: 3 [VX, PR] vs. 4 [WY, QS]) ANOVA was conducted. There was a significant main effect of outcome, F (1, 23) = 29.11, p < .001, η p 2 = .56, 90% CI [.30, .69], but no main effect of cue condition ( F < 1, p = .43). There was an interaction between cue condition and outcome, F (1, 23) = 18.96, p < .001, η p 2 = .45, 90% CI [.18, .61], demonstrating that the effect of outcome was significantly greater for the uncertain cues than for irrelevant cues. Simple main effects analysis of the interaction showed that there was a significant effect of outcome for the uncertain cues, F (1, 23) = 33.82, p < .001, η p 2 = .60, 90% CI [.34, .72], and the irrelevant cues, F (1, 23) = 13.88, p = .001, η p 2 = .37, 90% CI [.12, .55]. Analysis of the test phase results excluding the participants that failed to meet the Stage 1 learning criterion showed a similar pattern of results (see Table S1 and Figure S1a in the online supplemental material).

Following Stage 2 training, participants showed greater learning with the uncertain cues than with the irrelevant cues, indicating that, because of Stage 1 training, associability was greater for the uncertain cues than for the irrelevant cues. The results are not consistent with those reported by Livesey et al. (2011) . They failed to find any difference between irrelevant and uncertain cues, and, therefore, concluded that attention was controlled by the individual prediction error for each cue rather than the summed error for each compound. Instead, our results are consistent with the prediction that uncertain cues receive greater attention than irrelevant cues because associability remains high due to the summed error calculated using the combined associative strength of both cues. For the irrelevant cues, the summed error per trial by the end of training was low for irrelevant cues, by virtue of participants learning about the predictive cues. In other words, the uncertain cues were able to benefit from increases in associability caused by a Pearce–Hall mechanism to a greater extent than the irrelevant cues ( Haselgrove et al., 2010 ; Pearce & Hall, 1980 ). This possibility was explored in Experiments 2a and 2b.

In addition, whereas both our experiment and those of Livesey et al. (2011) tested changes in attention for irrelevant and uncertain cues, the procedure used in the current experiment differed from that used by Livesey et al. (2011) in a number of ways, which may have led to the difference in the results. The cause of the discrepancy between our findings and those of Livesey et al. (2011) were investigated in Experiments 2b, 4a, and 4b.