8b. The results are in! Part 2

Foundation Study Framework students are currently in the midst of completing their end of year assessments, and we wish them the best of luck with these!

Their modules finished just before Easter, and before they went off on exam leave, students were asked to complete a second COGS survey (identical to the one they completed in term one) to assess whether their level of self-efficacy had changed throughout the programme (see blog 2 and 3 for details on the COGS survey).

Whilst we were unable to capture results from the full student cohort, both COGS surveys were completed in full by 59 students, allowing us to compare a mean self-efficacy score at the start and end of their first year on the Foundation Study Framework. Whilst this may seem like a short time period, self-efficacy can be context-specific and therefore change fairly quickly (Ouweneel et al., 2013).

We were mainly interested to know if their mean self-efficacy score hand changed over time, and whether there was a difference between those that had attended the GRIT programme (see blog 5 for information on GRIT).

Statistically there was no difference in mean self-efficacy scores over time (term one and term two), and regardless of whether they attended the GRIT programme or not. The results of a Two-way mixed ANOVA showed that there was no difference in mean self-efficacy scores over time (F(1, 34)= .16, p= .69, ƞp2= <.005), and there was no significant interaction between time and attending the GRIT programme (F (1, 34) = .29, p= .58, ƞp2= <.009).

Total mean scores in term one were 3.18 (moderately true), and in term two were 3.2 (moderately true), thus indicating that students that engaged with both surveys generally had fairly good levels of self-efficacy to begin with, therefore it could be that any further development has little effect on those scoring 3 or 4 on the scale.

As there was also no difference found between those who attended the GRIT programme, it could be suggested that the GRIT programme is not of any benefit for self-efficacy in isolation. However participants reported a number of positive feelings regarding the GRIT programme that suggest other benefits aside from self-efficacy can be gained. Benefits surrounding self-esteem, relationships with peers, and acknowledging responsibility for their own future (see blog 6 and 7 for full details) formed part of the feedback from those that engaged with the GRIT programme.

It may also suggest that students that fully engaged with the research were already experiencing more positive levels of self-efficacy, and that those students that opted out are actually the ones who would benefit more from the GRIT programme. Tahmassian and Moghadam (2011) highlighted the link between self-efficacy and motivation, social engagement, and depression. Whilst there are very likely a host of other legitimate reasons for students not engaging with the research and the GRIT programme, it would be interesting to learn if those that didn’t engage had lower self-efficacy scores. In light of this, the mean scores of students who only completed the term one survey were compared with the scores of students who only completed the term two survey (therefore only partially engaging with the research). As term two survey completers were made up of students who were late enrollers, and generally had experienced difficulty engaging with their course to begin with, it was hypothesised that a lower self-efficacy score may be more prevalent in these students. Whilst an independent t-test showed no difference (t=1.61, df= 53, p=0.06) this was marginal and perhaps with a larger sample size a difference in mean self-efficacy scores may be discovered.

In terms of the methodology used, the four point LIKERT scale design within the COGS survey may be limited in detecting more subtle changes in self-efficacy. Research by Johns (2010) concluded that results from LIKERT scales (and those with similar rating scales) becomes ‘significantly less accurate when the number of scale points drops below five or above seven’. Also other studies have reported higher reliability in five point LIKERT scales (Bouranta et al. 2009) and seven point LIKERT scales (Lewis, 1993; Finstad, 2010).

Another methodological difficulty is the reliance on participants’ self-reporting of their self-efficacy. Whilst this allows for straightforward and wide scale data collection across the student cohort, it assumes that participants will a) report honestly and not succumb to Social Desirability Bias and / or Acquiescence Bias, and / or, b) be self-reflective enough to fully understand their levels of self-efficacy. Evidence suggests that the ability to self-evaluate (along with other higher level cognitive functions such as problem solving and not giving into peer pressure) develops in the pre-frontal cortex of the brain during our mid-twenties (Arain et al. 2013). The FSF cohort had 89% of its students aged under 24, therefore it could be suggested that self-scoring of self-efficacy is not going to yield accurate results in students under the age of 25. Also, things that hinder this development within the brain include negative social influences, social isolation, substance abuse, and high levels of stress. These kind of issues may have affected a number of students on the FSF, as the course is designed specifically for those who may not have had an easy time in life.  The good news is that the brain is still flexible after this critical period, therefore individuals experiencing low levels of self-efficacy and other negative thought patterns can still do things to improve their way of thinking.

Going forward, it would now be interesting to see how self-efficacy might affect performance, therefore the plan is to compare final year grades with self-efficacy scores.

 

 

8a. The results are in! Part 1

Foundation Study Framework students are currently in the midst of completing their end of year assessments, and we wish them the best of luck with these!

Following on from a mostly positive review of the GRIT programme from those that attended (see blogs 6 and 7 for details on this), we were interested to know how well the GRIT programme’s own survey compared with the UoN’s own COGS survey that student’s also completed in term one and term two (see blogs 2 and 3 for more details on this). Both these surveys have been based on previously published self-efficacy scales and have previously both shown good internal reliability.

The GRIT survey consisted of 10 statements whereby students self-scored on a four-point LIKERT Scale, and a further 17 key skills items whereby students self-scored on how good they felt they were (see Appendix 1 below for full details).

All scales showed good internal reliability using the current data when tested using Cronbach’s alpha (COGS Survey= 20 items; α= .846; GRIT LIKERT Scale= 10 items; α=.829, and GRIT Rating Scale= 17 items; α= .883). The GRIT LIKERT Scale contained a mixture of positive and negative statements. Whilst this can reduce the chances of acquiescence bias (a tendency to agree with statements, especially when in doubt) from the respondent (Dodd-McCue and Tartaglia, 2010), it can cause problems for statistical analysis, particularly when measuring internal reliability (Salazar, 2015). Negative statements data was therefore reverse coded for analysis.

Interestingly, whilst there were a few weak to moderate correlations between individual questions, overall there was no significant correlation between the mean scores of the GRIT LIKERT scale and the COGS LIKERT scale (r (59)= .045, p= .728).

As self-reporting scales are particularly vulnerable to acquiescence bias (Dodd-McCue and Tartaglia, 2010), it could be recommended that the COGS survey incorporates a mixture of positive and negative statements rather than just positive to avoid potential this potential bias.  

However there was a moderate positive correlation between the GRIT Rating Scale and the COGS LIKERT Scale (r(59)= .414, p= .001). This could be because the statements within both surveys are more similar in their design. However there was a also a moderate positive correlation between the GRIT Rating Scale and the GRIT LIKERT Scale (r(59)= .480, p= .001) which contradicts the idea that surveys benefit from being of similar design. A final argument could be that the COGS survey and the GRIT surveys are based on two different previously published self-efficacy surveys, which could explain some of the discrepancies when comparing the data.

On a final note, no correlations were found between any of the three self-efficacy scores and the highest level of qualification students held. There was also no association with highest level of qualification held and whether  students engaged with the GRIT programme. However the majority (n= 42 (69%)) of students held level 3 qualifications, with only 4 (7%) students holding Entry Level qualifications, 4 (7%) students holding level 4 qualifications, and 11 (17%) holding level 2 qualifications. It could be argued that there is not enough data in the latter categories to identify any patterns.  

Appendix 1: GRIT Self-efficacy baseline survey

Four Point LIKERT Scale (Strongly agree, Agree, Disagree, Strongly Disagree) Rating scale (1-10, with 1 being Poor and 10 being Excellent)
On the whole I am satisfied with myself Self-esteem
I often think I am no good at all Self-confidence
I feel that I have a number of good qualities Speaking in front of other people
I am able to do things as well as most other people Setting and reviewing goals for the future
I feel I do not have much to be proud of Developing positive relationships with new people
I feel useless sometimes Ability to work with others
I feel that I’m a person of worth, at least on an equal plane with others Supporting others
I wish I could have more respect for myself Accepting others
I am inclined to feel that I am a failure Developing relationships with adults
I take a positive attitude toward myself Being on time
 

Outside of the LIKERT scale students were also asked ‘What’s the highest qualification you have got?‘ and given a choice ranging from Entry Level to Level 4, and ‘other- please specify’

Handling mistakes
Taking responsibility for your actions
Commitment
Keeping promises
Ability to solve problems
Seeing possibility
Dealing with challenging situations