The Relationship between Internal Forms of Engagement (Cognitive-Affective) and Academic Success across Years of Study

Numerous factors have shown to affect students’ academic success; namely, family background, financial aid, socioeconomic background (Hu & St. John, 2001); students’ effort (Bauer & Liang, 2003); self-motivation, age of student, learning preferences (Aripin, Mahmood, Rohaizad, Yeop, & Anuar, 2008); and entry qualifications (Kuh & Hu, 2001). However, one of the most important factors has been student engagement (Floyd, Harrington, & Santiago, 2009).

            The term “student engagement” has evolved from student involvement (Astin, 1984), quality of effort (Pace, 1979), and interaction/integration (Tinto, 1993). Today, it refers to the effort, interest, and time that students invest in meaningful educational experiences.

            Kuh (2003) defines engagement as the time and energy that students devote to educationally sound activities inside and outside of the classroom, and the policies and practices that institutions use to induce students to take part in these activities.

            Fredericks, Blumenfeld, and Paris (2004) discuss three dimensions of engagement that they had identified in the engagement- related literature at school level; namely, behavioral, emotional, and cognitive, and address the multiple interpretations of each component. From their perspective, behavioral engagement is defined as (a) "positive conduct, such as following the rules and adhering to classroom norms, as well as the absence of disruptive behaviors such as skipping school and getting in trouble," (b) "involvement in learning and academic tasks and includes behaviors such as effort, persistence, concentration, attention, asking questions, and contributing to class discussion," and (c) "participation in school-related activities such as athletics or school governance" (p. 62). Emotional engagement is characterized as affective responses such as interest, excitement, stress, and attitude (Fredericks et al., 2004; Marks, 2000). Some conceptualizations have also tied emotional engagement to students' sense of belonging and identification with school. However, some researchers have argued that the components of belonging and value should be defined separately due to confounding antecedents, such as family, educational context, and cultural influences (Finn, 1989, 1993). Finally, cognitive engagement is identified by "flexibility in problem solving, preference for hard work, and positive coping in the face of failure" (Fredericks et al., 2004, p.64), as well as the ability to use metacognitive skills to evaluate task requirements (Connell & Wellborn, 1991).

            Admitting the three dimensions of engagement at school level, Fredericks et al. (2004) argue against the split between literature that represented psychological investments in learning (concentrated focus despite distraction), and the study of cognition and strategic learning (represented by a student's effort exerted to meet and exceed requirements).

            Similarly, Appleton, Christenson, Kim, and Reschly (2006, 2008) citing the need for an empirically and theoretically refined student engagement construct, attempted to incorporate the essential components of the multi-dimensional construct of student engagement research into taxonomy of student engagement. Appleton et al.' s (2008) taxonomy of student engagement addressed both the absence of academic engagement, and the definitional ambiguity of the constructs of behavioral, emotional, and cognitive engagement. They argued that the inclusion of academic engagement in the taxonomy was important, as time on task and work completion correlated with student success (Appleton et al., 2008; Fredericks et al., 2004). From this perspective, academic engagement consists of variables such as time on task, credits earned toward graduation, and homework completion; whereas, and behavioral engagement includes attendance, suspensions, voluntary classroom participation, and extra-curricular participation.

            More specifically, cognitive engagement for Appleton et al. (2006) included (a) "self-regulation," (b) "relevance of schoolwork to future endeavors," (c) "value of learning," and (d) "personal goals and autonomy" (p. 429), which represented a narrowing of the extant definitions of the components (Fredericks et al., 2004; Jimerson, Campos, & Greif, 2003); whereas, affective engagement consisted of "feelings of identification or belonging, and relationships with teachers and peers" (Appleton et al., 2006, p. 429). These two components--cognitive and affective--represented what Appleton et al. characterized as the "internal forms of engagement" (p.429) signifying a student's sense of connection to school and to the role of school in the student's future.

            Agreeing with the classification cited above, the researchers based this study on Appleton et al.’s (2008) taxonomy. However, they extended the scope from school level to college/university level. In other words, the present study focuses on the impact of internal forms of engagement (cognitive-affective) rather than the external ones (academic-behavioral) over the academic years. That is, the researchers wondered if cognitive-affective engagements are as influential as the academic-behavioral types of engagement in boosting academic success, and if yes, whether or not engagement changes with the change in academic years?

            Such an investigation is important for at least two reasons. First, it adds to a growing number of studies testing the validity of links between engagement constructs, specifically cognitive and affective types, and academic success. Second, the study provides findings, conclusions, and recommendations for researchers and administrators who are interested in the quality of college education.

LITERATURE REVIEW

Research on student engagement has been the focus of substantial attention in the last few years, particularly in the USA (Fredericks et al., 2004; Kuh, 2001; Kuh, Kinzie, Buckley, Bridges, & Hayek, 2006), and Australia (Krause & Coates, 2008; Krause, Hartley, James, & McInnis, 2005). Although most of them were similar in their theoretical perspective--inspired by tenets of constructivism which believed that education is fundamentally about students constructing their own knowledge and learning in contexts that stimulate and encourage student involvement (Davis & Murrell, 1993)—they were different in terms of the results they have found for the school and college levels across years of study.

            For instance at school level, despite using the same instrument in their studies—Student Engagement Instrument (SEI)—the researchers came up with diverse results regarding the student engagement subtypes of cognitive and affective engagement in predicting educational outcomes. While Burrows (2010) supported the significance of these kinds of engagement in educational outcomes, Videen (2009) indicated a weak relationship between psychological engagement and success, and essentially no relationship between psychological engagement and growth in reading success. He maintained that psychological engagement did not add to the prediction of success after accounting for prior success, and that the relationship between psychological engagement and success, as well as psychological engagement and growth in success, did not vary by grade level (elementary compared to middle school). 

            A similar trend exists in the studies carried out at the college and university level. The instrument used by most studies in this context has been National Survey of Student Engagement or NSSE. Kuh (2001), building on decades of research (Astin, 1985, 1993; Chickering & Gamson, 1987; Pace, 1979), developed NSSE, and since then it has been widely integrated into higher education practices and policies in the United States. NSSE framework divides student engagement into five dimensions: level of academic challenge, active and collaborative learning, student-faculty interaction, enriching educational experiences, and supportive campus environment. Having used this instrument, most researches such as Hughes and Pace (2003) reported positive relationships between NSSE results and persistence, as well as academic performance represented by college grade point average (GPA). However, they did not agree about which academic year is more affected by student engagement. For instance, Krause and Coates (2008) underscored the  complexity  of  first  year  engagement  and  argued  that  the  first  year  experience  is pivotal  in determining university  students’ persistence with  their  studies,  and  engagement with peers,  faculty,  and  the learning  environment.  However, Zhou (2010) indicated that engagement did not have significant impact on student persistence at the end of first year. He further argued although engagement as represented by NSSE grand total scores was significantly correlated with student academic performance in the first year of college; the results are not the same when engagement was broken into its five categorical variables; i.e., none of the variables showed significant impact on academic performance. 

            Based on the literature review, it can be explained that—at college/university level—most studies have assessed engagement at the “macro level,” including the National Survey of Student Engagement (NSSE) at Indiana University (NSSE, 2000, 2002). In other words, the surveys merely measured engagement as a global quality that students have in relation to elements such as level of academic challenge and supportive campus environments. Therefore, research is required to treat engagement at the micro level—as a multidimensional construct composed of both external and internal indicators—each of which has to be examined in their own rights in relation to students’ academic performance.

            Besides, many studies such as those NSSE-based surveys have studied only freshmen and senior students that are in contrast with the findings of the earlier studies; i.e., the longer the students stay in college, the more internally involved they might be (Zhou, 2010). Therefore, it can be postulated that engagement may have different impacts on the students as the academic years pass.  In addition, a potential drawback is that first-year may be too soon in the college experience to start the investigation of the impact of student engagement on their academic development (Gonyea, 2006). For example, one study speculated that the effects of student-faculty interaction and peer interaction may not be fully evident until the junior or senior years (Bean & Kuh, 1984). To tackle this issue, this study included students from all levels - freshmen to senior - in the sample to compare their cognitive and affective engagement and their academic success. By doing this, more reliable conclusions will be drawn which may clarify the impact of cognitive and affective types of engagement on students at different stages of college life.

            Finally, Appleton et al. (2006) criticized studies such as the National Student Survey of Engagement, Community College Student Engagement Survey, and Australasian Survey of Student Engagement, for grouping subtypes together into an "amalgamation" that failed to provide clarity to the construct, and argued that the limitations in measuring cognitive and affective engagement have led to a dearth of research and difficulties in establishing comparisons of the subtypes across studies. In other word, assessing cognitive and affective types of engagement is challenging, because the indicators are more internal than observable or external. Additionally, instruments which measure cognitive and affective engagement are rare, and among those available, there is only one instrument (SEI) (Appleton et al., 2006) that measures the two internal subtypes of engagement at the school and not at college/university level. This underscores the need to develop instruments that can measure the internal types of engagement for college/university level.

PURPOSE OF THE STUDY

As the aim of this study was to examine the relationship between students’ cognitive and affective types of engagement and their academic success across the college years, the researchers had to develop their own self-report survey scale—the Cognitive and Affective Engagement Questionnaire (CAEQ)—as the data collection tool. Thus, the following research questions guided this study:

1. What is the relationship between the internal forms of engagement (cognitive-affective) and students’ academic success (represented by Grade Point Average- GPA)?
2. How does the pattern of the relationship between the internal forms of engagement (cognitive-affective) and students’ academic success change across years of study?

METHOD

Data collection was carried out in two phases: (1) Scale development, and (2) Scale implementation. First, the researchers—based on a thorough review of the recent literature—compiled a list of items related to the cognitive-affective engagement. Then, the list was context-tailored and expanded through the incorporation of the participants’ views on the cognitive- affective engagement. Experts from both language teaching and psychology were asked to evaluate and judge the plausibility of the items. Having modified the list, the researchers developed a closed questionnaire, which was pilot tested with 34 students. The list of retained items (n=67) underwent an exploratory factor analysis (EFA) so that the potential factors could be extracted. Having done this, the researchers asked 312 participants to complete the questionnaire for the final stage of scale validation. Details about the procedures are presented below.

Scale Development

First, the researchers made a comprehensive list of items by reviewing the relevant literature on engagement. To enrich and context-tailor the list, the researchers designed a two-item open-ended questionnaire (approved by two Iranian university instructors, experts in EFL and psychology) and administered it to the students from two intact EFL classes (N =25) similar to those in the target population. The questions—in the participants’ L₁ (Persian)—read: (1) what are the internal and external factors that affect your active learning? (2) How do you take advantage of the environment and the people around you for learning? The participants were given 15 minutes to answer the questions. They were allowed to skip any part they did not want to answer.

            Accordingly, an initial pool of 80 scale items was generated. This initial  list  was  evaluated  by  the same experts for  overlap, content  validity,  clarity,  and  readability. These experts were chosen because of their research background and teaching experience in engagement- related courses at graduate level. Thirteen items were identified not to be fit (either in terms of form or ambiguity), so they were discarded, and hence leaving the researchers with 67 items.  

            The 67 items were incorporated into a closed 5-point Likert scale questionnaire (1=strongly agree, 2=agree, 3=not applicable, 4=disagree, & 5=strongly disagree).  In addition to the 67 items, the scale required the participants to provide the following demographic information: age, gender, GPA and year of study. Although all of the items were rated as appropriate indicators of engagement by the experts, a pilot test was also conducted with 34 undergraduate students for additional feedback on the clarity of the items.

            To describe the reliability and internal consistency of items, Cronbach’s Alpha was calculated. Given that reliabilities of 0.7 or higher are considered sufficient for narrow constructs (Van DeVen & Ferry, 1980), the researchers concluded that the value of α = .85 obtained for this study was large enough to be considered acceptable.

            In order to determine the final composition of the instrument, and also to assess its psychometric characteristics (reliability and validity), the CAEQ was administered to a sample of 312 students who participated in the study. To ensure the quality of administration, the researchers themselves administered the questionnaire.

Validation Procedures

Using SPSS Version 17, the researchers conducted a principal component analysis (PCA) with orthogonal rotation (Varimax) on the 67 items.  Prior to performing PCA the suitability of data for factor analysis was assessed. The two main issues considered were: sample size, and the strength of the relationship among the items. Tabachnick and Fidell (2007) suggested at least 300 cases for factor analysis, so the sample size of the present study (312) was considered to be acceptable. Moreover, the Kaiser-Meyer-Oklin value of (.83) (Kaiser, 1970, 1974), and the Barlett’s Test of Sphericity (Bartlett, 1954) with the statistical significance p < .001 (P=.000), supported the factorability of the correlation matrix (Table 1).

Table 1: KMO and Bartlett's test

To decide what components to retain for further investigation, the researchers ran the principal components analysis on the data. The results indicated the presence of 20 components with eigenvalues exceeding 1, and the scree plot of a clear break after the second component. Therefore, based on Catell’s (1966) scree test, it was decided to retain two components for further investigation. This was further supported by the results of Parallel Analysis, which showed only two Components with eigenvalues exceeding the corresponding criterion values for a randomly generated data matrix of the same size (67 variables × 312 respondents).

            To aid in the interpretation of these two components, the researchers used Varimax rotation to maximize the dispersion of the loadings within factors. The rotated solution revealed the presence of simple structure (Thurstone, 1947), with both components showing a number of strong loadings. The two-component solution explained a total of 24.36 per cent of the variance, with Component 1 contributing 13.02 per cent and Component 2 contributing 11.34 per cent                        (Table 2).  The items that cluster on the same components suggest that component 1 represents cognitive engagement and component 2, affective engagement.

Table 2: Total variance explained

In the Rotated Component Matrix, the loadings of each of the variables on the two factors are presented. There were some cases (items 18, 21, 26, 38, 44) that one variable loaded on both components, so the researchers looked for the highest loading variables on each of the components. However, only item 26 with the higher loading on component two was considered as component one according to the definition. Moreover, a few items were excluded because they did not load on either components (items 2, 14, 33, 34, 36, 51) or had negative loading (item 61).

            In the final step, the researchers rechecked the items in both components.  There were some items with high loadings in each component which did not match meaningfully with the component. For example, some items which were loaded as cognitive engagement looked more like affective engagement. Consulting with the experts, subjective criteria were employed to determine the inclusion and exclusion of certain items in the subscales—criteria that took into consideration the face validity of an item rather than the tendency of factor loadings in a given subscale. Specifically, items that involved the problem of face validity or a major inconsistency even with high factor loading (n=20) were eliminated. The resulting make-up of the two subscales comprised twenty items on the cognitive subscale, and nineteen items on the affective subscale. As a result, the final version of the CAEQ was obtained which consisted of 40 items to be scored on a 5-point response scale ranging from 1 (strongly agree) to 5 (strongly disagree) (See Appendix).

Reliability

To determine the internal consistency of the 40-item CAEQ, Cronbach’s coefficient was calculated, yielding a reliability estimate of α =.91 for total scale. Cronbach’s coefficient was also calculated for each of the two subscales (cognitive: α = .86; affective: α = .83) (Table 3). According to DeVellis (1991), these α values suggest that the two subscales of the CAEQ have good internal consistency. Thus, the CAEQ as a whole was considered to be a measure of high internal consistency.

Table 3: Reliability

In addition, a series of α coefficients for the scale were computed with one item being deleted at a time. It was found that while in cognitive scale, the reliability would not improve if we had removed any of the items; in the affective scale, if question 32 was deleted the value of alpha would increase to 0.84. There were no further improvements to be made by removal of any other variables, so α = 0.83 was the best for affective scale.

Scale Implementation

Participants

A sample of 312 undergraduate students (freshman, 20%), (sophomore, 30%), (junior, 25%), and (senior, 24%) participated in our study. The students were doing their BA in English related disciplines in the universities of the northern province of Iran in the academic year 2012-2013. Of the total number of respondents, 28% were male and 71% were female. The mean age of these participants was 19.

Procedures

All participants were briefed on the procedures. They  were  informed about the way they could complete the CAEQ,  and the option they had --  they could withdraw  from  this research  at  any  time  if  they  wished  to. Having done so, the researchers administered the questionnaire, emphasizing the importance of honest responses to each of the items.  The participants were asked to base their responses on their college experiences. They were also requested to report their GPA accurately and according to the university records. In case they could not recall the exact GPAs, the researchers were allowed to consult the official records to access the required data. Finally, as discussed above, students’ responses underwent exploratory factor analysis to identify the underlying factors. The results of the correlational analysis are reported in the following section.

RESULTS

Preliminary analyses were performed to ensure no violation of the assumptions of normality, linearity and homoscedasticity. Exploring our data, we found 10 outliners: ID numbers 24, 17, 45, 88, 102, 66, 64, 88, 286, and 56, so we omitted them, and hence left with 302 participants. Then, to seek the answers to the research questions, the researchers used correlation analysis to explore the relation between cognitive engagement and success through four years of study (see Table 4).

Table 4: Descriptive statistics

As Table 4 indicates, the correlation coefficient for cognitive engagement increased through years of study. In other words, in the second half of college years, higher levels of cognitive engagement are observed. The correlation between cognitive engagement and success for first year students is r= –.136, while for senior students it is slightly higher, r= –.434. To find out whether the correlations for the two groups are significantly different, we tested the statistical significance of the difference between these two correlation coefficients. Having checked for the assumptions and securing no violation of independency of cases, the number of cases and normality of the distribution, we ran the test of significance and obtained z_obs value of 1.33. As this value is inside the specified range (–1.96 < z_obs < 1.96), we concluded that there is not a statistically significant difference in the strength of the correlation between cognitive engagement and success for first and fourth year students. In other words, although these two values seem different, this difference is not big enough to be considered significant.

Table 5: Correlations

To answer the second research question, an inspection of Table 5 suggests that although in case of the affective engagement, the correlation coefficient reached r=.179 only in the second year of study, and stayed below r=.05 in the other years, the pattern is not the same for the cognitive engagement; i.e., cognitive engagement progressively increased during the college years.

DISCUSSION

The results of this study were partly in line with NSSE-based surveys which agreed on different impacts of students’ engagement during academic years, and concluded a one to one relationship between college years and involvement (e.g., Zhou, 2010). More specifically, while like many other studies, this research confirmed that students were cognitively engaged through four years of study and the level of engagement increased up to the graduation, unlike them (e.g., Hughes & Pace, 2003), the results of this study did not indicate a total increase in cognitive- affective engagement in relation with academic success as represented by college grade point average (GPA) from the first to the fourth year of study. Besides, whereas the NSSE-based studies did not agree on which academic year was more affected by student engagement, this research revealed that cognitive engagement increased over the college years, with affective engagement reaching its peak in the second year. Thus, the study supports Gonyea (2006) in that first-year is too soon in the college experience to start the investigation of the impact of student engagement on their academic development and it should be continued up to senior years to gain reliable findings. In other words, although the first year experience, as Krause and Coates (2008) maintained, is pivotal in determining students’ engagement, the findings of this study suggest that higher level of cognitive engagement is expected for the senior students. This is in contrast with Zhou (2010) that significantly correlated engagement in the first year of study with student academic performance. In addition, the last outcome of this study contradicts the earlier ones, in that a higher correlation between affective engagement and academic success is observed for second year students who were more engaged than students of the other years. In fact, the results revealed that, the longer the students stay in college, the less affectively they get engaged. This finding is in line with Videen (2009) that indicated a weak relationship between psychological engagement of school students and their academic success.

Furthermore, the observation that only the second year students were affectively engaged partly supports Bean and Kuh (1984) who claimed that affective engagement may not be fully evident until the junior or senior years. The results of this study found that only in the second year students were affectively engaged. This finding though, may be explained with reference to the participants of this study.  While the CAEQ was administered to a sample of EFL students, the earlier research studied ESL students, who for a multiplicity of reasons may be more affectively engaged.

CONCLUSIONS AND IMPLICATIONS

This study was conducted to seek the relationship between students’ cognitive and affective engagement and their academic success during the college years. The findings from this study suggest the following conclusions.

            Although considerable progress has been made during the past few decades in improving the quality of engagement scales; limited efforts have been observed in the development of a more diagnostic instrument for this purpose. This new instrument; therefore, is to the researchers’ knowledge, the first which diagnostically measures cognitive and affective engagement at college level, because the other existing instrument (SEI) (Appleton et al., 2006) which measures the two internal subtypes of engagement does so only at the school level. Besides, unlike NSSE which groups subtypes together into an "amalgamation" according to Appleton et al. (2006), CAEQ is more eligible to help the clarity of the construct. In conclusion, the present study, by introducing a two dimensional measure, is a step forward in the assessment of students’ engagement at micro level, and as such it opens up new avenues for future investigation.

            In addition, as the level of students’ internal engagement is able to predict students’ academic success (Burrows, 2010), the CAEQ with its potential for research and diagnostic utility can have several practical applications in instructional settings. The two dimensional nature of the CAEQ has made it possible to investigate the relationships between college students’ cognitive and affective types of engagement and their GPA.

Although part of the study corroborates what many other researchers have found: Students cognitive engagement increases as the college years pass, the other part which shows the engagement level reaches its peak in the second year of college, is inconsistent with the earlier studies which claimed students’ engagement gets higher in the last years of study.  This might be explained in terms of the existing limitations of this study. This study, like many of its counterparts, has some limitations that must be taken into account when interpreting the findings. For one thing the single geographical region studied here could have led to the patterns which may not reflect what would occur at other colleges and universities. In addition, due to the nature of this study (being cross-sectional) the researchers were prevented to compare the patterns of change in the same participants from different academic years—a problem which could have been overcome, had the study been longitudinal. So until then a more conservative interpretation should be adopted. Moreover, defining success with reference to grand GPA might be potentially problematic. Thus, further research can examine the relationship between engagement and the GPA for each academic year. The last limitation points to the skewedness of the sample. The fact that the sample was composed of predominantly female students also points to the importance of verifying results of this study in samples of a more balanced male–female ratio.

Limitations aside, the findings of this research can inform colleges and universities to create programs, develop policies, and educate faculty members to build specific pedagogies that intensify internal forms of engagement (cognitive & affective) necessary for academic success. The results also behoove colleges and universities to examine whether they can make the years of study more challenging and satisfying for students to be engaged up to graduation. The underlying belief is that although institutions cannot change who students are when they start college, colleges with the right assessment tools can identify areas where improvements in teaching and learning will increase the chances that their students attain their educational and personal goals (Kuh, 2009). College/University administrators can implement CAEQ to assess the current level of student engagement. Based on the survey data, administrators and teachers can enhance, change, or eliminate current practices in order to increase students’ cognitive and affective engagements.