Analysing the Cumulative Hierarchy of the Taxonomy of Learning Objectives in Flipped Classroom

The emergence of digital learning formats influences the planning and structuring of digital teaching. Especially in times of the Corona Pandemic, when many universities remain closed, new digital learning concepts are emerging that can be integrated into face-to-face teaching in future. In this context, old teaching formats are often revised and questioned. But while technology only determines the form of collaboration, the real quality of learning depends on cognitive trials that the teacher addresses to the students. To classify these trials, a teacher can use Bloom's revised taxonomy, which ranks Learning Objectives in a six-level order and assumes a cumulative hierarchy: achieving a required Learning Objective level includes all lower levels. Especially in blended learning scenarios, such as a Flipped Classroom, this theory can be used to develop the course structure and to form exam questions. However, the applicability of the cumulative hierarchy is controversial in the literature and is rarely analysed in blended learning courses. Our goal is therefore to verify the cumulative hierarchy in a Flipped Classroom Course and derive recommendations for action. Therefore, we use a quantitative written survey. Since the analysis is based on the students' perceptions, these are verified by correlation analysis with the actual exam results and the awareness of contents and activities. Afterwards, the cumulative hierarchy is tested by regression analysis of the different levels of Learning Objectives. As a result, it could be confirmed for all levels, but not always by direct but often by indirect influences of other levels.


Introduction
The growing digitalization enriches all areas of daily life and increasingly reaches educational institutions. Especially since the emergence of the Corona Pandemic, the number of digital learning tools is continually rising and offers teachers numerous possibilities. But educational technology and virtual media only determine the form of collaboration, whereas the actual quality of learning depends on the tasks and the cognitive trials of the course (Bower et al. 2010). An essential requirement of the teacher is to plan and conduct the course (Ganzert et al. 2017). The first step of such a curricular design should be defining the Learning Objectives (LOs) and the Learning Outcome to be achieved (Bower et al. 2010;Ferguson 1998;Houlden and Collier 1999). To develop clear and unequivocal course goals and to enable their assessment, the teacher can use a classification of LOs (Bloom et al. 1956). A common taxonomy of Learning Objectives was developed by Bloom and revised by Anderson and Krathwohl, which has been used by multiple universities to analyze Learning Success so far (Bloom et al. 1956, p. 19;Krathwohl 2002). The taxonomy assumes that Learning Objectives can be placed in a cumulative hierarchy, in which the achievement of more complex LOs necessarily involves all lower LOs (Krathwohl 2002). Previous study results on the cumulative hierarchy are contradictory. On the one hand, they confirm the cumulative hierarchy. On the other hand, some studies report exceptions to such a numeric ranking (Kreitzer and Madaus 1994). Nevertheless, the classification of LOs was already used to implement different digital learning environments, such as the Flipped Classroom (FC) (Ganzert et al. 2017.;Zainuddin and Halili 2016), which is the basis of our research. Since blended learning concepts such as the FC enable teachers to address Learning Objectives systematically (Zainuddin and Halili 2016), the cumulative hierarchy can be used to design the course and derive useful exam tasks. Therefore, this paper aims to analyze the cumulative hierarchy of the revised taxonomy of Learning Objectives in a Flipped Classroom course. Since we use students' perceptions for this purpose, it must be primarily investigated whether these perceptions are suitable for testing the hierarchy. Therefore, we determine whether the students are aware of the learning content and activities they need to achieve and whether their awareness correlates with the LOs. Besides, the extent to which the expectation of attaining a Learning Objective correlates with the test results of the respective Learning Objective is determined. The first research questions (RQs) are:

RQ1:
Are the students in our FC case study aware of the necessary learning contents/ activities and the corresponding LOs?
RQ2: Do the test results of the individual Learning Objectives reflect the perception of the students from the FC case study?
After the suitability of the data has been established, the cumulative hierarchy is tested by examining whether the assumption that the Learning Objectives are cumulated applies to all levels of the hierarchy in the FC case study. Based on these results, recommendations will be derived for the practice application of the cumulative hierarchy in FC, which led to the following RQs:

RQ3:
Does an accumulation of the Learning Objective apply to all levels of the LO hierarchy in the FC case study? RQ4: What recommendations can be derived for the application of the cumulative hierarchy in an FC?
These results should provide teachers with guidance on how to design digital education. For this purpose, the results will be used to make recommendations for selecting and combining particularly suitable Learning Objectives within an FC. Furthermore, it is assumed that learners would benefit from a tailor-made approach to Learning Objectives in the digital age. A reduction of less effective Learning Objectives and a focus on relevant Learning Objectives promises efficient learning. To achieve our goal, we start in Section 2 with the theoretical foundations of Learning Objectives, the cumulative hierarchy and the features of blended learning formats such as the Flipped Classroom. Subsequently, we present the assumptions derived from the theory and the resulting hypotheses in Section 3. The data collection in our FC case study through a quantitative survey of students is then described in Section 4. Section 5 presents the results. Section 6 summarizes key findings, identifies the work's limitations, and provides then recommendations for teachers and an outlook on future research.

Theoretical Background
Learning Outcomes are the starting point for the course design, as they form the basis for the goals of the course (Ferguson 1998;Houlden and Collier 1999). Adam describe a Learning Outcome as "a written statement of what the successful student/learner is expected to be able to do at the end of the module/course unit or qualification" (Adam 2004). Thus, they focus on the student's abilities achieved after successful completion of the course. This abilities are set and communicated by the teacher (Harden 2002). In addition to the contents that need to be learned during the course, Learning Outcomes can also describe behavior to be develop (Tyler 2013). A widely spread specification of Learning Outcomes is the taxonomy of Bloom and Krathwohl, which could be used to formulate and classify Learning Objectives (Bloom et al. 1956). When developing the taxonomy in 1956, Bloom made a distinction between cognitive, affective, and psychomotor education objectives (Bloom et al. 1956). According to Bloom, such precise formulations that clarify the extent to which students should change through the educational process can be called educational objectives. In his first common handbook concerning the taxonomy of educational objectives, he focused on cognitive objectives, "which deal with the recall or recognition of knowledge and the development of intellectual abilities and skills. This is the domain which is most central to the work of much current test development […] and where the clearest definitions of objectives are to be found" (Bloom et al. 1956). Bloom classified the cognitive goals into six categories, from knowledge (category 1) to evaluation (category 6). He assumes a hierarchical formation of ascending complexity and argued that the categories are based on each other. For example, if higher and more complex categories are addressed, this automatically includes all underlying, simpler categories (Bloom et al. 1956). Anderson and Krathwohl found that there was an imbalance in Bloom's taxonomy, since in category 1, as opposed to the other categories, two dimensions were addressed: on the one hand, the specific content, identifiable by a noun, and on the other hand, the description of an action, identifiable by a verb. Thus, they adapted Bloom's Taxonomy by splitting category 1 into two dimensions: the knowledge dimension describing the specific content and a cognitive process dimension, describing the necessary action. While the knowledge dimension consists of factual, conceptual, procedural and metacognitive knowledge, the cognitive process dimension is based on the six categories of Bloom's taxonomy. After renaming some categories, changing their order, and formulating them as verbs, Krathwohl proposes new terms for the six categories, as shown in Figure 1   Like Bloom, Krathwohl also considers the cognitive process dimension to be a hierarchy from simpler to more complex categories and assumes that the "mastery of a more complex category required prior mastery of all the less complex categories below it" (Krathwohl 2002). Krathwohl also points out that the categories in the cognitive process dimension can overlap. (Krathwohl 2002) Previous research on a cumulative hierarchy have produced different results. While some studies confirmed the cumulative hierarchy, others found exceptions to such a ranking when analysing students' study results (Kreitzer and Madaus 1994). For instance, Kropp and Stoker prove a cumulative order only for the levels Remember, Understand, Apply, and Analyse by investigating various studies in science and social classes (Kropp et al. 1966).
Also, Ormell analysed some exceptions from the cumulative order and argued that certain knowledge requirements are more complex than specific analysis and evaluation requirements (Ormell 1974). Nevertheless, the hierarchic order of Learning Objectives is continued in the revised taxonomy by Krathwohl (Krathwohl 2002). To address the Learning Objectives in a systematic manner und thus take advantage of the cumulative hierarchy in designing the course and exam, blended learning formats like the Flipped Classroom can be used. In an FC, teaching structure is changed and can be divided into online-and in-class-time. In online-time, materials such as videos, podcasts, and self-tests are provided to the students (Bergmann and Sams 2012). This enables students to acquire basic knowledge before students and teacher come together face to face. (Handke and Sperl 2012). Thus, in-class time can be used to deepen knowledge. However, the real strength of the Flipped Classroom doesn't lie in the use of digital media, but rather in the possibilities that arise for face-to-face teaching: The FC offers a division of the achievement of Learning Objectives (Little 2015). Thus, the online-time can be used to reach lower LOs such as Remember, Understand and Apply (Zainuddin and Halili 2016). In contrast, during the in-class-time, higher Learning Objectives (Analyse, Evaluate, Create) can be addressed through different classroom activities, such as group discussions or project presentations (Krathwohl and Anderson 2010;Zainuddin and Halili 2016). Previous research showed that this change of focus led to increased learning success and motivation (Giannakos et al. 2014). In the FC, the expectations about learning objectives' hierarchical order are diverging; some researchers support the assumption, others reject it (Dali 2019;Kvashnina and Martynko 2016;Munzenmaier and Rubin 2013). Regarding the Corona Pandemic, the FC could be a suitable course form that enables the newly created digital documents to be integrated into future face-to-face classes.

Analysis
To analyse the cumulative hierarchy of Learning Objectives in our course, we dissect the competencies into their constituent parts. For this purpose, we assume that a category of competencies needed at different taxonomy levels according to Bloom and Krathwohl can be measured numerically. The categories of competences are in the following described as ∈ ℕ. The ∈ {1, … , 6},is an index for all levels of the cognitive LOs.
is the sum of all aggregated competences at level , while is the core competence of the LO at the level and , ∈ {1, … , K − 1} is an index for all levels, which are smaller than, . The core competence of level 2 e.g. is the difference in competence to not just remember, but to understand the contents, which is 2 − 1 . The aggregate competence 2 , on the other hand, is the sum of all competences to achieve the second level Understand, which includes 1 Remember as well according to the cumulative hierarchy. For the category Remember of the first level, 1 is similar to the core competence 1 , since there are no lower categories than Remember. The following conditions results from the assumption of a cumulative hierarchy: (1), = ,,, ,,, = 1, The competences of a learning objective at level are thus composed of the core competence of the learning objectives and the sum of the core competence of all arranged lower LOs, e.g. 3 = 1 + 2 + 3 : To achieve the third LO-level Apply ( 3 ), the students must be able to achieve the core competences of Remember ( 1 ), Understand ( 2 ), and Apply ( 3 ).
In this paper, we do not focus on the proportion in which core competencies affect the aggregate competences. Moreover, we want to investigate, whether is affected by the core competence of all lower levels of Learning Objectives. Testing the cumulative hierarchy, we decide not to use actual Learning Success in the exam since the exam would have to put the different levels to the same content in the sense of comparability and such an exam structure is difficult to implement in practice. Thus, six questions on the same content would not be sensible. On the other hand, such a structure does not allow to measure whether a Learning Objective could be reached independently from the lower LOs: In an exam, where the first task is Remember and the second Understand, it is not possible to prove whether Understand can be answered independently of Remember, because it is not clear whether the pure attempt to answer Remember previously has already influenced the answers to the task Understand. For our analysis, we, therefore, consider the students' perceptions about whether they feel able to reach the different levels according to Bloom and Krathwohl after the course. Since we decide not to use actual exam results, a verification of the student's perceptions is necessary. This verification in considered in two steps: First, it is tested whether students believe to understand, what is meant by the Learning Objective. As shown in the previous Section, LOs are composed of the contents and activities to be learned, which led to the following hypothesis: H1a: There is a correlation between awareness of learning content and cognitive LOs.
H1b: There is a correlation between awareness of learning activities and cognitive LOs.
To test hypotheses H1a and H1b, we analyse the correlation between student's perception to have reached cognitive LOs during the course and the awareness of contents and activities. These correlations are shown from the ten percent significance level upwards. In the second step of verification, we compare the perceptions with the exam results per LO-level. For this purpose, we also use a correlation analysis considering a significant level from ten percent upwards. This correlation analysis is used to answer hypotheses H2a-H2f: H2a: There is a correlation between the student's perceptions about 1 Remember and the exam results concerning the level 1 .
H2b: There is a correlation between the student's perceptions about 2 Understand and the exam results concerning the level 2 .
H2c: There is a correlation between the student's perceptions about 3 Apply and the exam results concerning the level 3 .
H2d: There is a correlation between the student's perceptions about 4 Analyse and the exam results concerning the level 4 .
H2e: There is a correlation between the student's perceptions about 5 Evaluate and the exam results concerning the level 5 .
H2f: There is a correlation between the student's perceptions about 6 Create and the exam results concerning the level 6 .
After verifying the student' perceptions, the cumulative hierarchy can be tested in the last step. Therefore, the following hypotheses are derived:

H3a:
The aggregate competence of the second level 2 Understand includes the core competence 1 Remember.

H3b:
The aggregate competence of the third level 3 Apply includes the core competencies 2 ,Understand and 1 ,Remember. H3c: The aggregate competence of the fourth level 4 Analyse includes the core competencies 3 ,Apply, 2 ,Understand and 1 ,Remember. H3d: The aggregate competence of the fifth level 5 Evaluate includes the core competences H3e: The aggregate competence of the sixth level 6 Create includes the core competences 5, ,Evaluate, 4 ,Analyse, 3 ,Apply, 2 ,Understand and 1 ,Remember. H3a -H3e address equation (2) under consideration of (1).
To test H3, the students' perceptions of their achieved competence are analyzed. Using regression analysis, the aggregated competences are estimated for 2 ≤ K ≤ 6. A total of five regression analyses are predicted with 2 to 6 as the target variable and the lower aggregated competencies as possible predictors. The results are considered at the ten percent significance level.

Data Collection
As part of the study, the course "Fundamentals of Organization" participants are examined in the summer term 2019. This course is a mandatory part of the study program for economic and business informatics students. Law students can also attend it as an elective course. The examination is carried out on a weekly course unit, which was designed as an FC. In our course, a learning management system was used in FC's online-time, which included videos and quizzes to test comprehension. These quizzes ranged from level Remember to Apply. The activities used in the classroom, on the other hand, addressed the higher LOs from level Analyse to Create. For this purpose, we discussed different case studies, which cover the level Analyse. Subsequently, a fictive student's elaboration of an organisational configuration was to be evaluated using different criteria, which corresponds to the level Evaluate. Finally, the level Create was addressed by instructing the students to use the learned elements independently to develop an organisational configuration in group work. The survey was conducted at the end of the FC unit during the semester and consisted of 18 items. A 5-point Likert scale was used for each item, whose answers range from "I strongly agree" to "I strongly disagree". We assume an interval scaling of the data since we explicitly formulated at the beginning of the survey that the distances between the individual scales are to be treated as equal. A total of 118 students took part in the quantitative, written, and voluntary survey. In the first part of the survey, we asked whether the students believe in reaching each level of the cognitive Learning Objectives we addressed in the course. Based on the cognitive process dimension of Bloom and Krathwohl, we evaluate the following cognitive Learning Objectives in our case study: Remember ( 1 ), Understand ( 2 ), Apply ( 3 ), Analyse ( 4 ), Evaluate ( 5 ) and Create ( 6 ). For each cognitive Learning Objective, we ask for the student's perception, whether they feel able to achieve the Learning Objective after the course unit. The second part of the survey consists of the perceptions of being aware of contents and activities, asking if the students believe to be aware of the contents and activities they have to internalize during the course in order to learn. The third part of the data collection is considering the exam results at the end of the semester. Since students of law only need a certificate of attendance and no examination results to pass the course and the matriculation number entry was voluntary, there are 93 data records where the exam was written. A link between exam results and survey entries was agreed.

Results
To test H1a and H1b, the correlation between the awareness of contents and activities needed to learn, and the six levels of Learning Objectives is observed. Table 1 shows the correlation coefficient , the significant level ∝ and the six level of Learning Objectives shown as student's perceptions. A high correlation can be understood as an awareness of the contents or activities required for learning to achieve the particular Learning Objective. As shown in Table 1, for both the contents and the activities, there is a correlation with each level. Except for 6 Create, the correlation coefficient between the awareness of contents and the LO decreases the higher the LO level is. For the awareness of contents, the correlations are significant to the one-percent significance level for 1 -4 , whereas the one-percent significance level applies to the awareness of activities only from 1 to 3 . Since there is a significant between the awareness of contents and activities and all levels of Learning Objectives, hypotheses H1a and H1b are both confirmed.
In H2, it should further be examined to what extent the students' assumptions about the achievement of the different Learning Objective levels correspond to the actual exam results. This concerns the extent to which students are able to predict their learning success after the course. For this purpose, the correlation coefficients are compared accordingly in Table 2. All correlation coefficients shown above are significant at least to the ten percent significance level. The rows show the students' perceptions, and the columns show the actual exam results. Moreover, the area marked in grey indicates where no correlation is expected according to the theory of cumulative hierarchy. It first becomes apparent that for levels 1 -5, there is a correlation between assessment and exam results for each Learning Objective. This correlation is strongest for the Learning Objective at level 5 Evaluate. However, no correlation could be established for 6 Create. For some of the levels of Learning Objectives, a correlation with higher levels of exam results is also evident. However, since the exam results relate to different tasks, it is difficult to interpret this effect. Although it can be assumed that the correlations shown are valid for all LOs, there is no claim to completeness since there is not always a task with the same content at a lower level. It is noticeable, though, that the assessments of whether students feel able to analyse the course contents correlate with the exam results of the level Apply. According to the cumulative hierarchy, such a correlation should not exist. This could be one of the cases mentioned by Krathwohl, where LO levels overlap (Krathwohl 2002). Nevertheless, the hypotheses H2a, H2b, H2c, and H2d are confirmed. H2e, on the other hand, must be rejected.  Finally, H3 should test whether the lower Learning Objectives could be used to explain the LOs as shown in equation (2), considering the student's perceptions. To test this cumulative hierarchy, a regression analysis for 2 -6 is estimated. As mentioned above, we only analyse the relevance of all lower LO and assume that the aggregates competence always necessarily includes the core competence . The 2 and F-values of the regression models are shown in Table 3 to 6. Table 3-6 also show the ANOVA-significances, the significance levels of the beta coefficients, the coefficients themselves and their T-values. The regression analysis to explain 2 and 3 are presented in Table 3.

Conclusion
In our analysis, we investigate the cumulative hierarchy of Learning Objectives in a blended learning format. Therefore, we divide the six level of Learning Objectives by Bloom and Krathwohl into its core competencies. We call the sum of all core competencies needed to achieve a level of competencies an aggregate competency. According to the cumulative hierarchy theory, all core competencies of lower levels of LO are the requirements to achieve a Learning Objective. Thus, the achievement of an LO necessary includes the achievement of all lower LOs. To test this condition, we used students' perceptions about the ability to reach the different levels of LO in a blended learning format after course. For these perceptions, a correlation between the awareness of contents and activities could be confirmed all levels of the Learning Objectives. Besides, all LOs except 6 Create correlate with the actual exam results of the corresponding level. Therefore, we assume the perceptions of students as suitable to analyse the cumulative hierarchy of Learning Objectives. For the sixth level Create, however, the missing correlation between the exam results and the perceptions has to be considered. Testing the cumulative hierarchy in our case study, we could confirm a hierarchy structure for all six level of Learning Objectives, since all lower levels of LOs could explain the aggregate competencies. Nevertheless, some of the core competencies only indirectly influence the higher level of aggregated competencies. The most direct influences on higher level are found for the first level Remember, and the third level Apply. However, the second level Understand only influences the third level with a weak effect strength and thus has the smallest influence of all LOs. These results can be used to address the Learning Objectives systematically. We argue that LOs should primarily address the hierarchy levels that have a direct positive impact on each other. Accordingly, LOs of different levels should be deliberately combined to address a specific topic. This would allow skipping certain levels, which could lead to more efficient learning and teaching. For example, a subject area could be addressed by the LOs sequences: C1-C3-C6. Since the results show that C1 has a strong influence on C3 and C3 has a moderate impact on C6, it can be assumed that higher LOs may be achieved without reaching certain lower LOs in advance. This systematic use of LOs leaves teachers more time to fill their teaching process with useful Learning Objectives, which could optimize students' learning process. Also, the exams can be designed more effectively. Since the competencies of lower LOs are included in the higher ones, exam tasks could be focused on the higher levels of Learning Objectives. Remarkable is the fact that there is a correlation between assessment and exam results for each LO, except for C6 (Create). Since this is the most challenging LO, students may find it particularly difficult to assess this LO. It is also conceivable that the students could not comprehend what exactly is meant by the LO Create.
To avoid ambiguities, teachers should explain the different levels of LO at the beginning of a course using examples. It is also important for teachers to clearly formulate the Learning Objectices. However, these results refer only to the case study we conducted in our Flipped Classroom described above. Since each blended learning course has different conditions and a different structure, the results can only conditionally be transferred to other classes. This case specificity is the most significant limitation of our work. Other factors, such as the course size or the lecturer, could influence the results. Besides, the lack of correlation between the exam results and the perceptions for the highest LO level makes interpretation difficult. Further studies could examine the cumulative hierarchy for other circumstances and identify other influencing factors for the achievement of Learning Objectives. It could also be investigated whether the division of Learning Objectives in the different phases of a Flipped Classroom, as described above, actually leads to the achievement of all six levels. Furthermore, studies in the form of experiments or laboratory studies would be conceivable, in which tasks about different levels of learning objectives are asked of other experimental groups.