Effects Of Time Scale Focus On Improvement Of Nyna Essay, Research Paper

EFFECTS OF TIME SCALE FOCUS ON IMPROVEMENT OF DYNAMIC CAUSAL UNDERSTANDING by Dalton Ebun Manfred McCormack SUBMITTED TO THE DEPARTMENT OF INFORMATION SCIENCES IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF PHILOSPHY at the UNIVERSITY OF BERGEN Autumn, 1998 AcknowledgmentsMany people contributed to the development of this thesis. First I want to thank David Ford for his constant drive for perfection and his tireless efforts to instill in me the discipline of good research. He provided me with insightful feedback and suggestions during the entire project. I will also like to thank Paal Davidsen, Mike Spector and Jim Doyle for their kind assistance during the project. Special thanks to Jim Doyle and Khalid Saeed, Professor and Department Head, Social Science and Policy Studies W.P.I., Worcester, MA USA. for their assistance during the data collection exercise. I would also like to thank my colleague at the Department of Information Sciences, Jiang You for coding the online causal questionnaire. Joe, Hannah and Kobi thanks for your love and spiritual support during the raining days in Bergen. Above all I will like to thank God almighty for his continually guidance. Dalton E.M. McCormack22nd November 1998 Effects of Time-Scale Focus on Improvement in Dynamic Causal Understanding by Dalton Ebun Manfred McCormack AbstractManagers must understand the causal structure of complex dynamic systems to successfully control behavior of these systems. However the types of causal understanding required and missing to do so and the best means of improving causal understanding are not well understood. For example tools for improving causal understanding such as Management Flight Simulators (MFS) almost exclusively focus users on short time scales. Long time scale environments use one continuous simulation through the time horizon. Short time scale environments involve multiple decisions through the time horizon. The larger the number of decisions made over the time horizon the shorter the time scale. The length of the focus of managers in time can impact causal understanding. Recent research on MFS environments suggests that long time scale MFS environments can improve causal understanding better than short time-scale environments. I used two experiments to test this hypothesis and investigate the impacts of the size of a manager s time scale focus on two types of causal understanding. A disaggregation of causal understanding improvement results suggest that a long time scale helps to improve causal understanding of relatively simple causal relationships without delay more than a short time scale. I ran a second experiment to test the same hypothesis with several time scales and to investigate how the length of the time scale impacts causal understanding improvement. Results of the second experiment with four time scale lengths suggests that the degree of control and ability to perceive system responses to policies interact to affect causal understanding improvement. The results of these experiments suggest the need for more research into building effective system dynamics based learning environment tools. Table of Contents1. Introduction……………………………………………………………………………………………………..51.1 Motivation……………………………………………………………………………………………52. Problem Description………………………………………………………………………………………….73. Literature Review……………………………………………………………………………………………..83.1 Measuring Learning in Dynamic Environments………………………………………………….83.2 Learning and Decision Making Improvement in Dynamic Environments……………….104. Hypothesis………………………………………………………………………………………………………135. Methodology ………………………………………………………………………………………………….155.1 Treatment………………………………………………………………………………………………….165.2 The Experimental Tool………………………………………………………………………………..175.3 Measurement……………………………………………………………………………………………..175.4 Experimental Procedure……………………………………………………………………………….196. Results and Analysis of First Experiment …………………………………………………………….196.1 System Performance Results…………………………………………………………………………196.2 Causal Questionnaire Performance…………………………………………………………………216.3 Control Results…………………………………………………………………………………………..226.4 Quantitative Results ……………………………………………………………………………………246.4.1 Subject PGA (No. 10) …………………………………………………………………………246.4.2 Subject PGF (No. 15) ………………………………………………………………………… 256.4.3 Subject DGJ (No. 19) ………………………………………………………………………….266.5 Analysis …………………………………………………………………………………………………… 27 6.5.1 Further Analysis…………………………………………………………………………………..27 6.6 The Second Experiment………………………………………………………………………………..297. Discussion and Conclusion ………………………………………………………………………………….31References……………………………………………………………………………………………………………34Appendix 1 The Model…………………………………………………………………………………………..36Appendix 2 Participant Consent Form……………………………………………………………………….47Appendix 3 Questionnaire……………………………………………………………………………………….48Appendix 4 User Information…………………………………………………………………………………..51Appendix 5 Deer Management (Short Time Scale) ……………………………………………………..52Appendix 6 Deer Management (Long Time Scale)………………………………………………………58Appendix 7 The Kaibab Plateau Passage……………………………………………………………………64Appendix 8 Causal loop diagram of the Kaibab Plateau MFS Model………………………………69Appendix 9 Causal Questionnaire: Questions and Answers…………………………………………..70Appendix 10 Causal Questionnaire Code…………………………………………………………………..711. Introduction 1.1 MotivationSuccessfully managing complex dynamic systems is difficult and requires sound decision making strategies. Managers often work under pressure to implement decisions and lack tools for learning about the systems they manage. Management Flight Simulators (MFS) are tools used to help managers improve the management of complex systems by helping them understand the interconnected nature of systems and the consequences of their actions (Lane, 1995, Senge, 1994). Complex systems are made of multiple feedback loops, delays and non-linearity. Managers need help because they usually find it difficult to identify and understand the mechanism of feedback processes. Vennix (1996, 31) states that: … people tend to ignore feedback processes while at the same time feedback is a ubiquitous characteristic of life. As stated by Powers: All behavior involves strong feedback effects whether one is considering spinal reflexes or self actualization. Feedback is such an all-pervasive and fundamental aspect of behavior that it is invisible as the air we breathe…. we know nothing of our behavior but the feedback effects of our own outputs Powers suggested that its invisibility is one of the reasons why most people tend to ignore feedback processes. MFS can act as the central part of a learning environment in which managers are safe from the negative consequences of their decisions. Bakken, Gould & Kim(1994) describe MFS as: A MFS is a learning tool that allows managers to compress time and space, experiment with various strategies, and learn from making rounds of simulated decisions. Embedded in specially designed learning environments called learning labs, these simulators can be designed to provide organizational practice fields (Senge 1990) where managers can experiment and learn in environments that allow failure and reflection. Although there are many MFS in use their effectiveness in improving management performance has not been rigorously demonstrated. One reason for this is that researchers have found it difficult to understand how learners thinking processes change or how they are changed by the training intervention (MFS) (Bakken, Gould & Kim,1994). Another reason is the problem of measuring learning in dynamic environments which has not been easily solved. This is supported in the literature by (Bakken, Gould & Kim,1994): Measuring learning along certain dimensions is easier than along others. At a basic level, instruments can be used to measure a manager s ability to create a simple causal-loop diagram. At a more abstract level, determining the manager s understanding of a complete feedback structure and his ability to transfer that knowledge to another case is difficult to measure. Improving causal understanding of complex systems is required to improve the management of those systems. One reason why this is important to improve management of complex systems is that causality constitute a powerful way to express concisely causal statements and to identify feedback processes (Vennix, 1996). Causality embodies what Forrester (1975) refers to as closed-loop thinking. Vennix (1996, 43) states that: An open-loop approach to decision making implies that decisions are seen as unaffected by the decisions themselves (Forrester, 1975)…. In closed-loop thinking decisions are seen as a means to affect the environment, and changes in the environment in turn provide input to decisions which aim to influence the environment. MFS are potentially effective for improving causal understanding because managers can use MFS to experiment by testing several policies under different conditions. Designing effective MFS is central to the development of learning labs. Efforts to effectively identify the design features responsible for improved causal understanding with dynamic simulation have not been entirely successful (Langley, 1996). Spector and Davidsen (1997) report a lack of consistent success with regard to learning effectiveness in system dynamics based learning environments. They identify a learning strategy and reliable methods to measure learning effectiveness as critical aspects of future research. They note : Our first critical assumption is as follows: A simulation does not constitute a learning environment. Many people have and continue to confuse these two things. For example, persons responsible for planning and managing flight training are inclined to believe that building a physical flight simulator constitutes the creation of a learning environment. We believe that a simulator may be part of a learning environment, but for learning to occur efficiently there must be more than simple user interaction with a simulation, regardless of whether it is physical or computer-based. Frequent and constructive feedback from a tutor, especially in the early learning stages, is critical to learning (Collins, 1992). Explicitly stated learning goals and mechanisms to facilitate progress towards those goals (e.g., linkage to things already known, assessment of progress, helpful guides to improve performance, etc.) are also critical to learning (see, for example, Gagn , 1985; Hannafin & Hooper, 1993). Researchers have identified several characteristics of this problem which can explain why it has been difficult to improve causal understanding in dynamic environments. According to Bakken (1993) this is partly because experiments often involve dynamic environments which are unfamiliar to subjects, lack of relevant system information and require cognition beyond the bounded rationality of subjects (Bakken, 1993). Guidelines for the design of tools for improving manager s causal understanding of complex dynamic systems and protocols for how to change managerial practices which will improve causal understanding and thereby managerial performance are needed. These protocols should enhance managerial understanding of system structure, behavior and the interaction of the structure and behavior. Without ignoring structural and behavioral understanding, system dynamics has focused on the role of understanding how structure and behavior interact (Forrester, 1961; Richardson, 1991). One important aspect of this view studies the behavior of managers in response to system response and information. Hence I focus on the impact of different MFS designs on managerial understanding and performance behavior. Different time scale length have been suggested as effective design feature to investigate how this form of structure – behavior understanding enhancement impacts improvement in mental models and hence learning. To build effective tools researchers must understand the effects of multiple and different time-scales on causal understanding. As a step in addressing this issue I investigate the effects of time-scale focus on improvement in dynamic causal understanding. 2. Problem Description MFS potentially help managers by allowing them to safely experiment with different decision making strategies. An example of this is a manager who is unsure about the impact of potential changes in product pricing policies on customers. He can test policy alternatives and see the implications by implementing policy alternatives in a MFS which represents his organization and environment. Sustained improvement in developing pricing policies by the manager requires learning. Designing MFSs for learning is difficult because the necessary ingredients for learning in and about complex systems have not been completely tested. According to Sterman (1994) mangers fail to learn because of dynamic complexity, imperfect information about the state of the real world, confounding and ambiguous variables, poor scientific reasoning skills, defensive routines and other barriers to effective group processes, implementation failure and misperception of feedback. Ford (1997) describes the role of ambiguity and uncertainty in objectives as an additional barrier to learning. Researchers have shown that little learning occurs when decision makers are asked to manager dynamic complex systems. Bakken (1993) for example showed that managers learn better in familiar context than in unfamiliar contexts. He measured learning in two dimensions; decision making performance and improvement in causal understanding. Before we can design effective MFSs that impact learning, MFS developers need to understand what they want decision makers to learn about. Many designers are faced with the problem of what type of information they should include in MFSs and in what form these information should be presented. Senge & Sterman (1994) for example suggested the following three lessons for designing effective learning labs: focus on conceptualization, design opportunities for reflection and beware the computer (i.e. the computer should not be the focus of the learning exercise). I limit my focus on finding ways of designing opportunities for reflection. This problem of reflection is highlighted by Senge & Sterman (1994, 211) when they observed: In early tests of the simulation we found the manager-players were thoroughly engaged within fifteen minutes. They were, literally, on the edges of their seats. They argued with one another about the next decision. They bragged about cost reductions they achieved. But afterwards none could articulate a significant new insight about claims management [domain]. They had played to win without pausing to reflect or to formulate and test theories about the causes of the problem. These managers had fallen to the video game syndrome. How can MFS designers integrate opportunities for reflection into MFS? From the statement above we can conclude that one potential method of achieving opportunities for reflection is to design MFS that encourage managers to implement long-term strategies easily and see how these strategies affect system behavior. Langley and Morecroft (1996) suggest that one way of implementing this in MFS design is to focus on managerial time scales. Managerial time scales are defined as the number of decisions per system period over the time horizon. This research addresses the MFS design problem by testing and expanding the theory of the impact of managerial time scale focus on learning and dynamic decision making performance. The primarily research question is: How does managerial time scales impact causal understanding in a dynamic environment? 3.0 Literature ReviewSince I want to understand MFS design methods that impact learning in dynamic environments I first consider how researchers have measured learning as improvement in mental models. Next I consider how researchers have investigated methods to improve learners mental models in dynamic environments and the implications of these on designing effective MFS. 3.1 Measuring Learning in Dynamic EnvironmentsRelatively little literature directly addresses the design of learning oriented MFS due to a focus on improving dynamic decision making performance. One reason why most of the research in this area has being performance focused is that researchers have found it difficult to measure learning in dynamic environments. Researchers have measured learning in dynamic environments with changes in mental models (Bakken, 1993, Doyle, Radzicki & Trees, 1996). However this metric is problematic because of the meaning of the term mental models . Doyle and Ford (1997) suggest that although: … mental models are of central importance to system dynamics research and practice, the field has yet to develop an unambiguous and agreed upon definition of them . Some of these problems are highlighted in an experiment carried out by Doyle, Radzicki and Trees (1996) in which they measured learning as changes in mental models. Changes in mental models here is defined as changes in understanding of causal relationships in the system, which involves identification of important variables and strength of relationships between variables that are essential to the dynamic behavior of the system being managed. They found evidence of a positive relationship between MFS use and changes in mental models. Subjects in both control and experiment groups were asked to write down the most important variables and their relationships before and after MFS use. They found evidence that subjects in the group that interacted with the MFS changed their mental models of the system more than subjects in the other group. Although they measured dynamic decision making performance their research mainly concentrated on measuring changes in mental models. However, the researchers found it extremely difficult to interpret the data collected. Because mental models change quickly and deriving methods to collect and measure these changes is difficult. However this research based it s measurement of learning solely on changes in mental models (such as causal understanding) and not on dynamic decision making performance. This is because their primary focus were not on finding ways to improve dynamic decision making performance but rather on investigating whether MFS intervention impacts changes in mental models. This research shows that MFS can change mental models and hence learning. But whether MFS improve learning remains an open question that requires additional research. The problems of measuring mental models has led researchers to devise different methods of measuring learning. The most common metric is system performance. For example Sterman (1989) measured learning as improvement in decision making performance in a dynamic environment. However research by Bakken (1993) shows that learning is not always correlated with improvement in decision making performance. He did this by developing a methodology for measuring learning and the transfer of learning between context. He measured learning in two different forms; improvement in dynamic decision making performance and improvement in causal understanding. Causal understanding was measured by asking managers to provide answers to questions that relate to polarities of causal links of variables in the system. 3.2 Learning and Decision Making Improvement in Dynamic EnvironmentsSeveral researchers have investigated ways to design MFS for learning and decision making performance. Before I discuss some of these research I consider what Kahneman and Tversky (1982) suggest for learning in dynamic environments: Effective learning takes place only under certain conditions: it requires accurate and immediate feedback about the relation between the situational conditions and the appropriate response. The necessary feedback is often lacking for the decisions faced by managers…. because (i) outcomes are commonly delayed and not attributable to a particular action; (ii) variability in the environment degrades the reliability of the feedback, especially where outcomes of low probability are involved; (iii) there is often no information about what the outcome would have been if another decision had been taken; (iv) most important decisions are unique and therefore provide little opportunity for learning. One of the methods that researchers suggest can improve learning in dynamic environments is providing cognitive feedback to learners (Balzer et al,1989). Cognitive feedback refers to the process of presenting the manger information about the relations in the environment (task information), relations perceived by the manager (cognitive information), and relations between the environment and the managers perceptions of the environment (functional validity information) (Balzer et al, 1989). Task information includes structural information and how the structure of the real system and its behavior interact. It potentially helps managers to learn more about the structure of the system been managed. Langley (1996) suggests that task information is: …for example a decision rule [that] may be given for the price of a product, showing how the cues of historical demand and competitors price are weighted together. . Cognitive information refers to managers mental models and according to Langley (1996) cognitive information: …enables decision makers to gain greater insight into their own decision strategies. For example, the subject s decision rule for product price is estimated using the subject s own historical decisions. . Functional validity information refers to the relationships between the structure-behavior of the real system and the managers mental model. Langley (1996) suggests that: Functional validity information (allowing comparison of decision makers perceptions of cue-criterion relations to the actual situation) facilitates improved calibration with respect to the task structure. For example, a table is presented comparing the subject s own cue weights with the optimal cue weights. . Abdel-Hamid & Sengupta (1993) observed that two forms of cognitive feedback can induce improved performances in a decision making dynamic environment. In this research subjects were divided into three groups. Outcome feedback was provided to each group. In addition to that one group was given cognitive feedback (task information) and the other feed-forward (i.e. dynamic decision making rules or policies). Task information was provided by the system through plots of selected variables of the system. Feed-forward condition was provided with a set of guidelines used by experienced project managers (system s domain). The last group was given only outcome feedback. Subjects were then placed in decision making roles to mimic the job of a software project manager in a dynamic simulated environment. Abdel-Hamid & Sengupta (1993) concluded that subjects in the group with outcome feedback and cognitive feedback information performed best followed by the group with the feed-forward and outcome feedback information. They concluded that: The outcome feedback subjects fluctuated considerably from one interval to another in their staffing decisions, thereby reducing the productivity of the staff and paying the price through poor performance. The feed-forward and cognitive feedback subjects were more consistent. They fluctuated significantly less in their staffing levels and employed an effective staffing heuristic in controlling their staffing levels over the course of the project, there by attaining superior performance over the outcome feedback group. The cognitive feedback group outperformed the feed-forward group because their access to continuous diagnostic information gave them greater adaptability in recognizing system changes and calibrating their decision strategies accordingly. . Langley (1996) investigated ways in which cognitive feedback can improve performance and accelerate individual learning in a dynamic environment. More specifically he was interested in studying: …the principal question of how does the subject s performance changes, if we provide online cognitive feedback to the subject for a limited number of trials? Does it improve or worsen, relative to subjects who only receive outcome feedback? . His main hypothesis considers the improvement of performance of treatment groups (who received both cognitive feedback and outcome feedback) over the control group (who received only outcome feedback). He tested this hypothesis by conducting an experiment in which subjects were randomly divided into cognitive feedback treatment groups and control group. Subjects were required to complete a series of game related tasks, which involved playing the role of a group of independent oil producers, making yearly capital investment approval decisions, over 25 year period. The treatment and control groups receive various types of online cognitive feedback, and their performance in the tasks are compared. Improvement in performance is used as a surrogate for learning in the experiment. His results show that: … subjects mean performance, relative to a benchmark, was significantly higher (in trails 1 and 2) for subjects who received cognitive feedback. There were no significant differences in performance between subjects who received decision-rules cognitive feedback and subjects who received task-structure feedback. In fact, subject performance reached an upper limit in later trials (4,5,6) which was not improved by feedback treatments. This limit was 30% below a behavioral benchmark, and is consistent with prior work in experimental simulated environments. It was disappointing to find that subjects in the treatment groups did not perform significantly better than the control group by trial 6 . . Langley (1996) concluded that: The problem of how to increase the final level of performance, and overcome the difficulties of performing well in complex systems characterized by feedback, non – linearities and delays, is still a question that has not been fully answered. This is an important question, because there is much effort currently being expended on the design of electronic learning environments. Another example of a research to improve decision making performance and learning in dynamic environments was carried out by Machuca et al (1998). They stated that: … the main objective of this research was to create, develop and use TBBSs as well as to verify the hypothesis mentioned above. In my opinion, TBBSs should facilitate causal reflection and favor systemic learning of social and business problems, helping to prevent the video game syndrome which often arises in black-box games. The hypothesis tested was: … that by using a transparent simulation, the learning process and acquisition of a system approach for decision-making would be improved . More specifically they investigated the impact of Transparent Box Business Simulation (TBBS) and Black Box Business Simulation (BBBS) on learning. Learning was measured with a questionnaire which contained questions relating to the following: … basic knowledge, questions related to variables connected by relatively short and longer chains of interactions … questions linked to the [ subject s ] perception of increase in knowledge they believe they have attained through the simulation (BBBS or TBBS) in relation to different topics previously studied through traditional methods of learning.. Transparency refers to the possibility of relating the systems structure to its behavior when using a MFS. Machuca et al (1998) comment that : To facilitate transparency, the qualitative structure (for example, in the form of causal loop diagram) and even the main equations of the model on which it is based, could be at all times accessible in the computer to the user, who would therefore find it easier to make decisions based on previous study of the possible causes of the different behaviors of the variables and not only on observation of the latter, which are simply effects. This possibility, however, is not offered by black box games, which only allow access to the results that can be observed, that is, the effects or symptoms, but not the causal structure giving rise to them. With black box games a trial and error procedure is usually followed, based on the mental models each player has of the case being examined. Their results show that TBSS can have a positive influence on causal understanding especially the understanding of complex causal relations. They conclude that: in our judgment, the results from our experiment demonstrate the superiority of TBBSs to BBBSs in regard to the learning process, especially in relation to complex questions linked to an understanding of the structure and operation of the system under study. How much transparency is required for improved performance is not yet known, this research shows that managers can perform better in a transparent environment than a non-transparent environment.

All of these research sited in this section consistently investigate ways to learn in dynamic environments. They use MFSs as part of their research tools to investigate their hypotheses, however only Machuca et al (1998) showed specifically how TBBSs and BBBSs can be integrated into MFS. These research have identified some of the features of MFSs that constrained learning in dynamic environments. They suggest that a more structured methodology is needed to facilitate learning e.g. helpful guides to improve performance and constant feedback from a tutor. However with the exception of Machuca et al (1998) little work has been done to integrate these methodologies into the design of MFS. As suggested in my problem description section little work has been done to integrate the designing opportunities of reflection into MFS and to test how this impacts learning in dynamic environments. I investigate a methodology that potentially integrates this into MFS and observe how it impacts learning and improvement in dynamic decision making performance. 4. HypothesisIn this section, I discuss why designing MFS to enhance reflection should impact learning and decision making performance in dynamic environments. I also discuss why managerial time scales can potentially impact learning and decision making performance in dynamic environments. To better understand this we should view learning as a process where an action impacts result and this impacts reflection which leads to learning and this leads back to further action (Sterman, 1994, Bakken, Gould & Kim,1994). MFS can facilitate learning by shortening the delay between action and result, by forcing managers to search for a better understanding of why their decisions (actions) lead to the observed behavior (results). Sterman (1994) suggested that: Virtual worlds [MFS] are effective when they engage people [mangers] in what Dewey called reflective thought and what Schon (1992) calls reflective conversation with the situation. This process of learning is what researchers define as double-loop learning (Sterman, 1994, Bakken, Gould & Kim,1994, Argyris & Schon, 1978). Double-loop learning is defined as those sorts of organizational inquiry that resolve incompatible organizational norms themselves with associated strategies and assumptions. (Bakken, Gould & Kim,1994). Methods should be investigated to see how increasing reflection and enhancing learning can lead to better decision making performance. Senge & Sterman (1994, 211) stressed the importance of designing opportunities for reflection when they suggested: To enable mangers to experience the long-term side effects of decisions, simulations compress space and time. Good simulations also enable rapid trials with different strategies. But these very capabilities allow people to play without careful experimentation and without reflecting on the causes of the outcome. The players try a strategy; if it doesn t produce the desired outcome in a few months, they improvise. Rather than a series of controlled experiments, managers tend to vary multiple factors simultaneously. Instead of sticking with a strategy to see its long-term consequences, people quit a game that is going badly and start another (Moissis 1989) From the statement above we can infer that one potential method of achieving opportunities for reflection is to design MFSs that encourage managers to implement long-term strategies easily and see how these affect the behavior of the system. Senge & Sterman (1994, 212) suggests that: Before playing, the manager must state their strategy and what they expect to happen. . One potential method that researchers such as Simons (1990) and Langley & Morecroft (1996) have suggested which can impact reflection and hence learning is that of increasing managerial time scales relative to the systems natural period. They suggest that time scales which are relatively long when compared to the systems natural period facilitate causal understanding improvement more than shorter time scales. Langley & Morecroft (1996) stated that: Switching the mode of the user interaction with the microworld [ MFS ] from gaming [shorter time scale] to simulation [ longer time scale], allowing users to specify policies rather than decisions and run the model for continuous periods may improve their ability to identify high leverage policies. It is likely to allow more efficient use of time available, and an opportunity to apply the scientific method to systematically investigate the policy space. . They reason that mangers in longer time scales will have a longer time to design and test their strategies as well as to reflect on the behavior these strategies generate. They also suggest that managers in shorter time scales usually change their strategies during simulations and this causes them to have a different understanding of the system from managers in a longer time scale. We recognize this as a form of constructive experimentation i.e. mangers in longer time scale have more control of the system and thus can perform controlled experiments which should impact reflective thought (Sterman, 1994). Based on the preceding we propose the following hypotheses: H1a: Longer time scales in relation to the natural period of oscillation of the system will improve causal understanding more than shorter time scales. H1b: Longer time scales in relation to the natural period of oscillation of the system will improve system performance more than shorter time scales. 5. MethodologyI performed two experiments to test the hypotheses. In the first experiment the hypotheses were tested with a true experiment (Campbell and Stanley, 1963) with differing treatments in which subjects were randomly divided into two groups: long time scale and short time scale and asked to use a MFS to manage a small simulated ecosystem over a 40 year time horizon. Subjects in the long time scale group made only one decision per time horizon but subjects in the short time scale group made 40 decisions over 40 years. The managerial context was chosen to isolate the effects of time scale learning. Bakken (1993) has shown that using an MFS that is based on a domain with which subjects are familiar can have an influence on causal understanding and performance. Therefore I selected a dynamic environment with a relatively simple context and in which minimal domain knowledge is required of subjects. I also suspect that the bounded rationality of subjects is often exceeded by the complexity of MFS. Bounded rationality describes the limitations on human cognitive processing as central to understanding human behavior and performance (Simon, 1974). Significantly exceeding subjects bounded rationality can cause the loss of all treatment effects due to the overwhelming difficulty of the task. Therefore I used a relatively simple model of the ecosystem to not overwhelm treatment effects with system complexity. The simulated environment is based on a system dynamics model of the Kaibab Plateau (Goodman,1974; Roberts, Andersen, Deal, Garet and Shaffer, 1983; Sterman, undated). The specific version of this model has been previously used to study how information structures impact performance in a policy development environment (Ford, 1997). Ford s (1997) version of this classic system dynamics predatory-prey model (Goodman,1974; Roberts, Andersen, Deal, Garet and Shaffer, 1983; Sterman, undated) dynamically models three species (deer predators, deer and grass) and provides subjects with four annual control parameters: grass seeding, deer hunting, predator hunting and predator importation. Despite its structural simplicity the system remains dynamically complex with a frequency of two oscillations over the forty year time region of the simulation. The user interface of the groups were kept similar and simple, containing graphical and tabular information about some of the variables in the system in two frames which together fit legibly on a single page. 5.1 TreatmentThe difference between the two groups (the treatment) was that subjects in the short time scale group were required to make decisions annually (stepwise) throughout the forty year time horizon whereas those in the long time scale group were required to develop policies and to implement these policies at the start of each simulation. In other words subjects in the long time scale group were unable to stop the simulation after it has started. Since subjects in the short time scale group only implemented a policy once over the entire time horizon they were unable to change their policies during the simulation. Subjects in the longer time scale group however were able to change their policies since they implement their decisions annually. Group Pre-Test Treatment Post-TestLong Time Scale Causal Questionnaire Long Time Scale Tool Causal Questionnaire & PerformanceShort Time Scale Causal Questionnaire Short Time Scale Tool Causal Questionnaire & PerformanceTable 1 Experimental Design and Parameter Measures Subjects in two locations were used in the experiment to control for educational and environmental biases. The subjects in Bergen, Norway where paid a hundred Norwegian Kroner (about 13 US Dollars) for participating and an additional hundred Kroner was awarded to the subject with the highest average score in the causal questionnaire in each group. Subjects in Worcester, Massachusetts were awarded 50.00 US Dollars for the best average score, 25.00 for second best and 10.00 US Dollars for third best. The experimental design and measuring parameters are shown in Table.1. 5.2 The Experimental ToolThe Kaibab plateau MFS consists of a system dynamics model and a user interface. The system dynamics model consists of three species stocks (predators, deer and grass), two first-order delays and the connecting rates and auxiliaries with a natural period of approximately 30 years. A description and complete documentation equation listing based on Ford (1997) is included in Appendix 1 and file Kaibass on diskette 1 . Figure.1 shows the user interface of the Kaibab Plateau MFS. Subjects can see how the deer population unfolds over time in graphical as well as in tabular form (Figure.1). They can also see a record of their past decisions or policies on the table. Subjects also have a numerical performance indicator as shown to record system performance. Larger performance numbers indicate better performance. The interface is included in file Inter on diskette 1 . Figure. 1 The Kaibab Plateau MFS Interface 5.3 MeasurementQuantitative data about the dependent and control variables were collected as well as qualitative data about subject experience in the experiment. Two quantitative measurements were collected to measure system understanding: system performance and answers to a questionnaire. System performance was measured by how closely the subject maintained the deer population to the goal 30,000 deer over the time horizon with the aggregate variance over time. The causal understanding questionnaire (Figure 2) which was based on a similar experiment by Bakken (1993) contained questions about the causal relationships in the model. For example the answer to the question An increase in Grass Eaten by Deer leads to … in Grass (Figure 2) is Immediate Decrease since an increase in Grass Eaten by Deer will lead to an immediate decrease in the amount of grass in the plateau. The multiple choice questionnaire was computer based and contained 25 questions on causal relationships between variables in the system. The complete list of questions with solutions are in Appendix 9 and file question on diskette 1 . The order of the questions was randomized. The 25 questions were randomly selected and embedded into the computer code of the online causal questionnaire. This method was used to keep track of the questions and answers of each subject during both trails. Each question appeared on the monitor for a period of 25 seconds, followed by a new question. Each subject saw all 25 questions. The qualitative data collected included questionnaire with questions about how useful the description of the system they managed was, ease of causal questionnaire use, strategies used to manage the system and difficulty of the task managed (see also Appendix 2,3,4). Control data was also collected for subject s age, sex, primary language, English proficiency, education and experience background (see also Appendix 2,3,4). Figure 2 Example question in the Causal Questionnaire5.4 Experimental ProcedureBoth long time scale and short time scale groups were given a description of the Kaibab Plateau based on Goodman (1974) and the questionnaire. The description is included Appendix 7. The pre-test questionnaire was exactly the same for each group. Subjects in both groups were then asked to design and implement policies to control the deer population at a level of 30,000 for a period of 40 years. They used the four control parameters i.e. grass seeding, deer hunting, predator hunting and predator importation to do this for 20 minutes and saved each trial on a diskette. In each trial subjects type in their decision or policy on the input boxes and then click on the run button (shown in Appendix 5). The simulation starts at zero and then stops at the year 1940. At this point subjects start to implement their decisions or policies. At the end of the 20 minute session all subjects completed a post-test questionnaire. The post-test questionnaire was the same as the pre-test except that the order of the questions was changed at random. After the post-test questionnaire subjects were asked to fill out another questionnaire to gather data to control for variables such as age, educational background and experience in system dynamics and environmental studies. It also contained qualitative questions about the policy or decision rule the subject implemented during interactions with the MFS. 6. Results and Analysis of First Experiment In this section results of the first experiment will be described followed by an analysis of the results. The results are divided into four sections: System performance, causal questionnaire, control data and qualitative data results. 6.1 System Performance ResultsTable 2 and 3 show the data and results respectively of system performance of subjects in long time scale and short time scale groups. In Table 2 the columns contain subjects number, minimum system performance and maximum system performance. Table 3 shows the mean of the minimum system performance, standard deviation of the minimum system performance and the maximum of the minimum system performance for each group. Long Time Scale Group Short Time Scale Group Subject Minimum Maximum Subject Minimum Maximum1 1.04 2.63 1 1.83 1.842 0.63 1.85 2 1.83 1.933 1.73 2.66 3 1.80 1.844 0.70 3.06 4 1.62 1.865 1.49 1.85 5 1.04 2.826 1.80 1.87 6 1.75 2.157 1.75 1.83 7 2.06 4.048 1.84 1.94 8 1.41 5.639 1.04 1.79 9 1.30 1.8610 1.72 2.21 10 1.82 2.8111 1.85 1.85 11 1.96 3.7812 1.50 1.92 12 1.46 2.0613 1.84 1.92 13 1.46 3.0614 1.84 2.64 14 1.77 2.1515 1.85 1.85 15 1.38 3.0116 1.43 2.7 16 1.33 2.6517 1.84 2.76 17 1.58 1.8318 1.85 1.85 18 1.69 2.8519 1.8 1.84 19 1.64 2.24 Table 2 Subject Min and Maximum System Performance Data Long Time Scale Group Short Time Scale Group Mean of Min Standard Deviation of Min Max of Min Mean of Min Standard Deviation of Min Max of Min1.55 0.403902477 1.85 1.62 0.257311789 2.06 Mean of Max Standard Deviation of Max Max of Max Mean of Max Standard Deviation of Max Max of Max2.158947368 0.424380525 3.06 2.653157895 0.978911258 5.63 Table 3 Results of System Performance Data 6.2 Causal Questionnaire PerformanceTable 4 and 5 show causal questionnaire data and results respectively. Table 4 shows subjects pretest, post test causal understanding and causal understanding improvement (CUI) for each group. Long Time Scale Group Short Time Scale Group Subject Pre-Test Post-Test CUI Subject Pre-Test Post-Test CUI1 22 15.5 -6.5 1 11.00 9 -2.002 15 17.5 2.5 2 7.00 9.5 2.503 16.5 14 -2.5 3 13.50 12 -1.504 17 15 -2 4 13.00 12.5 -0.505 17 16 -1 5 19.50 19 -0.506 10.5 12.5 2 6 12.00 15 3.007 8.5 10.5 2 7 16.50 16.5 0.008 18 15 -3 8 19.00 19 0.009 15.5 18 2.5 9 11.00 11 0.0010 15.00 16 1 10 18.5 19.5 1.0011 12.00 17 5 11 18.5 18.5 0.0012 14.00 13.5 -0.5 12 15.5 18 2.5013 11 17.5 6.5 13 21.5 15 -6.5014 15.5 18.5 3 14 19.5 19 -0.5015 12 13 1 15 13.5 15.5 2.0016 13 16.5 3.5 16 14 16 2.0017 15.5 15.5 0 17 9 9 0.0018 18 19.5 1.5 18 14 18 4.0019 17 16 -1 19 21 21.5 0.50 Table 4 Subject Pre-Test and Post-test causal understanding Data for each group Table 5 shows the mean of pretest, post test causal understanding and causal understanding improvement for each group. It also shows the standard deviation of pretest, post test causal understanding and causal understanding improvement for each group. The minimum and maximum pretest, post test causal understanding and causal understanding improvement are also shown for each group. Long Time Scale Group Short Time Scale Group Mean of Pre-Test Standard Deviation of Pre-Test Min of Pre-Test Max of Pre-Test Mean of Pre-Test Standard Deviation of Pre-Test Min of Pre-Test Max of Pre-Test14.89 3.19104192 8.50 22.00 15.13 4.169472739 7.00 21.50 Mean of Post-Test Standard Deviation of Post-Test Min of Post-Test Max of Post-Test Mean of Post-Test Standard Deviation of Post-Test Min of Post-Test Max of Post-Test15.63158 2.235087031 10.5 19.5 15.4473684 3.901191901 9 21.5 Mean of CUI Standard Deviation of CUI Min of CUI Max of CUI Mean of CUI Standard Deviation of CUI Min of CUI Max of CUI0.736842 3.043063246 -6.5 6.5 0.32 2.28649736 -6.50 4.00 Table 5 Results of Pre-Test and Post-test causal understanding performance for each group 6.3 Control ResultsTable 6 shows results of the control variable data. The first column contains the subjects age, English proficiency, years after high school etc. English Proficiency was quantified along a scale of zero (lowest) to five (highest). The rows show for each control variable it s mean, standard deviation, minimum and maximum value for each group. Long Time Scale Group Short Time Scale Group Mean Standard Deviation Min Max Mean Standard Deviation Min MaxAge 24.333 4.4325003 18 34 Age 24.789 8.223131 17 53English Proficiency 4.5 0.6183469 3 5 English Proficiency 4.8421 0.374634 4 5Years after High School 4.6111 2.4528228 0 10 Years after High School 4.9342 4.301035 0.75 20General Management Education (Months) 7 13.941052 0 48 General Management Education (Months) 4.7368 9.005197 0 24General Management Experience (years) 0.4444 1.3382263 0 5 General Management Experience (years) 0.6447 2.293281 0 10Environment Management Education (Months) 0.9444 2.8382311 0 12 Environment Management Education (Months) 0.8158 2.794889 0 12Environment Management Experience (Years) 0 0 0 0 Environment Management Experience (Years) 0 0 0 0Computer Programming Education (Months) 14.889 23.166631 0 100 Computer Programming Education (Months) 8.6842 11.60485 0 36Computer Programming Experience (Years) 0.6639 1.5561836 0 6 Computer Programming Experience (Years) 0.3026 0.648604 0 2Modeling Information Systems Education (Months) 5.4444 8.1761649 0 24 Modeling Information Systems Education (Months) 4.0526 8.044831 0 26Modeling Information Systems Experience (Years) 0.0972 0.412479 0 1.75 Modeling Information Systems Experience (Years) 0.1053 0.315302 0 1Engineering Control Theory Education (Months) 0.4444 1.0966378 0 4 Engineering Control Theory Education (Months) 0.7895 2.800376 0 12Engineering Control Theory Experience (Years) 0 0 0 0 Engineering Control Theory Experience (Years) 0 0 0 0Systems Thinking or Dynamics Education (Months) 3.1667 6.8449208 0 24 Systems Thinking or Dynamics Education (Months) 4.6842 10.72408 0 47Systems Thinking or Dynamics Experience (Years) 0.0588 0.1660528 0 0.5 Systems Thinking or Dynamics Experience (Years) 0.0105 0.045883 0 0.2Table 6 Results of Control data for each group 6.4 Quantitative ResultsIn order to get some insight about the quantitative data results, three subjects (one from short time scale group and two from long time scale group) were selected based on their causal understanding improvement scores. These three subjects scored close to the average causal understanding improvement of each group. Table 7 shows their identifying number, age, causal understanding (CU) pretest and posttest scores, causal understanding improvement (CUI), average system performance (average MFS), maximum system performance (MAX MFS), minimum system performance (MIN MFS) and the group they represent. The subjects with identifying numbers PGA and PGF are subjects 10 and 15 respectively (see Table 4) of the long time scale group. The subject with identifying number DGJ is subject 19 (see Table 4) of the short time scale group. Long time scale group Identifying Number Age Sex (M/F) Primary Language CU Pretest CU Post test CUI Average MFS MAX MFS Min MFSPGA 34 F ENGLISH 15 16 1 1.89 2.21 1.72PGF 21 F ENGLISH 12 13 1 1.85 1.85 1.85 Short time scale group DGJ 19 F ENGLISH 21 21.5 0.5 1.90 2.24 1.64Table 7 Three quantitative subject data, with system performance data 6.4.1 Subject PGA (No. 10)When this subject was asked whether the passage (Appendix 7) was helpful in answering the causal questionnaire, she wrote: Helped in that it got me thinking about effects and their relationships on deer population. Did not help in that many of the terms used in the questions [causal questionnaire] were not used in the context of the reading. . On the question of the ease of causal questionnaire use; this subject wrote Easy . When asked about the difficulty of the task (i.e. controlling the deer population at 30,000), she wrote: Frustrating – was able to make improvement overtime but not completely clear – for instance if bringing predators each year does that just add up or does that include their natural growth/ breeding? Also I wanted to be able to set a midway policy as well to get the deer population up and then keep it stable. [This suggest that she would have preferred a shorter time scale.]. When asked about whether she would have preferred more time to interact with the MFS, she wrote. Yes, more time for experimentation would have tried more radical changes to see what happened On the question on what strategy she used while interacting with the MFS she wrote: try to look at curve given [suggesting that she used some form of outcome feedback ], see what makes it high enough then flatten out (by varying only 1 or 2 variables at a time) [this also suggest she used some form of controlled experimentation]. The last question asked about the usefulness of the MFS (i.e. did the interaction with the MFS help her improve her score in the second trial of the causal questionnaire?). She wrote: May have performed better [in the second trial] but I think it was because I had a better sense of what the questions were asking (terminology) after the game [i.e. MFS; this can be interpreted as MFS use helped her with domain understanding]. . 6.4.2 Subject PGF (No. 15)When this subject was asked whether the passage (Appendix 7) was helpful in answering the causal questionnaire, she wrote: It helped because it gave important background information. It also gave me an understanding of the problem. On the question of the ease of causal questionnaire use; this subject wrote: yes it was [easy]. The timer [this showed the time remaining before another question appeared on the monitor] caused me stress though. I don t know if I would include that. . When asked about the difficulty of the task (i.e. controlling the deer population at 30,000), she wrote: It was pretty difficult. I know something about SD [system dynamics] but this example (model) was very unsuccessful. When asked about whether she would have preferred more time to interact with the MFS, she wrote: No. The time allotted was perfect. I think it gives the user enough time to play the game [MFS], learn it and apply any policy changes to it. On the question on what strategy she used while interacting with the MFS she wrote: I tried to increase/decrease by small numbers in order to try to keep equality, not have a dramatic increase/decrease later on. The last question asked about the usefulness of the MFS (i.e. did the interaction with the MFS help her improve her score in the second trial of the causal questionnaire?). She wrote: Yes I think it gave me more insight to the problem and how things affected each other. 6.4.3 Subject DGJ (No. 19)When this subject was asked whether the passage (Appendix 7) was helpful in answering the causal questionnaire, she wrote: Yes it gave background information and a quick refresher in knowledge On the question of the ease of causal questionnaire use; this subject wrote yes [easy] . When asked about the difficulty of the task (i.e. controlling the deer population at 30,000), she wrote: I thought it would be easy but it was harder than I thought. The more I did it, the better I understood what worked and didn t When asked about whether she would have preferred more time to interact with the MFS, she wrote: Yes, once I got the hang of it, it was time to stop On the question on what strategy she used while interacting with the MFS she wrote: I was looking at the slope trying to manipulate the variables to achieve the line I wanted. [This suggests that she used some form of outcome feedback to implement her decisions] The last question asked about the usefulness of the MFS (i.e. did the interaction with the MFS help her improve her score in the second trial of the causal questionnaire?). She wrote: Yes it clarified some fuzzy concepts. 6.5 AnalysisTo check whether long time scale (LCUI) focus impacts causal understanding more than short time scale (SCUI), I calculated the improvement in causal understanding (i.e. the difference between the Post-Test and Pre-Test) for each subject in both groups. Statistical tests (using the two sample t-tests with P > 0.1) failed to support the hypothesis that p(LCUI ) > p(SCUI). No correlation between long or short time scale focus decision making performance and causal understanding, supporting a similar hypothesis tested by Bakken (1993). This result contrasts sharply with the intuitive belief that improved causal understanding improves performance. The results suggest that other necessary requirements for improved performance were not provided in ours or Bakken’s experiment. The analyses of the remaining results are presented in two sections. The first section reports summary statistics on aggregate levels and then uses them in the second section to test several hypotheses with the disaggregated data. Table 8 shows the means of the Pre-Treatment and Post-Treatment causal understanding and causal understanding improvement. It also includes the average of the best decision making performance for both groups. Group Mean Pre-TreatmentCausal Understanding Mean Post-TreatmentCausal Understanding Mean Causal Understanding Improvement Best System PerformanceLong Time Scale Focus 14.87 15.63 +0.76 2.16Short Time Scale Focus 15.13 15.45 +0.32 2.65Treatment Effect P >0.1 P >0.1Table 8 Means and Causal Understanding Improvement 6.5.1 Further Analysis To understand how the time scale focus impacts system understanding I disaggregated the causal understanding questionnaire data into two dimensions: degree of difficulty and type of understanding. There were three levels of difficulty: Easy, Medium and Difficult. The nine “Easy” questions involved one or two causal links between the variables (Appendix 8). The twelve “Medium” questions had three to five causal links and the four “Difficult” questions had six or more causal links between the two variables. I also divided the causal understanding questions into two types of causal understanding: links with immediate impacts and links with delayed impacts. The nine questions with “Immediate Increase or Decrease” as the correct answer are referred to as “links with immediate impacts” questions and the sixteen questions with “Delayed Increase or Decrease” as the correct answer are referred to as “links with delayed impact” questions (i.e. based on whether the linked passed through a stock). Although the option “No change” was given as an answer alternative in the questionnaire interface, no questions had “No change” as the correct answer. Table 9 shows the resulting hypotheses which can be tested based on this disaggregation. For example the hypothesis H2 states that increasing the time scale focus results in an increase in understanding of the easy causal links with immediate impact causal relationships. Degree of Difficulty Immediate ImpactCausal Relationships Delayed Impact Causal Relationships All Causal RelationshipsEasy (1-2 links) H2 H3 H9Moderate (3-5 Links) H4 H5 H10Difficult (6+ links) Not applicable H6 H11All Causal Relationships H7 H8 H1Table 9 Tested using disaggregated results of experiment 1. The cell marked Not Applicable indicates that this hypothesis was not tested because the causal questionnaire did not contain questions of this type. The results only support hypothesis H2 and H9 that increasing the time scale focus leads to an improvement in the understanding of the simple causal relationships (p Degree of Difficulty Immediate ImpactCausal Relationships Delayed Impact Causal Relationships All Causal RelationshipsEasy (1-2 links) p 0.1, r = – 0.007 p Moderate (3-5 Links) p > 0.1, r = – 0.2 p > 0.1, r =- 0.11 p > 0.1, r = – 0.15Difficult (6+ links) Not applicable p > 0.1, r = -0.35 p > 0.1, r = -0.35All Causal Relationships p > 0.1, r 0.1, r 0.1, r =- 0.03Table 10 Results of the Hypotheses Tested with Disaggregated results of Experiment 1 The disaggregated results show that 58% of the long time scale focus group improved their causal understanding more than the average short time scale focus group. This suggests that long time scale focus does have a positive impact on causal understanding than short time scale focus. However I also found that the long time scale focus group on average had a lower system performance than the short time scale focus group and only 21% of the long time scale focus group performed better in terms of system performance than the average short time scale focus group system performance. This does not suggest support of the hypothesis that long time scale focus has a positive impact on system performance. Although the main hypothesis was not statistically supported, the results indicate that a longer time scale focus may improve system understanding more than a short time scale focus based on the support of the hypotheses for easy immediate and all easy relationships (H2 and H9) results. This may be because the time spent using the MFS in both groups was too small (20 min) to cause a significant improvement in causal understanding or because other necessary learning factors were absent. This explanation was suggested by Prof. James Doyle of Worcester Polytechnic Institute, MA. It s possible that a more sophisticated MFS in which subjects can build feedback loops into the system such as was used by Ford (1997) can generate improvements in causal understanding. By allowing subjects to build feedback loops into the system they can better explore the policy space, and this can provide better transparency which as Machuca et al (1998) can lead to learning. Our results raised as many questions concerning the relationship between time scale focus and causal understanding as they provide answers. 6.6 The Second ExperimentBased on the findings of the first experiment a second experiment was designed as part of an exploratory work to investigate how different time scale lengths impact different forms of causal understanding. In this section I highlight some of the change