ࡱ > L 7 bjbjUU % 7| 7| -
l & & L X l n n n 8 n $ p T l B n Lw `| " | | | )~ Z ? ` O Q Q Q Q Q Q $ о h u %~ )~ u C | | L C C C Ł B | | C O C f C r O T h O | @w @ [7l $g n
O \ ֺ l B 8 2 8 O C l l
User Frustration with Technology in the Workplace
Jonathan Lazar1, Adam Jones1, Katie Bessiere2, Irina Ceaparu3, and Ben Shneiderman3
1Department of Computer and Information Sciences, &
Center for Applied Information Technology, & Universal Usability Laboratory
Towson University, Towson, Maryland, 21252
2Human-Computer Interaction Institute
Carnegie-Mellon University
Pittsburgh, PennsylvaniaA, ??????15213
3Department of Computer Science, Human-Computer Interaction Laboratory,
Institute for Advanced Computer Studies & Institute for Systems Research
University of Maryland, College Park, Maryland 20742
E-mail: HYPERLINK "mailto:jlazar@towson.edu" jlazar@towson.edu; HYPERLINK "mailto:ajones5@towson.edu" ajones5@towson.edu; HYPERLINK "mailto:katieb@cmu.edu" katieb@cmu.edu; HYPERLINK "mailto:irina@cs.umd.edu" irina@cs.umd.edu; HYPERLINK "mailto:ben@cs.umd.edu" ben@cs.umd.edu
Keywords
Frustration, Usability, Technology Acceptance, Cost justification, Interface design, User satisfaction, Human-Computer Interaction
Abstract
When hard to use computers that are hard to use cause users to become frustrated, itthis can affect workplace productivity, user mood, and interactions with other co-workers. Previous research has examined the frustration that graduate students and their families face in using computers. To learn more about the causes and effects of user frustration with computers in the workplace, we collected modified time diaries were collected from 50 workplace users, who spent an average of 5.1 hours on the computer. In this experiment, users reported wasting on average, 42-4328-35% of their time on the computer due to frustrating experiences. The causes of the frustrating experiences, the time lost due to the frustrating experiences, and the effects of the frustrating experiences on the mood of the users are discussed in this paper. Implications for designers, managers, usersinterface designers, and employees a, information technology staff, and policymakers are discussed.
Introduction
A September 2001 recent report released byof the National Telecommunications and Information Administration reports that as of September 2001, 56.7% adults (employed and over age 25) in the United States use a computer in the workplace ADDIN EN.CITE NTIA20017710NTIA2001A Nation Online: How Americans are Expanding Their Use of the InternetWashington, DCNational Telecommunications and Information Administration(NTIA, 2001). In addition, 81% of those employed in managerial and professional jobs, and 71% of those in technical, sales, and administrative support jobs utilized computers as part of their work environment. This indicates that in white-collar jobs, computer use is becoming very prevalent.
With the rising ubiquity of computer usage in American society in the home, school, and workplace, research has begun to focus on the possible consequences of such use. Research on computer anxiety, attitudes, and frustration has shown that a disturbing portion of computer users suffer from negative affective reactions towards the computer, which can subsequently affect whether or not they use the computer, and whether or not they use the computer effectively. Research on frustration, both in individuals and organizations, has shown that frustration can lead to maladaptive behaviors that can subsequently lower effective goal-oriented behavior. In addition, research has shown that between one third and one half of the time spent in front of the computer was lost due to frustrating experiences -- when considering both the time it took to fix the problem and any additional time that was lost due to the problem ADDIN EN.CITE Ceaparu2002650I. CeaparuJ. LazarK. BessiereJ. RobinsonB. Shneiderman2002Determining Causes and Severity of End-User Frustrationforthcoming(Ceaparu, Lazar, Bessiere, Robinson, & Shneiderman, 20032).
Because computers are so prevalent in organizations, it is important to examine the role of computers in the organization and the possible consequences arising from their use. In this experiment, 50 workplace users recorded their frustrations with computers through the use of modified time diaries. There are solutions to the causes of user frustration.these are not impossible to solve. However, the first step is to understanding the causes themselvesof user frustration, which can lead to experimental testing of improved interfaces to address these frustrations, and then implementation of these solutions in industry. Computers play an important role in affecting the performance of individuals within organizations, therefore, this research should be of great interest to businesses and other organizations, because improved interfaces can improve the bottom-line and corporate profit.
Background Research
Frustration
Frustration is often defined in different ways, making the subject itself somewhat ambiguous creating an ambiguity that surrounds the term. Frustration was first introduced by Sigmund Freud as a concept both external and internal in nature and related to the concept of goal attainment. Frustration occurs when there is an inhibiting condition which interferes with or stops the realization of a goal. All action has a purpose or goal whether explicit or implicit, and any interruption to the completion of an action or task can cause frustration. For Freud, frustration included both external barriers to goal attainment and internal obstacles blocking satisfaction ADDIN EN.CITE Freud1921447Freud, Sigmund1921Types of Onset and NeurosisJames StracheyThe Standard Edition of the Complete Psychological Works of Sigmund FreudLondonHogarth Press12227-230(Freud, 1921).
This concept of frustration as a duality is continued in the analysis of frustration as both cause and effect ADDIN EN.CITE Britt194050Steuart Henderson BrittSidney Q. Janus1940Criteria of FrustrationThe Psychological Review476451-469(Britt & Janus, 1940). As a cause, frustration is an external event, acting as a stimulus to an individual and eliciting an emotional reaction. In this case, the emotional response is the effect, and the individual is aroused by this external cause and a response is often directed towards the environment.
Dollard et al. (1939) define frustration as an interference with the occurrence of an instigated goal-response at its proper time in the behavior sequence ADDIN EN.CITE Dollard1939451J. DollardL. W. DoobN. E. MillerO. H. MowrerR. R. Sears1939Frustration and AggressionNew HavenYale University Press(Dollard, Doob, Miller, Mowrer, & Sears, 1939). Because an instigated goal response entails only that the goal be anticipated, frustration is due to the expectation and anticipation of a goal, not the actual attainment of the goal ADDIN EN.CITE Berkowitz197810Leonard Berkowitz1978Whatever happened to the frustration aggression hypothesis?American Behavioral Scientist215691-708(Berkowitz, 1978). If the goal is unfulfilled, frustration is experienced because satisfaction was not achieved and hopes were suddenly thwarted. The thwarting or hindrance -- terms often used synonymously with frustration -- is not limited to the actual activity in progress, but relates to what the individual is expecting ADDIN EN.CITE Mowrer1938470Mowrer, O. H.1938Preparatory Set (Expectancy) -- a Determinant in Motivation and LearningPsychological Review4562-91(Mowrer, 1938a).
Frustrations, in all cases, are aversive events ADDIN EN.CITE Ferster1957110C. B. Ferster1957The Function of Aggression and the Regulation of Aggressive DriveThe Psychological Review71257-272(Ferster, 1957) having as their main defining feature the element of a barrier or obstruction. This barrier can take the form of an actual barrier, or an imaginary one such as the response to anticipated punishment or injury ADDIN EN.CITE Mowrer1938120O. H. Mowrer1938Some Research Implications of the Frustration Concept as Related to Social and Educational ProblemsCharacter and Personality7129-135(Mowrer, 1938b). A frustrating situation, then, is defined as any in which an obstacle physical, social, conceptual or environmental prevents the satisfaction of a desire ADDIN EN.CITE Barker1938130R. Barker1938The Effect of Frustration upon the Cognitive AbilityCharacter and Personality7145-150(Barker, 1938). These blocks to goal attainment may be both internal and external ADDIN EN.CITE Shorkey198120Clayton T. ShorkeySalena Bell Crocker1981Frustration theory: a source of unifying concepts for generalist practiceSocial Work265374-379(Shorkey & Crocker, 1981), similar to the duality proposed by Freud. Internal blocks consist of deficiencies within the individual such as a lack of knowledge, skill, or physical ability. External blocks could include the physical environment, social or legal barriers such as laws or mores, or the behavior of other people.
Factors Affecting Level of Frustration
The level of frustration experienced by an individual clearly can differ depending on the circumstances surrounding the frustrating experience and on the individuals themselves. One major factor in goal formation and achievement is goal commitment, which refers to the determination to try for and persist in the achievement of a goal ADDIN EN.CITE Campion1982480M. CampionR. Lord1982A Control Systems Conceptualization of the Goal-Setting and Changing ProcessOrganizational Behavior and Human Performance30265-287(Campion & Lord, 1982). Research on goal theory indicates that goal commitment has a strong relationship to performance and is related to both two factors: 1) the importance of the task or outcome and 2) the belief that the goal can be accomplished ADDIN EN.CITE Locke20024320Edwin A. LockeGary P. Latham2002Building a Practically Useful Theory of Goal Setting and Task Motivation: A 35-Year Odysseyreceived from author(Locke & Latham, 2002).
Individuals will have a high commitment to a goal when the goal is important to them and they believe that the goal can be attained ADDIN EN.CITE Locke1996490Locke, Edwin A.1996Motivation Through Conscious Goal SettingApplied Preventative Psychology5117-124(Locke, 1996). The importance of the goal, in addition to the strength of the desire to obtain the goal ADDIN EN.CITE Dollard1939451J. DollardL. W. DoobN. E. MillerO. H. MowrerR. R. Sears1939Frustration and AggressionNew HavenYale University Press(Dollard et al., 1939), will affect the level of goal-commitment as well as the strength of the subsequent reaction to the interruption. Self-efficacy, the belief in ones personal capabilities, can also affect goal commitment ADDIN EN.CITE Locke1990501Locke, Edwin A.Latham, Gary P.1990A Theory of Goal Setting and Task PerformanceEnglewood Cliffs, NJPrentice Hall(Locke & Latham, 1990) in that the belief about how well a task can be performed when it involves setbacks, obstacles, or failures may affect how committed individuals are to that goal ADDIN EN.CITE Bandura1986511Bandura, Albert1986Social Foundation of Thought and Action: A Social-Cognitive TheoryEnglewood Cliffs, NJPrentice Hall(Bandura, 1986).
Judgments of efficacy are related to the amount of effort expended, how long they persist at the task, and resiliency in the case of failure or setback ADDIN EN.CITE Bandura1986511Bandura, Albert1986Social Foundation of Thought and Action: A Social-Cognitive TheoryEnglewood Cliffs, NJPrentice HallBandura1997531Bandura, Albert1997Self-Efficacy: The Exercise of ControlNew YorkFreeman(Bandura, 1986, 1997b). Self-efficacy influencesaffects emotional states as well; how much stress or depression people experience in difficult situations is dependent on how well they think they can cope with the situation ADDIN EN.CITE Bandura1997520Bandura, Albert1997Self-EfficacyHarvard Mental Health Letter1394-6(Bandura, 1997a). The level of frustration that people experience, therefore, would be affected influenced by how important the goal was to them, as well as how confident they are in their abilities. Because goal-directed behavior involves valued, purposeful action, failure to attain goals may therefore result in highly charged emotional outcomes, ADDIN EN.CITE Lincecum2000542LeAnn Lincecum2000The Effects of Software Disruption on Goal Commitment, Task Self-Efficacy, Computer Self-Efficacy, and Test Performance in a Computer-Based Instructional TaskInstructional TechnologyTexas Tech University160Doctoral Dissertation(Lincecum, 2000) including frustration.
Cultural factors can also play a role in the level of frustration experienced by individuals when coming across obstacles to their path of action. Social Learning Theory ADDIN EN.CITE Bandura1973101Albert Bandura1973Aggression: A Social Learning AnalysisEnglewood Cliffs, NJPrentice-Hall(Bandura, 1973) states that rather than frustration generating an aggressive drive, aversive treatment produces a general state of emotional arousal that can facilitate a variety of behaviors, depending on the types of responses the person has learned for coping with stress and their relative effectiveness (p. 53). Ways of coping with frustration are therefore learned from the society and are governed and constrained by the laws of a society. This can contribute to the level of frustration tolerance that individuals have, which is also affected by their prior experience and task specific self-efficacy.
According to Freud, it is not simply the nature of the frustrating incident that determines how people will react to it. Rather, there is an interplay between the situation and the psychological characteristics of individuals. The level of maturity of the individual also plays a part in the reactions to frustration ADDIN EN.CITE Barker1965247R. BarkerT. DemboK. Lewin1965Frustration and Regression: An Experiment with Young ChildrenReed LawsonFrustration: The Development of a Scientific ConceptNew YorkMacMillan Publishing Co.(Barker, Dembo, & Lewin, 1965). With maturity, there is an increase in the variety of responses to a situation employed by individuals, in the control of the environment, and in their ability to employ problem-solving behavior and plan steps to obtain the goal. It would appear that learning, which is culturally determined, is a major factor in developing socially acceptable responses to frustration.
Two additionalfinal factors that may influenceaffect the force of the frustration are the severity of the interruption and the degree of interference with the goal attainment (Dollard et. al. 1939). All obstructions are not equally frustrating, and the severity and unexpectedness of the block will also factor into the strength of the response. In addition, if individuals perceive that the thwarting was justified by socially acceptable rules, as opposed to being arbitrary, the frustration response may be minimized ADDIN EN.CITE Baron197781R.A. Baron1977Human AggressionNew York, NYPlenum(Baron, 1977). This may be due to the lowering of expectations because of extra information available to the individual. As stated above, it is the anticipation of success that affects frustration, and not the actual achievement of the goal. Therefore, if individuals expect to be thwarted or have a low expectation of success, frustration may be minimized.
Responses to Frustration
The responses to frustration by individuals can be either adaptive or maladaptive ADDIN EN.CITE Shorkey198120Clayton T. ShorkeySalena Bell Crocker1981Frustration theory: a source of unifying concepts for generalist practiceSocial Work265374-379(Shorkey & Crocker, 1981). Adaptive responses are constructive and are implemented to solve the problem that is blocking goal attainment. They may include preemptive efforts to avoid the problemblock, or once the block problemi is encountered, problem solving strategies to overcome or circumvent the problem. Freud lists two types of adaptive responses: 1) transforming stress into active energy and reapplying this energy towards the original goal, and 2) identifying and pursuing alternative goals. Maladaptive responses, on the other hand, are characterized by a lack of constructive problem solving and often make the frustrating experience worse by creating additional problems. These maladaptive responses may be further categorized into objective (aggression, regression, withdrawal, fixation, resignation) and subjective (extrapunitive, intropunitive, impunitive) responses (Britt and Janus 1940).
Organizational Frustration
Organizational frustration has been defined by Paul Spector in a very similar fashion, and refers to an interference with goal attainment or maintenance that is caused by some stimulus condition within the organization ADDIN EN.CITE Spector1978940Spector, Paul E.1978Organizational Frustration: A Model and Review of the LiteraturePersonnel Psychology31815-828(Spector, 1978). It has been further narrowed to be defined as the interference with an individuals ability to carry out their day to day duties effectively ADDIN EN.CITE Keenan1984930A. KeenanT. J. Newton1984Frustration in Organizations: Relationships to Role Stress, Climate, and Psychological StrainJournal of Occupational Psychology5757-65(Keenan & Newton, 1984). The sources of organizational frustration put forth by Spector include the physical environment (both natural and man-made), the organizational structure and climate, the rules and procedures of the organization, and individuals both in and out of the organization. In addition, the concept of situational constraints ADDIN EN.CITE Peters1980910Peters, Lawrence H.O'Connor, Edward J.1980Situational Constraints and Work Outcomes: The Influences of a Frequently Overlooked ConstructAcadamy of Management Review53391-397(Peters & O'Connor, 1980) has been hypothesized to contribute to organizational frustration ADDIN EN.CITE Storms1987820Philip L. StormsPaul E. Spector1987Relationships of Organizational Frustration with Reported Behavioural Reactions: The Moderating Effect of Locus of ControlJournal of Occupational Psychology60227-234(Storms & Spector, 1987). Spector (1978) suggested four reactions to organizational frustration: 1) an emotional response of anger and increased physiological arousal, 2) trying alternative courses of action, 3) aggression, and 4) withdrawal. Of the behavioral reactions, only the secondfirst one that of trying alternative courses of action to obtain the goal is an adaptive response, while the other two three are maladaptive. It is likely that the emotional reaction accompanies one of the three behavioral reactions, although the emotional reaction may be maladaptive by itself and become a further impediment to goal attainment. Clearly, should an individual become frustrated, it is in the best interests of the organization to have the individual respond in an adaptive way and attempt to find another solution to the problem in a clear decisive manner. Spector also put forth the idea that some mild forms of frustration may be seen as challenges rather than problems for some individuals, thus causing a motivational effect rather than a hindering effect and increasing the likelihood of an adaptive response rather than a maladaptive one.
Behavior exemplifying two of the threetwo maladaptive responses, in an organization, are described by Spector in his model. Examples of withdrawal behavior in an organization could include the abandonment of a goal, absenteeism, or turnover. Examples of organizational aggression include interpersonal aggression, sabotage, and withholding of output. Both of these maladaptive responses are thought to lead into a decrease in job performance. However, evidence for the frustration-performance link is mixed, as some cases of mild frustration are found to increase task-performance presumably due to increased arousal ADDIN EN.CITE Spector1975920Spector, Paul E.1975Relationships of Organizational Frustration with Reported Behavioral Reactions of EmployeesJournal of Applied Psychology605635-637(Spector, 1975), whereas other studies find that frustration actually inhibits both task performance and learning of a new task.
Other relationships with organizational frustration have also been tested. In a sample of employed individuals, significant relationships were found between both self-reported sabotage and interpersonal aggression with level of frustration as measured by the Organization Frustration Scale ADDIN EN.CITE Spector1975920Spector, Paul E.1975Relationships of Organizational Frustration with Reported Behavioral Reactions of EmployeesJournal of Applied Psychology605635-637(Spector, 1975). Frustration was also found to be strongly correlated to a self-reported desire to leave the place of employment. In another study of 401 employed engineers, Keenan and Newton found that organizational climate, role stress, and social support all correlated positively with environmental frustration ADDIN EN.CITE Keenan1984930A. KeenanT. J. Newton1984Frustration in Organizations: Relationships to Role Stress, Climate, and Psychological StrainJournal of Occupational Psychology5757-65(Keenan & Newton, 1984). Additionally, they found that frustration was significantly related to angry emotional reactions, latent hostility and job dissatisfaction.
Additional research has shown that organizational frustration is positively correlated with several negative behavioral reactions - aggression, sabotage, hostility and complaining, withdrawal, and intent to quit ADDIN EN.CITE Storms1987820Philip L. StormsPaul E. Spector1987Relationships of Organizational Frustration with Reported Behavioural Reactions: The Moderating Effect of Locus of ControlJournal of Occupational Psychology60227-234(Storms & Spector, 1987). In an effort to examine the antecedents of the response choice (adaptive or maladaptive) Storms & Spector also tested for the moderating effect of locus of control, hypothesizing that individuals with an external locus of control would exhibit more counterproductive behavior during times of frustration than those with internal locus of control. This hypothesis was supported, externals increased their counterproductive behavior as frustration increased, whereas the reactions of internal stayed constant.
Using the same Organizational Frustration scale, Jex and Gudanowski examined the role of self-efficacy beliefs and work stress ADDIN EN.CITE Jex1992830Steve M. JexDavid M. Gudanowski1992Efficacy Beliefs and Work Stress: An Exploratory StudyJournal of Organizational Behavior13509-517(Jex & Gudanowski, 1992). They found that individual efficacy beliefs were significantly negatively correlated with level of organizational frustration, indicating that those with less belief in their abilities at their job suffered more frustration than those with high efficacy beliefs. However, they did not find that efficacy beliefs mediated the relationship between stressors and frustration, indicating that self-efficacy does not affect the level of frustration experienced due to external stressors such as situational constraints.
Situational Constraints
The concept of situational constraints was introduced in 1980 by Peters and OConnor in response to the perceived hole in the human performance literature ADDIN EN.CITE Peters1980910Peters, Lawrence H.O'Connor, Edward J.1980Situational Constraints and Work Outcomes: The Influences of a Frequently Overlooked ConstructAcadamy of Management Review53391-397(Peters & O'Connor, 1980). They argued that it has long been assumed that the characteristics of the work setting play a role in performance, but it had never been empirically tested. As such, they introduced a framework for the study of such situational constraints that might affect task performance, which takes into account the idea that individuals who are otherwise capable and motivated to perform may be inhibited by characteristics of the situation. The situational factors that they believed to be relevant to performance, using a critical-incidents method, were job-related information, tools and equipment, materials and supplies, budgetary support, required services and help from others, task preparation, time availability, and work environment.
As such, Peters and OConnor hypothesized a direct link between situational constraints and performance, as well as a direct link between situational constraints and affective reactions such as job satisfaction or frustration. In addition, they thought that the severity of the constraints would affect performance differentially, in accordance with expectancy theory ADDIN EN.CITE Vroom1964981V. Vroom1964Work and MotivationNew YorkWiley(Vroom, 1964). Persons who work in situations where severe constraints are the norm may develop the belief that additional effort on their part will not increase performance. With regards to goals, this indicates that a long history of experience with situational constraints inhibiting progress towards a goal (in their model, performance) would reduce expectations of goal achievement and inhibit productive goal-oriented action. For example, research has found that abilities are positively related to affective responses when situational constraints were low, but negatively related when they were high, indicating that the level of situational constraint on the individual affects their levels of job satisfaction and frustration ADDIN EN.CITE Peters1980990Peters, Lawrence H.O'Connor, Edward J.Rudolf, Cathy J.1980The Behavioral and Affective Consequences of Performance-Relevant Situational VariablesOrganizational Behavior and Human Performance2579-96(Peters, O'Connor, & Rudolf, 1980).
Subsequent research has examined the relationships between situational constraints and such dependent variables as task performance, job satisfaction, frustration, turnover, and goal commitment. In addition, several studies reported the correlations between the negative affect caused by situational constraints and other outcome variables such as performance, job satisfaction, and turnover to demonstrate the direct link between affect and these variables, as well as the indirect link to situational constraints.
In another study of 237 employed individuals in a range of managerial and non-managerial jobs, OConnor et al. examined the relationship between situational constraints and the negative emotional reactions of job satisfaction and frustration ADDIN EN.CITE Peters1982890Lawrence H. PetersMarilyn B. ChassieHarold R. LindholmEdward J. O'ConnorCathy Rudolf Kline1982The Joint Influence of Situational Constraints and Goal Setting on Performance and Affective OutcomesJournal of Management827-20(Peters, Chassie, Lindholm, O'Connor, & Kline, 1982). Their results indicated that the higher the severity of overall situational constraints on the job, the greater the reported frustration and dissatisfaction. In addition, they also found that the average constraint score was negatively correlated with level of effort (p<.001), motivation (p<.01) and organizational commitment (p<.001).
Peters et al (1982) performed another experiment to examine the link between situational constraints, performance, and goal setting, hypothesizing that the link between goal difficulty and goal performance would be mediated by the presence of situational constraints. Their analysis found that there was a direct link between situational constraints and performance, but that goal difficulty was unrelated to performance in the high constraint condition, but positively related to performance in the low constraint setting. This indicates that the effect of goal difficulty and goal setting is dependent on the absence of constraints, showing that situational constraints also have an indirect effect on performance through this relationship. In other words, highly motivated individuals (those who have set difficult goals) are unable to perform at the level anticipated in the presence of situational constraints.
Another study examined the link between goal commitment and performance, and the impact of situational constraints on this relationship ADDIN EN.CITE Klein1998810Howard J. KleinJay S. Kim1998A Field Study of the Influence of Situational Constraints, Leader-Member Exchange, and Goal Commitment on PerformanceAcadamy of Management Journal41188-95(Klein & Kim, 1998). They based this study on the idea that as employees become frustrated with constraints, they cannot perform as well as they feel they ought, and subsequently lose motivation because they no longer expect to perform well. This also fits in with the idea that reduced expectations would lower goal commitment. Their study of 105 salespersons found that situational constraints was negatively related to goal commitment, indicating that the presence of situational constraints negatively impacts the commitment to goals.
Phillips and Freedman performed another study of possible indirect links between situational constraints and motivation and satisfaction, and whether or not degree of perceived personal control over the job had any relationship ADDIN EN.CITE Phillips1984880James S. PhillipsSara M. Freedman1984Situational Performance Constraints and Task Characteristics: Their Relationship to Motivation and SatisfactionJournal of Management103321-331(Phillips & Freedman, 1984). They found a direct link between situational constraints and motivation and satisfaction. They also found that in individuals feeling a high level of personal control, the perceived presence of high constraints actually increased the motivation and satisfaction of individuals in jobs that were characterized by a low motivating potential. This suggests that individuals who perceive that they are in control of their job (internal locus of control) will perhaps find that mild constraints increase the challenge of an otherwise boring job. This could be because these individuals continue to feel in control of the outcomes of the situation, and that a constraint does not necessarily inhibit the achievement of the goal and is seen as challenging rather than problematic.
Locus of control was also examined as a factor in a field study by Storms and Spector (1987), where they found that situational constraints were positively related to perceived frustration, and that frustration was positively related to negative behavioral reactions. As discussed above, locus of control was found to moderate the relationship between frustration and negative behavior, in that individuals with an internal locus of control did not increase their negative behaviors as a result of frustration, whereas externally located individuals did.
Frustration and Situational Constraints
Clearly, situational constraints can have many consequences for an employee, ranging from frustration and dissatisfaction to negative behaviors. Situational constraints not only have a direct effect on these negative consequences, but they also have an indirect effect through the affective reaction of frustration. Constraints in the environment cause frustration in individuals, and the subsequent frustration also has a direct effect on satisfaction, motivation, performance, and the increase in negative behaviors. A meta-analysis undertaken in 1993 on the effect of constraints and work related outcomes reports that situational constraints have a direct effect on turnover (.21), performance (-.14), frustration (.39), satisfaction (-.32), and commitment to the job (-.22) ADDIN EN.CITE Villanova1993800Peter VillanovaMarie A. Roman1993A Meta-Analytic Review of Situational Constraints and Work-Related Outcomes: Alternative Approaches to ConceptualizationHuman Resource Management Review32147-175(Villanova & Roman, 1993). In addition, frustration has an additional effect on performance (-.08) and satisfaction (-.31). It would appear that both the situational constraints themselves, as well as the affective reaction towards it (which can be moderated by such factors as locus of control and efficacy beliefs) can influence organizational outcomes.
Computer Anxiety
The reactions of people to computers have been studied extensively, particularly attitudes towards the computer ADDIN EN.CITE Loyd1984300B. H. LoydC. Gressard1984The Effects of Sex, Age, and Computer Experience on Computer AttitudesAEDS Journal18267-77Murphy1989620C. MurphyD. CooverS. Owen1989Development and Validation of the Computer Self-Efficacy ScaleEducational and Psychological Measurement49893-899Nash1997630J.B. NashP.A. Moroz1997An Examination of the Factor Structures of the Computer Attitude ScaleJournal of Educational Computing Research174341-356(Loyd & Gressard, 1984; Murphy, Coover, & Owen, 1989; Nash & Moroz, 1997) computer anxiety ADDIN EN.CITE Raub1981312A. Raub1981Correlates of Computer Anxiety in College StudentsUniversity of PennsylvaniaDoctoral DissertationGlass1988280C.R. GlassL. A. Knight1988Cognitive Factors in Computer AnxietyCognitive Therapy and Research124351-366Cohen1989590B. A. CohenG. W. Waugh1989Assessing Computer AnxietyPsychological Reports651735-738Cambre1985270M. A. CambreD. L. Cook1985Computer Anxiety: Definition, Measurement, and CorrelatesJournal of Educational Computing Research1137-54Torkzadeh1992360G. TorkzadehI. E. Angulo1992The Concepts and Correlates of Computer AnxietyBehavior and Information Technology11199-108Maurer1994370Maurer, M.M.1994Computer Anxiety Correlates and What They Tell UsComputers in Human Behavior103369-376(Cambre & Cook, 1985; Cohen & Waugh, 1989; Glass & Knight, 1988; Maurer, 1994; Raub, 1981; Torkzadeh & Angulo, 1992), and computer self-efficacy ADDIN EN.CITE Meier1985600S. Meier1985Computer AversionComputers in Human Behavior11171-179McInerney1994410V. McInerneyD. M. McInerneyK. E. Sinclair1994Student Teachers, Computer Anxiety, and Computer ExperienceJournal of Educational Computing Research111177-189Compeau1995610D. R. CompeauC. A. Higgins1995Computer Self-Efficacy: Development of a Measure and Initial TestMIS Quarterly192189-211Brosnan199840M. J. Brosnan1998The Impact of computer anxiety and self-efficacy upon performanceJournal of Computer Assisted Learning14223-234(Brosnan, 1998; Compeau & Higgins, 1995; McInerney, McInerney, & Sinclair, 1994; Meier, 1985). Each of these variables, combined with the factors listed above, can affect how frustrated individuals will become when they encounter a problem while using a computer.
The number of times a problem has occurred before can affect their perception of the locus of control, and therefore influence their reaction as well. This may be related to anxiety, as people with low computer self-efficacy may be more anxious ADDIN EN.CITE Meier1985600S. Meier1985Computer AversionComputers in Human Behavior11171-179Brosnan199840M. J. Brosnan1998The Impact of computer anxiety and self-efficacy upon performanceJournal of Computer Assisted Learning14223-234(Brosnan, 1998; Meier, 1985) and more likely to view the computer suspiciously and react with great frustration when something occurs, especially when they have run into it before. Different levels of anxiety will affect performance when something unforeseen or unknown occurs, causing anxious people to become more anxious ADDIN EN.CITE Brosnan199840M. J. Brosnan1998The Impact of computer anxiety and self-efficacy upon performanceJournal of Computer Assisted Learning14223-234(Brosnan, 1998). On the other hand, the level of experience may temper this if the prior experience increases computer self-efficacy ADDIN EN.CITE Gilroy1986400F. GilroyH. Desai1986Computer Anxiety: Sex, Race, and AgeInternational Journal of Man-Machine Studies251711-719(Gilroy & Desai, 1986) by lowering anxiety and reducing frustration when a problem occurs. The perceived ability to fix problems on the computer, as well as the desire to do so may also affect levels of frustration. If instead, these problems are seen as challenges rather than problems,, they may not be as frustrating, which is most likely directly related to level of prior experience as well as computer self-efficacy.
Computer Frustration
Frustration with technology is a major reason why people cannot use computers to reach their goal, hesitate to use computers, or avoid computers altogether. A recent study from the Pew Internet and American Life study found that a large percentage of people never go online, because they find the technology to be too frustrating and overwhelming (Pew, 2003). Currently, 42% of Americans do not use the Internet, in large part because they find it to be frustrating and confusing. This is not surprising; previous research on user frustration found that users wastedr nearly one-third to one-half of the time spent on the computer, due to frustrating experiences Research on computer frustration ADDIN EN.CITE Bessiere2002670Bessiere, K.Lazar, J.Ceaparu, I.Robinson, J.Shneiderman, B.2002Social Psychological Factors of End User Frustration and Their ImplicationsforthcomingBessiere20021002Bessiere, K.2002COMPUTER FRUSTRATION: A TEST OF A SOCIAL-PSYCHOLOGICAL MODELSociologyCollege Park, MDUniversity of Maryland, College Park51Masters Thesis(Bessiere, 2002; Bessiere, Lazar, Ceaparu, Robinson, & Shneiderman, 20032).
Unfortunately, computer applications are often designed with interfaces that are hard to use, and features that are hard to find. Even government web sites, which are supposed to provide easy access to government information for all citizens, are frequently hard to use and produce high levels of user frustration ADDIN EN.CITE Ceaparu20034500Ceaparu, I.2003Governmental Statistical Data on the Web: A Case Study of FedStatsIT and Society131-17, available at: http://www.itandsociety.orgHargittai20034510Hargittai, E.2003Serving Citizens’ Needs: Minimizing Online Hurdles to Accessing Government InformationIT and Society1327-41(Ceaparu, 2003; Hargittai, 2003). Frustration with technology can lead to wasted time, changed mood, and sufferedaffected interaction with colleagues. When users in a workplace are frustrated with their computers, it can lead to lower levels of job satisfaction ADDIN EN.CITE Murrell19934570Murrell, A.Sprinkle, J.1993The impact of negative attitudes towards computers on employees' satisfaction and commitment within a small companyComputers in Human Behavior957-63.(Murrell & Sprinkle, 1993). In some cases, user frustration with technology can even lead to increased blood volume pressure and muscle tension ADDIN EN.CITE Riseberg19984563Riseberg, J.Klein, J.Fernandez, R.Picard, R.1998Frustrating the user on purpose: Using biosignals in a pilot study to detect the user's emotional stateCHI 1998: ACM Conference on Human Factors in Computing Systems227-228(Riseberg, Klein, Fernandez, & Picard, 1998)
Research on computer frustration has shown that that computer self-efficacy and attitudes play a significant role in reducing the frustration levels in computing. Level of comfort with the computer and the determination to fix a problem, which are associated with a high level of computer self efficacy, both appear as important factors in both the immediate experience of frustration as well as the overall frustration level after a session of computer use. In this study, cIn the previous study on computer frustration, computer attitude variables mediated the experience of frustration but experience did not. Simply using a computer, therefore, does not lessen user frustration;, rather it is ones attitude towards it and comfort with it.
There is a measurable benefit to improved usability of user interfaces for lower user frustration ADDIN EN.CITE Bias19944299Bias, R.Mayhew, D.1994Cost-Justfying UsabilitySan FranciscoMorgan Kaufmann Publishers(Bias & Mayhew, 1994). Many well-known companies, such as IBM, Staples, the National Football League, and Macys, do focus on improving their interface design, which leads to measurable improvement ofin the bottom line ADDIN EN.CITE Clarke20013963Clarke, J.2001Key factors in developing a positive user experience for children on the web: A case studyHuman Factors and the Web 2001Available at: http://www.optavia.com/hfweb/index.htm(Clarke, 2001; ADDIN EN.CITE Tedeschi19992875Tedeschi, B.1999Good Web Site Design Can Lead to Healthy SalesThe New York TimesAugust 30, 1999Tedeschi, 1999). For instance, when Macys made their web site search engine easier to use, the conversion rate (the rate at which site visitors are converted into buyers) went up 150% ADDIN EN.CITE Kemp20014445Kemp, T.2001Macy's Doubles Conversion RateInternetWeek.comAvailable at: http://www.internetwk.com/story/INW20011128S0004November 28, 2001(Kemp, 2001). Staples.com used feedback from users to improve their online registration pages, to make them easier to use. After improving the usability of the registration pages, the registration drop-off rate (the number of people who begin registering but fail to complete the registration) decreased by 53% ADDIN EN.CITE Roberts-Witt20014458Roberts-Witt, S.2001A Singular FocusPC MagazineSeptember 25, 2001(Roberts-Witt, 2001). After losing market share, AOL yielded to customer complaints and removed a majority of the pop-up advertisements from their service ADDIN EN.CITE Hu20024525Hu, J.2002AOL's Pop-Up SacrificeCNET NewsAvailable at: http://news.com.com/2100-1023-962345.html(Hu, 2002). Companies that have re-designed interfaces for log-on screens and for user forms have seen improvements in employee productivity that can be measured, in tens or hundreds of thousands of dollars (Nielsen, 1994).
Having a proactive attitude towards the computer, particularly seeing the problems with the computer as challenges rather than problems, was shown to decrease frustration levels.
The consequences of negative attitudes towards the computer have not been studied extensively within organizational settings. However, Murrell and Sprinkle examined the relationship between feelings of frustration and confusion about the use of computers and found that these were associated with lower job satisfaction ADDIN EN.CITE Murrell1993780Audrey J. MurrellJodi Sprinkle1993The Impact of Negative Attitudes Toward Computers on Employees' Satisfaction and Commitment Within a Small CompanyComputers in Human Behavior957-63(Murrell & Sprinkle, 1993). This indicates that the consequences of negative attitudes towards computers may extend to the organizational level, as the literature on situational constraints would also suggest.
Organizational Frustration, Situational Constraints, and the Computer
The growing ubiquity of computers and information technologies as part of the organizational environment suggest the increased role that equipment and tools, and situational constraints more generally, will play in both organizational level and individual level frustration. The dearth of literature examining the causes and associated factors surrounding computer frustration make it difficult to hypothesize about the links between computer frustration and organizational frustration and other negative consequences. However, the literature on situational constraints gives some indications for areas of future research. For instance, we might speculate that the complexity of the computer as a tool leads to greater opportunities for situational constraints to produce organizational frustration.
Proposition 1: The computer, as a piece of equipment and a tool necessary to accomplish many jobs, will contribute to levels of organizational frustration as a situational constraint.
In addition, the increasing reliance on technical support adds another dimension to the idea of the computer as a situational constraint insufficient support and help in the face of computer problems will also hinder the individuals progress towards their goals in the workplace.
Proposition 2: Increasing reliance on technical support both within and without the organization will serve as an additional situational constraint.
If the computer becomes a situational constraint, as well as the increased reliance on other individuals to keep this equipment running properly, all the attendant organizational consequences of situational constraints will therefore apply to the computer.
Proposition 3: Frustration with computer problems will lead to decreased job satisfaction, increased organizational frustration, a decrease in performance due to both frustration and loss of time, a loss of motivation, and the exhibition of maladaptive goal-attainment behaviors.
However, as the literature on frustration indicates, particularly the literature on computer frustration, greater computer self-efficacy will lead to a tendency to view computer problems as challenges rather than problems, and will moderate the relationship between computer frustration and the resultant consequences.
Proposition 4: Greater computer self-efficacy will lead to adaptive goal-attainment behavior and moderate the relationship between computer frustration and the resultant consequences.
Literature on the role of training and experience indicates that these lead to greater computer self-efficacy, but do not have a direct link to the consequences of negative attitudes towards the computer. However, since training and experience increase computer self-efficacy, and computer self-efficacy moderates this relationship, training and experience are deemed necessary for the development of proactive attitudes.
Proposition 5: Training and experience will increase computer self-efficacy.
Conclusion
As the reliance upon computers as tools in organizations becomes more prevalent, it becomes more important for individuals to have positive attitudes towards the computer in order for them to have adaptive reactions to the problems that they are sure to encounter. The computer, as a piece of equipment and a tool, is a situational constraint that has consequences for affective reactions to the organization and job, as well as behavioral consequences of decreased performance and possible interpersonal and organizational aggression. As such, it is important to look at the possible factors that could lessen these maladaptive reactions, and find ways to lessen the impact of computer problems. One such method might be to increase training and experience of users in order to heighten their computer self-efficacy and subsequently lower the negative consequences. However, to ignore the problem of the computer would be detrimental to an organization, especially as technology becomes more complex and harder to learn. As such, it is vital that organizations give the support to their computer users that is needed to lower the negative attitudes and frustration, which could have devastating organizational impact.
RESEARCH METHODOLOGY Research Methodology
To learn more about user frustration with technology in the workplace, data was collected through the use of modified time diaries. With a modified time diary, usersUsers recorded data about their frustrations as the frustrations occuroccurred. Surveys would not be an appropriate data collection methodology for this research, since users trying to recall frustrations from their past experiences might over-estimate or under-estimate the level of frustration and the time wasted ADDIN EN.CITE Fowler1993871Fowler, F.1993<styles></styles>Survey Research MethodsNewbury Park, CaliforniaSage Publications2nd(Fowler, 1993). In addition, data logging cannot effectively measure frustration, since data logging would only work for system errors, or other occasions when the systems indicated an error state. There are many events that are frustrating for users (such as spam or pop-up advertisements), and occur when the system is operating in a correct state. This same methodology was used in the previous study of computer frustration in students (Ceaparu, Lazar, Bessiere, Robinson, and Shneiderman, 2003).
Subjects in this experiment study were encouraged to perform their typical work-related tasks, and record, as a part of their time diaries, any frustrating experiences. Tasks are not pre-assigned to subjects, because user frustration is correlated to the importance of the task ADDIN EN.CITE Bessière20034070Bessière, K.Ceaparu, I.Lazar, J.Robinson, J.Shneiderman, B.2003Understanding Computer User Frustration: Measuring and Modeling the Disruption from Poor DesignsPaper under review(Bessire, Ceaparu, Lazar, Robinson, & Shneiderman, 2003). When tasks are important to users, users report higher levels of frustration than when tasks are not important. Pre-assigned tasks would therefore not accurately model the user frustration in an average workday. The following protocol was used:
1. Fill out demographic information (age, gender, computer experience, etc.)
2. Fill out a pre-session survey (noting current mood) (Appendix A)
3. Perform work-related computer tasks of their choosing, for a minimum of one hour total.
4. Fill out frustration experience forms, whenever the subject feels frustrated. These forms describe the cause, nature, and severity of the frustrating experience. (Appendix B)
5. Fill out a post-session survey (measuring frustration after the session ended) (Appendix C)
6. After completing the post-session survey, subjects fill out a reimbursement form and return all of the materials via postalsnail-mail to the researchers.
RESULTS
Data collection took place from mid-2002 until 2003.
A total of 50 subjects took part in the research experimenstudy. Each of these subjects was a workplace user of computers, and each subject was paid $25 for their participation. The workplaces represented in this study include healthcare (15), law (3), education (8), information technology (11), non-profit-other (5), for-profit-other (2), government (3), and 3 subjects did not indicate their workplace. The average age of users was 35.95 years (with a range of 23 to 76 years old). The average number of years of computer experience was 2.38 years (with a range of less than a year, to 25 years of experience). A total of 149 frustrating experiences were reported, with each participant reporting between 1 and 6 experiences. Users recorded their experiences, in time diaries, for a period of 5.1 hours, on average. This paper reports the causes and severity of the frustration, highlighting the responses to frustration, as well as the time lost. A separate paper will address how the frustration impacted on the individuals mood and interaction with others.
Figure 1: Demographic information
Average age of subjectAverage Experience for SubjectHigh / Low AgeHigh / Low Experience36 years2.38 years(23 years / 76 years)(0.5 years / 25 years)
WWeb browsing and ord processingemail and e-mail produced the largest number of frustrating experiences, probably reflecting that these applications were used most often (Table 1). Table 1 reports the applications that were the source of the frustrating experience. Often the frustrating experience affected the entire system, and was during activities that participants had previously completed successfully a number of times without error. There were several frustrating experiences involving moving data from one application to another type of application, such as email content into word processing and even moving content among similar applications, such as Word to WordPerfect. Many frustrating experiences were inhibiting but did not ultimately prevent the task from completion.
Table One: Summary and Demographic information from the study
Average age of users
35.94 years
(high was 76, low was 23)Average computer experience2.38 years
(high was 25, low was < 1)Percent of total time lost28%
(35% including outliers)Time lost outlier High (to error; to recovery)540 minutes; 540 minutesTime lost outlier Low (to error; to recovery)0 minutes; 0 minutesMost common problem sourceWord ProcessingMost common problem categorySystem-Wide CrashesMost common solution
User knew how to fix from previous experienceLeast common solutionConsulting Manual or BookMost common level of frustration9 Very Frustrating (Highest level)
Participants came from a wide variety of backgrounds and work environments. There were 50 participants with a total of 149 frustrating experiences. With each participant reporting between 1 and 6 experiences. The most common response to errors was to reboot or restart the computer, implying that the problem would fix itself eventually.
Table On1eTwo: Applications that were the source of frustrating experiencesProblem Sources of reported incidents
Problem sourceFrequency of problem sources
Reports
Web browsing17Email28Other Internet Use11Video/Audio Software1Word Processing34Chat and Instant Messaging1File Browsers1Programming Tools2Spreadsheet Programs9Graphic Design Programs1Presentation Software1Database Programs6Other37Total149
Table 2 reports the solutions that users took in the attempt to solve their frustrating experiences. Most participants were already familiar with the frustrating experience from previous experiences and know how to solve it (Table 2). Most solutions involved simply redoing the task either orafter a restarbooting and then redoing the task or twice in a row. Other solutions involved work-arounds and as a last resort finding help externally. The type of solution taken was independent of other demographic informationdifferences. Int is interesting to note that only one frustrating experience did the user consult a manual, and only in two experiences did the user consult online help.
1oneProblem sources were based on those listed in a previous study. Often the problem affected the entire system, and was during activities that participants had completed successfully a number of times without error. There were several incidents involving moving data from one application to another type of application, such as email content into word processing and even moving content among similar applications, such as word Word to WordPerfect. Many problems were inhibiting but did not ultimately prevent the task from completion.
Table 23: Solutions taken by participants
Solution takenFrequency of solutions
Reports
I knew how to solve it because it happened before35I figured out a way to fix it myself9I was unable to solve it16I ignored the problem or found an alternative20I tried again5I restarted the program15I consulted online help2I asked someone for help16I rebooted29I consulted a manual or a book1
Total148
Note: some responses were left blank by participant
TIn this study, the time lost due to frustrating experiences was one of our key measuresd. In terms of time lost, one frustrating experience was considered to be an outlier. The one outlier frustrating experience was reported as 540 minutes to fix the problem, and another 540 minutes to recover the problem. The cause of the frustrating experience was a hardware problem, where the user reported that they would assign IRQs to hardware, and every time that the computer re-booted, the operating system would re-assign those IRQs. The user reported disabling the problem devices so that they could complete their current tasks. Due to the large amount of time wasted, we therefore felt that this one frustrating experience should be separated out as an outlier. The user reported two other frustrating experiences, but those were well within the typical range reported. The other frustrating experiences from the same subject are therefore included.. Users, in general, spend more time recovering from an incident than initially working through the incident. Both the initial time spent on responding to the frustrating experience, as well as the time to recover from any work lost due to the problem, contribute to the total time lost. The method for computing percentage of time lost is as follows:
Percent Time Lost = (MS + MR) / MT
Where MS is
percentage of time lost=
(minutes spent to solve the problem, MR is) +( minutes spent to recover from lostany work loss due to the problem, and MT is)
---------------------------------------------------------------------------------------------------------------------------
total minutes spent on the computer
(Ceaparu, Lazar, Bessiere, Robinson, and Shneiderman, 2003).
For each user, the amount of time lost to respond to the initial problem, as well as the time lost to recover from the problem, was added for all frustrating experiences reported by that user and then divided by the overall time spent by that user on the computer. The final figure represents the percent of total time lost by that user of the time that they spent on the computer. . Table 3 gives a sample of these data calculations from one user.
Table 3. Sample of time lost statistics for one user
Minutes Lost Spent tTo InitialSolve ProblemMinutes LostSpent to To Recover Lost Work Total Minutes LostTotal Minutes Spent on the ComputerPercent Time Lost tTo InitialSolve ProblemPercent Time Lost to Recovering Lost WorkPercent Total Time Lost A sample user 30508018316.4%27.3%43.7%
The percentages for time lost dueto solve to initial the problem, time lost to recover lost work, and total time lost, after being normalized for each user, were then averaged over the population of 50 users (. The resulting averages are displayed in tTable 4). Each column in table 4 is calculated from the data itself and not from the previous numbers in the table. Therefore, percentage time lost is not exactly equal to the sum of the figures in the first two columns. FiguresPercentages are given for all users, and for all users minus the one outlier frustrating experience frustrating experience (see paragraph below).
One user reported an extremely long duration frustrating experience, which we considered to be an outlier. The one outlier frustrating experience was reported as 540 minutes to solve the problem, and another 540 minutes to recover lost work. The cause of the frustrating experience was a hardware problem, where the user reported assigning IRQs to hardware, and every time that the computer re-booted, the operating system would re-assign those IRQs. The user reported disabling the problem devices to complete the current task. Due to the large amount of time wasted, we therefore felt that this one frustrating experience should be separated out as an outlier. The user reported two other frustrating experiences, but those were well within the typical range reported. The other frustrating experiences from the same subject are therefore included (Table 4).
Most Participants were already familiar with problems from previous experience. Most solutions involved simply redoing the task either after a restart or twice in a row. Other solutions involved work arounds and as a last resort finding help externally. The type of solution taken was independent of other demographic information.
Table a4: Breakdown of time spent and lost by participants
Total Minutes Lost To Initial ErrorTotal Minutes Lost To Recover Total Minutes LostTotal Time Spent on the ComputerPercent Wasted Time for ErrorsPercent Wasted time RecoveringPercent total wasted time1934238643201534012.6%15.5%28.1%
Table 3b: Breakdown of time spent and lost by participants (with Outliers)
Total Minutes Lost To Initial ErrorTotal Minutes Lost To Recover Total Minutes LostTotal Time Spent on the ComputerPercent Wasted Time for ErrorsPercent Wasted time RecoveringPercent total wasted time2474292654001534016.1%19.1%35.2%
Table 43c.: NEW Breakdown of averageAverages for percent time lost due to initialsolve problem, average time to recover lost work, and average total time lost for all users time spent and lost by participants
Percent
Time Lost
to Solve Problem Percent Wasted Time to Respond to Initial Problemfor ErrorsPercent Time to Recover Lost WorkPercent TWasted time Lost RecoveringPercent Ttotal Time Lostwasted timeTotal wWithout outlier20.325%20.218%42.72%Total wWith outlier21.5%22.2%43.72 %
Table 3d: NEW Breakdown of time spent and lost by participants (with Outliers)
Total Minutes Lost To Initial ErrorTotal Minutes Lost To Recover Total Minutes LostTotal Time Spent on the ComputerPercent Wasted Time for ErrorsPercent Wasted time RecoveringPercent total wasted time2474292654001534021.54%22.18%43.7 %
Table 3e: Time Statistics
Avg. min lost per user To Initial errorMost min lost by user to Initial errorLeast min lost by user to Initial error19.5 minutes540 minutes0 minutes
Table 5 reports the total minutes lost by application source. Problems with word processing cost participants the most amount of time in total (1225 minutes lost), followed by problems with email (666 minutes lost) (Table 5). This did correlate towith the applications that caused the highest number of frustrating experiences. The applications that were uncommon sources of frustrating experiences (such as programming tools, database software, and presentation software) often required more time per incident, as the problems were rare and complex to solve. Tables 3a-d represent important statistics with and without the outliers of the study. A and B were calculated by first adding up the times of all participants and then calculating percentages. C and D were calculated by finding the percentages for each participant and then adding up and averaging those results. Table 3e lists some other miscellaneous statistics.
Time lost was divided by the time to fix the problem and the time required to get back up to speed. Participants spent more time recovering from an incident than initially working through the incident. This includes recovering lost work and the time to reaccomplish the initial task. Time spent was based on start and stop times recorded by the user. The percentage is the amount of time that could have been utilized, had the incident not occurred.[ADAM: WE NEED TO SHOW TWO DIFFERENT ANALYSES HERE: WITH THE OUTLIER, AND WITHOUT THE OUTLIER]
Table 55: Breakdown of total minutes lost and average time minutes lost per frustrating experienceincident by application problem source
Problem sourceReports Total minutes. lost# of fFrustrating experiencesEAverageg minutes lost per FEfrustrating experienceEmail6662823.8Web Browsing2441714.4Other Internet Use105119.5Word Processing12253436.0File Browsers414.0Video/Audio Software20120.0Programming Tools140270.0Graphic Design Programs515.0Database Programs335655.9Chat and Instant Messaging212.0Presentation Software1051105.0Spreadsheet Programs604967.1Other8653723.4Total432014928.99Note: FE=number of frustrating experiences
Table 6 lists the specific causes of the frustrating experiences. Problems with word processing cost participants the most amount of time, followed by problems with email. This did correlate to the two most common problem sources. The uncommon sources often required more time per incident as the problems were rare and complex to solve. In some cases minutes lost was also upinfluenced by to external factors such as help desks and tech support. More often than not time would not have been saved with a backup copy of the data taken just before the error and restoring afterwards. Categories of frustrating experiences were based loosely on a previous study (Ceaparu, Lazar, Bessiere, Robinson, and Shneiderman, 2003) with minor changes to accommodate terms used by our participants (Table 6). Major categories were grouped by the behavior described in each frustrating experience. System crashes were the most commonly-reported frustrating experience, accounting for 21 of the 149, and were caused by specific programs as well as the operating system itself. While many of these problems are hardware-related or technical-related (such as printing problems and system freezescrashes), there were a number of frustrating experiences that were caused by interface-related issues (such as uncontrollable pop-up windows, hard to find features, and unpredictable behavior of application, and unclear error messages). For instance, there were 19 experiences with missing/hard-to-find/unusable features, 4 experiences with uncontrollable pop-up windows, and 5 experiences with unclear error messages. These interface-related causes of user frustration are easily solvable, when attention is paid to appropriate user interface design. As discussed in previous portions of the paper, when these interface improvements are made, it leads to improved user productivity and organizational profitability.
Table 66: Specific causes of frustrating experiencesBreakdown of 5 main categories and many subcategories of problems
InternetApplicationsOperating SystemHardwareOtherSending / Receiving Email and accessing attachments
{ 6 } Lost / Dropped Connections
{ 7 }Missing / Hard to find Unusable Features
{ 19 }Unknown File Format
{ 1 }System FreezeCrash Caused by Operating System
{ 21 }File Browser Operations
{ 2 }Printing Problems
{ 10 }User Kicked from System
{ 5 }Virus / Malicious Program
{ 1 }Sending / Receiving Email and accessing attachments
{ 6 }Lost / Dropped Connections
{ 7 }Program-Only Crash; System okApplication Crash
{ 11 }File Browser Operations
{ 2 }System-Wide Crashes caused by OS
{ 21 }Hardware Conflicts
{ 3 }Multi user File Access and Permission Issues
{ 5 }Password Not working
{ 2 }Uncontrollable Pop-up window
{ 4 }Browser Failure
{ 2 }Buggy, Incorrect behavior of program
{ 10 }System-Wide crashes caused by Program
{ 4 }Multitasking Failure
{ 1 }Device Failures
{ 3 }Password Not working
{ 2 }Multi user File Access and Permission Issues
{ 5 }Plug-in Failure
{ 1 } Internet Login Failures
{ 3 }Excessive Slow Operation
{ 8 }Virus / Malicious Program
{ 1 }Local Network Connection Access Failure
{ 1 }Browser Failure
{ 2 }Uncontrollable Pop-up window
{ 4 }Unpredictable Response of program
{ 6 }Missing / Hard to find Unusable Features
{ 19 }Local Network Connection Access Failure
{ 1 } Power Failure
{ 1 }File Download Failures
{ 2 }Unclear Error Messages
{ 5 }Unpredictable Response of program
{ 6 }Power Failure
{ 1 }User Kicked from System
{ 5 }Plug-in Failure
{ 1 }Internet Login Failures
{ 3 }Installation Issues
{ 4 } Unclear Error Messages
{ 5 }Application Crash that Froze the entire System
{ 4 }Buggy / Undesirable behavior of program
{ 10 }Installation Issues
{ 4 }Unknown File Format
{ 1 } Note: some responses were left blank by participant
The participants expresses strong eTables 7 and 8 display the data on the emotional reactions to the frustrating experiences (. Table 7) reports on how users felt after the frustrating experiences. For instance, in 60 of the frustrating experiences, users felt angry at the computer, in 34 experiences, users felt helpless/resigned, and in 15 experiences, users felt angry at themselves. It is important to note thatS since users may have more than one emotional reaction, these numbers for table 7 will add up to more than the 149 frustrating experiences reported. Table 8 reports the level of frustration for each of the frustrating experiences. It is important to note thanUsing a 1 to 9 numeric scale, 106/149 of the frustrating experiences were reported to have frustration levels of 7, 8, or 9 , very high frustration levels, indeed(. Figure 1) displays the same frustration levels as table 8. These high levels of frustration can have an impact on the human bodyphysiological variables. For instance, in a previous study of user frustration, it wasresearchers found that when typical users get frustrated with their computer, it affects blood volume pressure ADDIN EN.CITE Riseberg19984563Riseberg, J.Klein, J.Fernandez, R.Picard, R.1998Frustrating the user on purpose: Using biosignals in a pilot study to detect the user's emotional stateCHI 1998: ACM Conference on Human Factors in Computing Systems227-228(Riseberg, Klein, Fernandez, & Picard, 1998).Categories were based loosely on a previous study. Major categories were grouped by the behavior described in each incident. System crashes were the most commonly frustrating, and were caused by specific programs as well as the operating system itself. Most of these problems were non repeatable and subcategories involved more than one participant. For example all of the pop-up window errors did not belong to one person.
Table 76: User feelings per incident
Expressed FeelingNumber of
ReportsAngry at the computer 60Angry at yourself15Helpless / Resigned34Determined to fix it 27Neutral17Other26Note: Some participants had multiple feelings per incident
Table 8767: Number of incidents for each level of frustration
Level of FrustrationNumber of Frustrating Experiences1 Not Very Frustrating0263641356687238339 Very Frustrating50Total:145Note: some responses were left blank by participant
Figure 17: Bar graph of reported frustration levels
EMBED Excel.Chart.8 \s [ADAM: WE SHOULD ALSO HAVE A BAR GRAPH OF THIS CHART]???
Unfortunately, it seems that these frustrating experiences are not rare occurrences that happen on an infrequent basis. From the frustrating experiences reported, users were asked to indicate whether this same event had occurred previously, and if so, how often (. Table 8)9 displays their responses related to the frequency of the problem.. For instance, for 25 of the frustrating experiences, users reported that the same event occurs more than once a day. For 21 of the frustrating experience, users reported that the event occurred several times a week. It is clear that users repeatedly face problems, and these problems interrupt their day.
Most users reported high levels of frustration, and some level of effect on their coworkers. The more frequent problems and the ones that took the most time were associated with higher levels of frustration. Unimportant tasks had lower levels of frustration.
[ADAM: WE NEED THE DATA ON HOW OFTEN THESE INCIDENTS HAPPEN]
Table 898: Frequency of problems
Frequency of problemNumber of reportsMore than once a day25One time a day7Once a week 15Once a month11Several times a week21Several times a month29Several times a year9First time it happened29
Note: Some responses left blank by participant
Discussion
From the results of this study, it is clear that user frustration is a problem in the workplace. Since uUsers lose largemore than 40% quantities of their time, and these frustrationg experiences have an impact on both the individuals, and their time lost has an impact on the organizations. From an individual point of view, users waste a large amount of time, which impacts onslows their completion of work, limiting their time with family, friends, and co-workers. It also can affect their emotional state. TAnd these frustrating experiences also impactharm on the organizations which these individuals work for. Just think about how much more profitable a company would be if more of that time was productive. And this isnt the workers slacking off or wasting their timethis is the users that are trying to complete their tasks, but are facing frustrating experiences, leading them to take more time on their tasks. The time wasted due to poor computer interface design is staggering. The time wasted has a large monetary value to organizations. by undermining productivity, lowering quality, and raising stress levels. The moneyinvestment in spent to improvinge user interfaces iswould yield large payoffs; several studies suggest more than justified by the monetary value of time saved. In some cases, the that the cost of the interface improvement is made up 5, 10, or 20 times over (Bias and Mayhew, 1994).
It is clear from this study and previous work that user frustration is a major problem. In the previous study of 111 subjects, the subjects were university students, and their friends and family members. This current study was limited to workplace users. However, the results of the two studies were surprisingly similar.
Next Section: A Comparison of the Workplace data and the student data
Table 9 reports the The top 3 problem applications causing a frustrating experiencesources from both the previous study and the current study. The applications causing the most frequent frustrating experiences for the student frustration studystudy were web browsing, email and word processing. In the workplace frustration study, the top 3 application problems sources encountered by the users were the same, but in reverse order: word processing, email, web browsing.
The top 3 solutions taken by the participants to solve the problems that occurred in the student study were: for the student study - they knew how to solve it from previous experience, they figured out a way or they were unable to solve it. For the workplace study, the top 3 solutions cited were; for the workplace study - that they knew how to solve it from previous experience, they rebooted, or ignored the problem/found an alternative.
Table 9: Applications causing the largest numbers of frustrating experiences
Student Study-Causes of FrustrationWorkplace Study-Causes of Frustration1. Web Browsing1. Word Processing2. E-mail2. E-mail3. Word Processing3. Web Browsing
In both studies, the levels of frustration wereis at the high end of the scale. For instance, 7,8, and 9 are the highest scores on the frustration scale, and in both studies, large percentages of subjects reported their frustrations being in that range. Levels 7, 8 and 9 of frustration are the top 3 levels for both the student study (7: 68, 8: 77, 9: 91) In the student frustration study, 63.3% (236 out of 373) of the frustrating experiences caused high levels of frustration. In the workplace study, 71.1% (106 out of 149) of frustrating experiences caused high levels of frustration. and the workplace study (7:23, 8:33, 9:50).
The amount of time lost was also similar in the different frustration studies. Table 10 describes the time lost in the various studies, both with and without outliers. The numbers are very similar. In the previous study with students, there were two phases: self-reports and observations. These data points are listed separately. In the previous study with students, the average percentage of time lost ranged from 38.9% (for self-reports without the 5 outliers) to 50.1% (for self-reports with the 5 outliers). In this workplace study, the average percentage of time lost ranged from 42.7% (without outliers) to 43.7% (with outliers). We think that the difference in ranges was logical, due to the numbers of subjects involved in the two studies. In the previous student study, 111 subjects took part, whereas in this workplace study, only 50 users took part. With a larger number of users taking part, it logically follows that there will be more outliers, and therefore, a wider percentage spread.
Table 10109: Time lost in the variousthree studies, with and without outliers.
Average time lost per user
(with outliers)Average time lost per user (without outliers)Student study self-reports50.1%38.9%Student study observations49.9%41.9%Workplace study43.7%42.7%
From the analysis in the student study, we get the following results in terms of time lost: the average time lost per individual for UMD reports was 47.8% (37.9% without the outliers) and for Towson 53.1% (43.5% without the outliers); the average time lost of 50.1% (38.9% without the outliers) from self-reports and 49.9% (41.9% without the outliers) from observations.
From the analysis in the workplace study, the average time lost per individual is 28.1%.
When looking at the specific causes of the frustrating experiences categories and subcategories of problems that occurred, the student study finds that the top 5 were: error messages, timed out/dropped/refused connections, application freezes, missing/hard to find/unusable features, long download time. The workplace study finds that the top 5 were: OS crashes, missing/hard to find/unusable features, application crashes, hardware problems, buggy/undesirable behavior of program. Many of these problems (such as error messages, hard-to-find features, and undesirable behavior) are caused by poorly-designed interfaces, and therefore, can be solved with more usability testing and more user involvement in the interface development. It is interesting to note that some of the causes of frustration for the student participants, such as timed out/dropped connections, and long download times, did not appear as frequent frustrations for workplace participants. It is likely that the network connection at a workplace is of higher quality and speed, therefore less likely to cause frustration due to either response time or dropped connection. However, many of the software applications are the same, regardless of user population or location, and are highly likely to cause frustration.
ConclusionSolving the Causes of User Frustration
It is clearThis study with 50 workplace users adds to the growing evidence that user frustration is a major problem. Further studies with a narrower focus may isolate and measure contributing factors, but there is enough evidence to encourage change in the industry.
This study, in a new population of workplace users, helped to confirm the problem of user frustration. What can be done about this problem? BadImproving user interface design, causing user frustration, is a solvable problem is one clear opportunity because the payoffs will be immediate and benefit many users. To build better interfaces, more user involvement is needed in the interface design process. Designers should follow the interface guidelines that exist. AndUser tra users needs to be provided with more training and documentationwill also help, especially if it addresses problem solving strategies that will help build self-efficacy. Even small changes in the interface can make a big impact on user satisfaction. For instance, in recent studies of the FedStats web site, changing the interface of a governmental web site increased user satisfaction and performance nearly 100% (Ceaparu and Shneiderman, 2004). But these frustrations can actually be engineered-out of the system. And many of these frustrations are easily solvable. For instance, it is noted in the data thatSincereported beingare , interface designers should be directed to review all messages and instructions. The cause of these specific frustrations are generally unclear wordIf link titles are unclear for users, or are not where the users expect, they may not be able to complete their tasks (Daniel and Lazar, 2004). Unclear wording has been found as a major problem in interface design, regardless of the user population or the task. For instance, in a usability study of a university web site, 5 users all failed to find the information that they were looking for (current course schedules) because the information was listed under an unclear heading (Student Life). From a technology coding point of view, changing the words displayed is relatively simple. In addition, the usability methods needed to find out that the wording is unclear, are also relatively simple. Paper-based usability testing methods such as card sorting or paper prototypes can help find flaws in interface wording. Since many users reported being frustrated by unclear error messages or by hard-to-find features, interface designers should be directed to review all messages and instructions. Good guidelines for error message design have existed since 1982, but these guidelines are rarely followed. (Shneiderman, 1982). Error messages should be positive, provide information for users (in their language) on what occurred, and offer suggestions on how to continue. Current error messages rarely assist users (see figure 2 for an example of this). Improved error messages can reduce user frustration while making users more satisfied and productive. (Lazar and Huang, 2003). While all causes of user frustration are not as easily solvable, a large percentage of user frustrations ARE solvable. And there are many resources out there to help improve interface design, such as books, automated software tools, guidelines, and other resources (See HYPERLINK "http://www.hcibib.org" www.hcibib.org or HYPERLINK "http://www.hcirn.com" www.hcirn.com for more information).
Figure 2. An unclear error message
The iImplications for sDifferent Stakeholders might be separated out by:
Designers -
As designers, utilizing the results of studies such as this can yield the most benefits of all. B : can build more productive systems by learning what frustrates users in the workplace. designers can build more productive systems, that is systems with less frustration. Systems can be modified not only to have fewer errors but also to be more helpful. This may include better error messages, better and helpful descriptions of problems which can reduce the time needed to fix an issue, as well as designs based more closely on the way users work particularly with respect to how end users handle errors. This would improve efficiency overall as systems would be better equipped to handle problems faster and allow for the system to get back to operating normally (without problems) and in general make things more usable.
Managers -
Managers can benefits but by learning where frustrations occur within computing systems of their employees. This would let them see the bigger picture of how to improve processes as well as how to deal with frustrations in the workplace by understanding them. One such example might be allowing the manager to make better decisions in deploying systems and choosing the right IT support for employees. In other words less frustrating systems yield ahelp them to construct a more productive workplace, reduce workflow bottlenecks, and can make the environment more manageableproduce more satisfied employees in general. They can recommend training for employees and make more appropriate choices in software acquisitions.Managerial choices involving systems with varying frustration levels can only be made when understanding the causes and implications of IT related workplace frustrations.
Users -
End Users will ultimately benefit from the improvements made to systems based on this and similar studies. This study also gives users a method for detailing and explaining errors, rather than simply working around them or ignoring them. Without such interaction and insight into where users get frustrated, improving things and reducing the frustration in the workplace is more difficult.of computers will appreciate learning that they are not alone in their frustrations. They can take steps to improve their training and increase their knowledge, but they can accelerate improvement by being consumer activists who report problems, complain to designers, and suggest improvements.
Policymakers -
Policy makers, like managers, benefits from the larger picture and by seeing where inefficiencies occur. It is highly possible to improve these frustrations through policy. Policies could be adapted to provide better solutions for handling incidents. Of particular importance would be policies and procedures for users interacting with help desk and IT support (which was a source of frustration in this study). By knowing where bottlenecks occur, policy can be modified to give quicker solutions and to make things easier not only for users, but for a companys IT support and infrastructure as well.
ITInformation Technology Staff -
IT staff can be better prepared to handle frustrated users and learn which type of technical problems related to which levels ofproduce the largest frustration. This can help things move more smoothly and even help IT staff make better recommendations to managers and policymakers. IT Staff should also be better situated as the middleman and be able to get better information between users and vendors when understanding frustration in the workplace.
Policymakers, in industry and government, should recognize the severity of the productivity loss due to user frustration. Increased research funding, improved training, better data collection, and increased public awareness of the problems will help produce appropriate changes.
Essentially one ideal situation in which the implication might be turned into to practices is this. A frustrated user is dealing with a system problem. They contact the IT staff, explain the source of the frustration. The IT staff tries to help as best they can, then relays a summary of the incident to the manager, who then evaluates the frustration and talks to the policy makers. Together they improve the process of handling such incidents by making small adjustments. In addition they are better informed for future decisions that involve potentially frustrating systems. This results in filtering back down to the user, who now has an easier time when a frustrating incident occurs and can be more productive and recover quicker. IT Staff can then do their job more efficiently and give better information to the managers This repeatedly refined process raises the bar with each iteration and makes a workplace better as a whole and is based on understanding the frustrating incidents themselves (and this understanding results from studies such as this).
Summary
Acknowledgements
The lead author of this article was partially supported by Training Grant No. T42/CCT310419 from the Centers for Disease Control and Prevention/National Institute for Occupational Safety and Health. The contents are solely the responsibility of the author and do not necessarily represent the official views of the National Institute for Occupational Safety and Health.
We appreciate partial support from National Science Foundation grant for
Information Technology Research (#0086143) Understanding the Social Impact of
the Internet: A Multifaceted Multidisciplinary Approach and National Science Foundation grant for the Digital Government Initiative (EIA 0129978): Towards a Statistical Knowledge Network.
We acknowledge the assistance of Deborah Carstens and Robert Hammell, who both provided comments on an earlier draft of this paper. REFERENCES
QUOTE EN.REFLIST Bandura, A. (1973). Aggression: A Social Learning Analysis. Englewood Cliffs, NJ: Prentice-Hall.
Bandura, A. (1986). Social Foundation of Thought and Action: A Social-Cognitive Theory. Englewood Cliffs, NJ: Prentice Hall.
Bandura, A. (1997a). Self-Efficacy. Harvard Mental Health Letter, 13(9), 4-6.
Bandura, A. (1997b). Self-Efficacy: The Exercise of Control. New York: Freeman.
Barker, R. (1938). The Effect of Frustration upon the Cognitive Ability. Character and Personality, 7, 145-150.
Barker, R., Dembo, T., & Lewin, K. (1965). Frustration and Regression: An Experiment with Young Children. In R. Lawson (Ed.), Frustration: The Development of a Scientific Concept. New York: MacMillan Publishing Co.
Baron, R. A. (1977). Human Aggression. New York, NY: Plenum.
Berkowitz, L. (1978). Whatever happened to the frustration aggression hypothesis? American Behavioral Scientist, 21(5), 691-708.
Bessiere, K. (2002). COMPUTER FRUSTRATION: A TEST OF A SOCIAL-PSYCHOLOGICAL MODEL. Unpublished Masters Thesis, University of Maryland, College Park, College Park, MD.
Bessiere, K., Lazar, J., Ceaparu, I., Robinson, J., & Shneiderman, B. (2002). Social Psychological Factors of End User Frustration and Their Implications. forthcoming.
Bias, R., & Mayhew, D. (Eds.). (1994). Cost-Justfying Usability. San Francisco: Morgan Kaufmann Publishers.
Britt, S. H., & Janus, S. Q. (1940). Criteria of Frustration. The Psychological Review, 47(6), 451-469.
Brosnan, M. J. (1998). The Impact of computer anxiety and self-efficacy upon performance. Journal of Computer Assisted Learning, 14, 223-234.
Cambre, M. A., & Cook, D. L. (1985). Computer Anxiety: Definition, Measurement, and Correlates. Journal of Educational Computing Research, 1(1), 37-54.
Campion, M., & Lord, R. (1982). A Control Systems Conceptualization of the Goal-Setting and Changing Process. Organizational Behavior and Human Performance, 30, 265-287.
Ceaparu, I., Lazar, J., Bessiere, K., Robinson, J., & Shneiderman, B. (20023, in press). Determining Causes and Severity of End-User Frustration. International Journal of Human-Computer Interactionforthcoming.
Ceaparu, I., and Shneiderman, S. (2004, in press). Finding Governmental Statistical Data on the Web: Three Empirical Studies of the FedStats Topics Page. Journal of the American Society for Information Science and Technology.
Clarke, J. (2001). Key factors in developing a positive user experience for children on the web: A case study. Proceedings of the Human Factors and the Web 2001, Available at: HYPERLINK "http://www.optavia.com/hfweb/index.htm" http://www.optavia.com/hfweb/index.htm
Cohen, B. A., & Waugh, G. W. (1989). Assessing Computer Anxiety. Psychological Reports, 65(1), 735-738.
Compeau, D. R., & Higgins, C. A. (1995). Computer Self-Efficacy: Development of a Measure and Initial Test. MIS Quarterly, 19(2), 189-211.
Daniel, J., and Lazar, J. (2004). Improving the Usability of Governmental Statistics Presented on the Web. Working paper.
Dollard, J., Doob, L. W., Miller, N. E., Mowrer, O. H., & Sears, R. R. (1939). Frustration and Aggression. New Haven: Yale University Press.
Ferster, C. B. (1957). The Function of Aggression and the Regulation of Aggressive Drive. The Psychological Review, 71, 257-272.
Fowler, F. (1993). Survey Research Methods (2nd ed.). Newbury Park, California: Sage Publications.
Freud, S. (1921). Types of Onset and Neurosis. In J. Strachey (Ed.), The Standard Edition of the Complete Psychological Works of Sigmund Freud (Vol. 12, pp. 227-230). London: Hogarth Press.
Gilroy, F., & Desai, H. (1986). Computer Anxiety: Sex, Race, and Age. International Journal of Man-Machine Studies, 25(1), 711-719.
Glass, C. R., & Knight, L. A. (1988). Cognitive Factors in Computer Anxiety. Cognitive Therapy and Research, 12(4), 351-366.
Hu, J. (2002). AOL's Pop-Up Sacrifice. CNET News, pp. Available at: HYPERLINK "http://news.com.com/2100-1023-962345.html" http://news.com.com/2100-1023-962345.html.
Jex, S. M., & Gudanowski, D. M. (1992). Efficacy Beliefs and Work Stress: An Exploratory Study. Journal of Organizational Behavior, 13, 509-517.
Keenan, A., & Newton, T. J. (1984). Frustration in Organizations: Relationships to Role Stress, Climate, and Psychological Strain. Journal of Occupational Psychology, 57, 57-65.
Kemp, T. (2001, November 28, 2001). Macy's Doubles Conversion Rate. InternetWeek.com, pp. Available at: HYPERLINK "http://www.internetwk.com/story/INW20011128S20010004" http://www.internetwk.com/story/INW20011128S20010004.
Lazar, J. and Huang, Y. (2003). "Improved Error Messages for Web Browsing: An Exploratory Experiment". In J. Ratner (ed.) Human Factors and Web Development, 2nd Edition, Mahwah, NJ: Lawrence Erlbaum Associates, 167-182.
Klein, H. J., & Kim, J. S. (1998). A Field Study of the Influence of Situational Constraints, Leader-Member Exchange, and Goal Commitment on Performance. Acadamy of Management Journal, 41(1), 88-95.
Lincecum, L. (2000). The Effects of Software Disruption on Goal Commitment, Task Self-Efficacy, Computer Self-Efficacy, and Test Performance in a Computer-Based Instructional Task. Unpublished Doctoral Dissertation, Texas Tech University.
Locke, E. A. (1996). Motivation Through Conscious Goal Setting. Applied Preventative Psychology, 5, 117-124.
Locke, E. A., & Latham, G. P. (1990). A Theory of Goal Setting and Task Performance. Englewood Cliffs, NJ: Prentice Hall.
Locke, E. A., & Latham, G. P. (2002). Building a Practically Useful Theory of Goal Setting and Task Motivation: A 35-Year Odyssey.Unpublished manuscript.
Loyd, B. H., & Gressard, C. (1984). The Effects of Sex, Age, and Computer Experience on Computer Attitudes. AEDS Journal, 18(2), 67-77.
Maurer, M. M. (1994). Computer Anxiety Correlates and What They Tell Us. Computers in Human Behavior, 10(3), 369-376.
McInerney, V., McInerney, D. M., & Sinclair, K. E. (1994). Student Teachers, Computer Anxiety, and Computer Experience. Journal of Educational Computing Research, 11(1), 177-189.
Meier, S. (1985). Computer Aversion. Computers in Human Behavior, 1(1), 171-179.
Mowrer, O. H. (1938a). Preparatory Set (Expectancy) -- a Determinant in Motivation and Learning. Psychological Review, 45, 62-91.
Mowrer, O. H. (1938b). Some Research Implications of the Frustration Concept as Related to Social and Educational Problems. Character and Personality, 7, 129-135.
Murphy, C., Coover, D., & Owen, S. (1989). Development and Validation of the Computer Self-Efficacy Scale. Educational and Psychological Measurement, 49, 893-899.
Murrell, A. J., & Sprinkle, J. (1993). The Impact of Negative Attitudes Toward Computers on Employees' Satisfaction and Commitment Within a Small Company. Computers in Human Behavior, 9, 57-63.
Nash, J. B., & Moroz, P. A. (1997). An Examination of the Factor Structures of the Computer Attitude Scale. Journal of Educational Computing Research, 17(4), 341-356.
Nielsen, J. (1994). Usability Engineering. Boston: Academic Press.
NTIA. (2001). A Nation Online: How Americans are Expanding Their Use of the Internet. Washington, DC: National Telecommunications and Information Administration.
Peters, L. H., Chassie, M. B., Lindholm, H. R., O'Connor, E. J., & Kline, C. R. (1982). The Joint Influence of Situational Constraints and Goal Setting on Performance and Affective Outcomes. Journal of Management, 8(2), 7-20.
Peters, L. H., & O'Connor, E. J. (1980). Situational Constraints and Work Outcomes: The Influences of a Frequently Overlooked Construct. Acadamy of Management Review, 5(3), 391-397.
Pew Internet and American Life Project. (2003). The Ever-Shifting Internet Population: A new look at Internet access and the digital divide. Report available at: http://www.pewinternet.org/reports/toc.asp?Report=88 and downloaded on April 17, 2003
Peters, L. H., O'Connor, E. J., & Rudolf, C. J. (1980). The Behavioral and Affective Consequences of Performance-Relevant Situational Variables. Organizational Behavior and Human Performance, 25(79-96).
Phillips, J. S., & Freedman, S. M. (1984). Situational Performance Constraints and Task Characteristics: Their Relationship to Motivation and Satisfaction. Journal of Management, 10(3), 321-331.
Raub, A. (1981). Correlates of Computer Anxiety in College Students. Unpublished Doctoral Dissertation, University of Pennsylvania.
Riseberg, J., Klein, J., Fernandez, R., & Picard, R. (1998). Frustrating the user on purpose: Using biosignals in a pilot study to detect the user's emotional state. Proceedings of the CHI 1998: ACM Conference on Human Factors in Computing Systems, 227-228
Roberts-Witt, S. (2001, September 25, 2001). A Singular Focus. PC Magazine.
Shneiderman, B. (1982). System message design: Guidelines and experimental results, In Directions in Human-Computer Interaction, Badre, A. and Shneiderman, B. (Editors), Ablex Publishing Company: Norwood, NJ, 55-78.
Shorkey, C. T., & Crocker, S. B. (1981). Frustration theory: a source of unifying concepts for generalist practice. Social Work, 26(5), 374-379.
Spector, P. E. (1975). Relationships of Organizational Frustration with Reported Behavioral Reactions of Employees. Journal of Applied Psychology, 60(5), 635-637.
Spector, P. E. (1978). Organizational Frustration: A Model and Review of the Literature. Personnel Psychology, 31(815-828).
Storms, P. L., & Spector, P. E. (1987). Relationships of Organizational Frustration with Reported Behavioural Reactions: The Moderating Effect of Locus of Control. Journal of Occupational Psychology, 60, 227-234.
Tedeschi, B. (1999, August 30, 1999). Good Web Site Design Can Lead to Healthy Sales. The New York Times
Torkzadeh, G., & Angulo, I. E. (1992). The Concepts and Correlates of Computer Anxiety. Behavior and Information Technology, 11(1), 99-108.
Villanova, P., & Roman, M. A. (1993). A Meta-Analytic Review of Situational Constraints and Work-Related Outcomes: Alternative Approaches to Conceptualization. Human Resource Management Review, 3(2), 147-175.
Vroom, V. (1964). Work and Motivation. New York: Wiley.
Pre-Session Survey (Appendix A)
Email address: __________________________________
Section I: Demographic Information
1. Age __________
2. Gender: F M
Education:
___ High School Graduate___ Fresh/Soph in College___ Jr./Sr. in College___ College Graduate___ Masters Degree___ Doctoral-level
In what field are you employed? ______________________________
What is your job title? ________________________________
Section II: Computer Experience and Attitudes
How many years have you been using a desktop or laptop computer for home or work use? _______
How many hours per week do you use a desktop or laptop computer? ______
What type of Operating System is installed on the computer that you are currently using?
___ DOS___ MacOS___ Unix/Linux___ Windows 95___ Windows NT___ Windows 98___ Windows ME___ Windows 2000___ Windows XP
What type of applications and programs do you typically use? (check all that apply)
___ Email___ Other Internet Use___ Graphic Design Programs___ Chat/Instant Messaging___ Word Processing___ Programming Tools___ Web Browsing___ Spreadsheet Program (Excel)___ Database management/ Searching___ Presentation Tools (powerpoint)___ Other (please explain)
______________________________ Multimedia (audio/video)
How many years have you been using the internet? ___________
How many hours per week do you spend online? Please indicate the amount of time that you are actually using the computer while online, not simply the amount of time you are connected to the internet. _________
At work, do you ___ Have a permanent connection to the internet OR ___ dial in through a modem
Which of the following do you do when encountering a problem on the computer or application that you are using?
___ try to fix it on my own___ consult a manual/help tutorial___ Ask help desk/ consultant for help ___ Ask a friend/relative for help___ Give up or leave it unsolved
How sufficient is your computer software and/or hardware for the work that you need to do?
Not at All 1 2 3 4 5 6 7 8 9 Very Sufficient
Section III: For the following questions, please choose the number that best corresponds to your feelings
1. Computers make me feel:
Very Uncomfortable1 2 3 4 5 6 7 8 9 Very Comfortable
2. When you run into a problem on the computer or an application you are using, do you feel:
Anxious 1 2 3 4 5 6 7 8 9 Relaxed/Indifferent
3. When you encounter a problem on the computer or an application you are using, how do you feel about your
ability to fix it?
Helpless 1 2 3 4 5 6 7 8 9 Confident I can fix it
4. How experienced do you think you are when it comes to using a computer?
Very Inexperienced 1 2 3 4 5 6 7 8 9 Very Experienced
5. When there is a problem with a computer that I can't immediately solve, I would stick with it until
I have the answer.
Strongly Disagree 1 2 3 4 5 6 7 8 9 Strongly Agree
6. If a problem is left unresolved on a computer, I would continue to think about it afterward.
Strongly Disagree 1 2 3 4 5 6 7 8 9 Strongly Agree
7. Right now, how satisfied with your life are you?
Very Unsatisfied 1 2 3 4 5 6 7 8 9 Very Satisfied
8. How often do you get upset over things?
Not Very Often 1 2 3 4 5 6 7 8 9 Very Often
9. Right now, my mood is:
Very Unhappy 1 2 3 4 5 6 7 8 9 Very Happy
FRUSTRATING EXPERIENCE FORM(Appendix B)
Please fill out this form for each frustrating experience that you encounter while using your computer during the reporting session. This should include both major problems such as computer or application crashes, and minor issues such as a program not responding the way that you need it to. Anything that frustrates you should be recorded.
What were you trying to do?
On a scale of 1 (not very important) to 9 (very important), how important was this task to you?
Not very important 1 2 3 4 5 6 7 8 9 Very Important
What software or program did the problem occur in? If the problem was the computer system, please check the program that you were using when it occurred (check all that apply).
___ ___ email_______ file browsers___ ____presentation software
(e.g. (e.g. powerpoint)___ ___ chat and instant messaging___ ___ spreadsheet programs
(e.g. (e.g. excel)mult____media (audio/video software)___ ___ web browsing___ ___ graphic design________other _________________________ other internet use___ ___ programming tools)))_____ word processing_______ database management/
s searching software
Please write a brief description of the experience:
How did you ultimately solve this problem? (please check only one)
___ I knew how to solve it because it has happened before___ I ignored the problem or found an alternative solution___ I figured out a way to fix it myself without help___ I was unable to solve it___ I asked someone for help. Number of people asked _______I tried again___ I consulted online help or the system/application tutorial____I restarted the program___ I consulted a manual or book___ I rebooted
Please provide a short step by step description of all the different things you tried in order to resolve this incident.
How often does this problem happen? (please check only one)
___ more than once a day ___ one time a day ___ several times a week ___ once a week
___ several times a month ___ once a month ___ several times a year ___ first time it happened
On a scale of 1 (not very frustrating) to 9 (very frustrating), how frustrating was this problem for you?
Not very frustrating 1 2 3 4 5 6 7 8 9 Very frustrating
Of the following, did you feel:
___ angry at the computer ___ angry at yourself ___ helpless/resigned
___ determined to fix it ___neutral ___ other: ___________
How many minutes did it take you to fix this specific problem? (if this has happened before, please account only for the current time spent) _____________________________
Other than the amount of time it took you to fix the problem, how many minutes did you lose because of this problem? (if this has happened before, please account only for the current time lost; e.g. time spent waiting or replacing lost work). ____________
Please explain:
12. Until this problem was solved, were you able to work on something else?
____Yes ____No
Please explain:
Post-Session Survey (Appendix C)
Email address: __________________________________
For the following questions, please circle the number that best corresponds to your feelings.
1. Right now, my mood is:
Very Unhappy 1 2 3 4 5 6 7 8 9 Very Happy
2. We asked you to record your frustrating experiences. Overall, how frustrated are you after these experiences?
Not Frustrated at All 1 2 3 4 5 6 7 8 9 Very Frustrated
3. How will the frustrations that you experienced affect the rest of your day?
Not at All 1 2 3 4 5 6 7 8 9 Very Much
4. Are the incidents that occurred while you were recording your experiences typical of your everyday computer experience?
Yes No
5. In general, do you experience more or less frustrating incidents while using a computer on an average day?
Less 1 2 3 4 5 6 7 8 9 More
6. Did these frustrating experiences impact your ability to get your work done?
No impact 1 2 3 4 5 6 7 8 9 Severe impact
7. Did these frustrating experiences impact your interaction with your co-workers?
No impact 1 2 3 4 5 6 7 8 9 Severe impact
PAGE
PAGE 2
PAGE \# "'Page: '#' '" broken sentence
2 3 A B N O _ ` o p
5 6 7 8 2 3 g h p q 實化 j CJ U
j CJ UH htCJ H hCJ CJ cH dh CJ cH dh'SFH h'SFCJ CJ cH dh+SFH h+SFCJ CJ H*CJ aJ CJ
5CJ H*\5CJ PJ \
5CJ \ 5 3 3
6 7 ] x 3 h N $d d C$Eƀ t [$\$a$ $H$ a$ H$ $a$
d ! # $ 7 G I J K [ \ ^ _ r " . 3 ? I ^ # 6
H h(u&
H h/(u&
cH dh;ԁ&
H h;ԁ&5H htCJ j CJ UjY CJ Uj CJ Uj CJ UCJ H htCJ
j CJ UH ht0J CJ 9 # $ -
b :! ) 3 3 *3 8 A eK d ` d d [$\$ $a$ $d d [$\$a$ +
X
d
i
n
& - 8 : h i $ % \ ] ^ _ ` * = ž
H h(u&
cH dh(u&
H h(u&
cH dh(u&
H h(u&
cH dh+SF
H h+SF j U
H h(u&
cH dh(u&
H h(u&5
cH dh
H h
cH dh"xF
H h"xF
H h4
cH dh(u& 4 2 _ e i k l m v ~ c d I! P! ! ! # # # # $ $ 0& 1& B& C& y' z' ) ) ) ) M) N) * * * * + + o- p- - - #. $. / / / / / / 1 1 1 1 3 *3 z4 {4 >6 ?6 U6 V6 6 6 6
cH dhl&
H hl& j U
cH dh(u&
cH dh(u&
cH dh(u&
H h(u& 56\]P6
7 7
7 ;7 <7 8 8 8 8 u9 v9 : : ; ; ]; ^; '= (= >= ?=
> > ? ? ? ? Z@ [@ A A A A B B QE RE hE iE zE E E 3F 4F G G G G G H H I I ;K ? % & # $ ˽Ҷ {{t
H h(u& H htcH dh(u& j H htU
H ht
H ht
cH dh,SF
H h,SF j U
cH dht H htcH dh(u&
H h(u&
H ht
H h{t
H hzt
H hwt
H hyt
H hxt6]- X o ' G d C$Eƀ t G d C$Eƀ t G d C$Eƀ t P Q 9 : " ' ( y { } ~ T U b c e f 0 1 = > i j V W ž
H h(u& H htcH dh(u& j H htU
H ht
cH dhx&
H hx&6
H ht
H ht
cH dht
cH dht
H h(u&
H ht
cH dht ; T U V 1 A : ; F ( < > $ ) 1
\ żżżżżż߳߬ ~ ~ ~ j U
H h0uF
H h(u&
cH dh(u&5H ht5 5cH dht
cH dht 5cH dht 5cH dhtt
cH dhtt
cH dhst j UcH dht
cH dht
H ht
H htj H htU0X U V z { / 0 1 J K C D 9 : ; F = >
d ` d ]^` d ` d \
@ A + 8 E W X Y \ ] c l s t ykykyk H htcH dh(u&
H ht
H h(u&
cH dh(u& \cH dht H ht\ H h, 8 : ' 9
d `
d ` H
C$Eƀ 6
D E h 8 R
d C$Eƀ 0uF ` R
d C$Eƀ 7 `
d C$`
d ` $ ( / 5 6 9 B O C D E M f g xj\ H h(u&cH dh6 H h0uFcH dh6 H h0uF5\cH dh6
H h0uF
H h7
H h1SF
H h0SF
H h/SF
H h;f
H hy&
H h;f
H h;f
H ht
H ht
H h(u&
H ht
H ht
H ht H htcH dh(u&!g h o p ' , ߲xjcU H hcH dh-SF
H h-SF H hcH dhy&
H hy& H h0uF5\cH dh6 H h0uF5\cH dh6 H h0uF5\cH dh6 H h0uF5\cH dh6 H h0uFcH dh6 H h0uFcH dh6 H h0uFcH dh6 # H h0uFB*cH dh6ph h U
d $Eƀ 0uF If ` l U
d $Eƀ 0uF If ` $$If l l\ K x^ t F 0 6 4
l a R W
d $C$Eƀ 0uF If ` U
d $Eƀ 0uF If ` l # H C$Eƀ 7 ` $$If l ^\ K x^ t F 0 6 4
l a , - 8 2 R U " * ( [ ǹDz㲫|unu _ H ht5cH dh(u&
cH dht
cH dht 5cH dht 5 H h65cH dh
cH dh6 H h0uFcH dh6
H h-2 W2 I $C$Eƀ 2 F2 L2 P2 V2 w2 x2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 X3 Z3 3 3 3 3 3 3 3 74 b4 i4 t4 ۽䶨xj H h+
> > > > > > > > > > > ³ԀqԀq³˪˘˪b˪ H h5cH dh;f H hGvF5cH dh;fH hGvF5 H hFvF5cH dh;fH h;f5 H hFvF5 H h@5 H h@5cH dhH h5 H hA5 H hAvF5 H hAvFcH dhJ
H hAvF H hAvFcH dhF%= = = = = m $ $ $ I $C$Eƀ AvF If $$If l \ 6
, V 0 4
l a = = = > > $ $$If l %\ 6
, V 0 4
l a I $C$Eƀ AvF If > > > > $ $$If l %\ 6
, V 0 4
l a I $C$Eƀ AvF If > > > > > m> Y Y R
d C$Eƀ ;f ` R
d C$Eƀ AvF ` > > > $> &> '> (> -> ]> l> m> n> > > > > > > > > ? ? "? #? ? B? C? G? H? L? M? R? S? Y? Z? `? a? ټټټټټټټ~~~o`` H hFvF5cH dhB H hGvF5cH dhB H hHvF5cH dhB H hAvFaJ cH dhB H hAvF5\cH dhB H hAvFcH dh9 H hAvFcH dhB H hAvF5cH dh9 H hAvF5cH dhB H hAvF5cH dh, n> > > > > ? #? =? q q q q q q q I $C$Eƀ AvF If D C$Eƀ AvF =? >? 4 $$If l ֞ fH 1* ! 0 4
l a >? C? H? M? S? Z? a? h? I $C$Eƀ AvF If a? g? i? j? k? p? r? s? t? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? @ ̽uguYugugH H h0uF5\cH dh,G NG SG VG [G n| e \ \ \ nx e \ \ \ $$If a$ $$If a$ $$If l 4 \ # 0 4
l al
[G \G jG lG nG rG sG G G G G n\ e \ \ \ n e \ \ \ $$If a$ $$If a$ $$If l 4 \ # 0 4
l al
G G G G G G G G G G G nx e \ \ \ n e \ \ \ $$If a$ $$If a$ $$If l 4 \ # 0 4
l al
G G G G G G G H
H H H nx e \ \ \ n e \ \ \ $$If a$ $$If a$ $$If l 4 \ # 0 4
l al
H H *H .H 0H 6H 7H LH PH RH WH n e \ \ \ n e \ \ \ $$If a$ $$If a$ $$If l 4 \ # 0 4
l al
WH XH ^H bH eH jH kH qH vH zH nL e \ \ \ nX V \ \ $If $$If a$ $$If a$ $$If l 4 \ # 0 4
l al zH H H H e c $$If l 4 \ # 0 4
l al $$If a$ H H P P P P I I I R
d C$Eƀ & `
d ` R
d C$Eƀ ` J K K 'K /K ~K K L !L HL QL ]L hL jL L L BM IM PM SM 6N DN N N #O *O 1O pO O O O P P P P P !P "P #P &P PP P P P P ιΤ yr
cH dhW` ?` e` I I R
d C$Eƀ 0uF `
d ` R
d C$Eƀ ` W W W W W X 1X 3X 5X ` E` F` G` e` ` ` DzDzǦǦǦǟ{q`{VL H hx5\H h0uF5\ H h0uF5\cH dh H h5\H h0uF5\H h0uFB*ph H h(u&cH dh
cH dhj H hU H hcH dh
H h
H h
H h H hcH dh{ H hcH dh{
H h{
H h e` w` ` ` R W
d $C$Eƀ x If ` U
d $Eƀ 0uF If ` ` ` ` l < U
d $Eƀ 0uF If ` m $$If l 0 T4
T 0 6 4
l a ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` a a a
a >a ?a @a Aa Ba Ha Ia Ja Ka La Oa ں |m^U 5cH dht H ht5cH dh0uF H h0uF5cH dh H h5cH dh{ 5cH dh{
cH dh+u&
cH dh0uFH h0uFW zH h0uFW zH h0uFW zH h0uF5\H h0uF5\H h0uF5\H h0uF5\
H h0uFH h0uF5\ ` ` ` 9X m $$If l ^0 T4
T 0 6 4
l a X $
d $Eƀ 0uF If ` a$ ` ` ` P X $
d $Eƀ 0uF If ` a$ U
d $Eƀ 0uF If ` ` ` ` ` < U
d $Eƀ 0uF If ` m $$If l ^0 T4
T 0 6 4
l a ` ` ` 9h m $$If l ^0 T4
T 0 6 4
l a X $
d $Eƀ 0uF If ` a$ ` ` ` P X $
d $Eƀ 0uF If ` a$ U
d $Eƀ 0uF If ` ` ` ` 0 < U
d $Eƀ 0uF If ` m $$If l ^0 T4
T 0 6 4
l a ` ` ` 9( m $$If l ^0 T4
T 0 6 4
l a X $
d $Eƀ 0uF If ` a$ ` a a P X $
d $Eƀ 0uF If ` a$ U
d $Eƀ 0uF If ` a a ?a ? R
d C$Eƀ 0uF ` m $$If l ^0 T4
T 0 6 4
l a ?a @a Aa Ba a a a a I J $@& C$Eƀ x If $@& If
d ` R
d C$Eƀ 0uF ` Oa a a a a a a a a a a a a a a a a a a a a a b b Qb Rb Sb Ub mb tb ub vb wb b b b b b b úvj j H h+u&U H ht5cH dh+u&5H h0uF5 H h0uF5 H h0uF5cH dhH h5 H h&5 H h0uF5 H h+u&W z\
W z\ W z\cH dh{ aJ cH dh{ 5cH dh{
cH dh{ &a a a a a a a a a a a a a a h ~ ~ ~ v ~ $@& If
$$@& If a$ $@& If i $$If l 0 T4
T 0 4
l a
a a a a a a a a a a a a a b v \
! $@& If i $$If l 0 T4
T 0 4
l a
$$@& If a$ $@& If
b b b b b b Sb 0 r
d ` $@& If i $$If l 0 T4
T 0 4
l a
$$@& If a$ Sb Tb Ub Vb Wb Xb Yb Zb [b \b ]b Y Y Y Y Y Y Y Y R
d C$Eƀ & ` R
d C$Eƀ 0uF `
]b ^b _b `b ab bb cb db eb fb gb hb ib jb kb lb mb R
d C$Eƀ & ` mb b b b Y I
d ` R
d C$Eƀ t ` R
d C$Eƀ +u& ` b b b b b b b b b Oc rc tc d d d d d
d d Pd Qd e e e *e ˷~ne~Un~LCLC H hCJ H hCJ H hCJ cH dh H hCJ H hCJ cH dh H hCJ H hCJ H hCJ cH dh H hCJ
H h 5B*ph & H h(u&5B*cH dh+u&ph & H ht5B*cH dh+u&ph j H h+u&U j Uj&aB
UV
H h+u& b b b b b b e f u s D C$Eƀ D C$Eƀ *e e e f f f f f f f f f f g "g #g %g 'g 5g 6g 7g xng`VLgE; H h0uF5\
H h0uFH h0uF5\H h0uF5\
H h0uF
H h0uFH h0uF5\ H h0uFcH dh H hcH dh
H h
H h0uF# H h0uFB*cH dh&ph H h0uFB*ph # H htB*cH dh0uFph H htcH dh0uF
cH dhH hCJ H hCJ cH dhf f f f f u 0 - C$ D C$Eƀ 0uF D C$Eƀ t D C$Eƀ t f f
g g #g &g Jd ? 1
$
! $If a$
! $If m $$If l 0 T V 0 6 4
l a G $Eƀ 0uF If &g 'g 6g 8g 9g Fg Ig H < D < J $$Eƀ 0uF If a$
! $If m $$If l 0 T V 0 6 4
l a 7g 9g Eg Fg Hg Jg Vg Wg Yg [g og pg rg tg g g g g g g g g g g g g g g g g g g _h »tmf]V
H hH hCJ
H h
cH dh0uF H h0uF5W \$ H htW cH dh0uF H h0uF5\
H h0uFH h0uF5\H h0uF5\
H h0uF
H h0uFH h0uF5\H h0uF5\H h0uF5\
H h0uF
H h0uFH h0uF5\ Ig Jg Wg Zg [g pg sg D < d < J $$Eƀ 0uF If a$
! $If m $$If l 0 T V 0 6 4
l a sg tg g g g g g h < ` < J $$Eƀ 0uF If a$
! $If m $$If l 0 T V 0 6 4
l a g g g g g l < J $$Eƀ 0uF If a$
! $If m $$If l 0 T V 0 6 4
l a g g g g g u 0 0 D C$Eƀ D C$Eƀ 0uF D C$Eƀ 0uF g g m m m o o D C$Eƀ 0uF D C$Eƀ d C$ _h fh gh qh vh h h h h h h h h h h h h h h i i i !i Li Mi hi ni xi }i i i j $j *j .j /j 1j 2j 5j Cj Uj Wj k ιDzΫΝz H hcH dh
H h H hcH dh H hcH dh
H h
H h H hcH dh
H h
H h H hcH dh
H h H hcH dh
H h *k l cl dl hl ml zl l l l l l l l l m *m m m m m m m m
n n ^n fn n ݬ{qg]gSg H h/SF5\H h5\H h5\H h5\
H h0uF H htcH dh0uF H h0uFcH dh H h0uFcH dh
H h H hcH dh
H h
H h H hcH dh
H h
H h[ T i q D C$Eƀ t H C$Eƀ t i v e e e I $C$Eƀ t If Q
& F
hhC$Eƀ t ^h _ _ _ I $C$Eƀ t If W $$If l F d
t$ 6 4
l a ] H C$Eƀ t ^Y $$If l F d
t$ 6 4
l a = > x y z i i i i i D C$Eƀ t Q
& F
hhC$Eƀ t ^h Q j j j j M
$C$Eƀ t If H
& F C$Eƀ t ) * H H H H L H H M
$C$Eƀ t If j $$If l \ d
V $ 6 4
l a * + , - F l $$If l ;\ d
V $ 6 4
l aM
$C$Eƀ t If - . j M
$C$Eƀ t If H
& F C$Eƀ t L
C$Eƀ t ^ [ [ W $$If l F $ 6 4
l a M
$C$Eƀ t If 7 j U U
m$C$Eƀ t If ^`mU
d$C$Eƀ t If ^`d j k S U
h$C$Eƀ t If ^h`W $$If l F $ 6 4
l a [ W $$If l F $ 6 4
l a M
$C$Eƀ t If " V j j j j H
& F C$Eƀ t L
hC$Eƀ t ^h - ] U
t$C$Eƀ t If ^`tM
$C$Eƀ t If - . Q r s [ [ [ M
$C$Eƀ t If W $$If l F *$
6 4
l ah s t u Y L
hC$Eƀ t ^hY $$If l ;F *$
6 4
l ah u g O $
hC$Eƀ t ^ha$H
& F C$Eƀ t | } n % H hC$Eƀ t ^h D C$Eƀ t K $hC$Eƀ t ^ha$ | } s t
} ~ v w ^
/ a b ǺѳѪ } 0J j U0J mH nH u0J
j 0J UH ht5CJ \
H htH htCJ
H htH htCJ OJ QJ H ht5>*
H htH htaJ H htaJ H ht5\aJ H ht0J 1} E k 4 7 $K$L$If X 0 l6 3 4 a J $C$Eƀ t If K$I $C$Eƀ t If E } 4 4 4 I $C$Eƀ t If 7 $K$L$If X 0 l6 3 4 a J $C$Eƀ t If K$ s t u
}
4 4 I $C$Eƀ t If 7 $K$L$If X 0 l6 3 4 a J $C$Eƀ t If K$
t D 4 J $C$Eƀ t If K$I $C$Eƀ t If 7 $K$L$If X 0 l6 3 4 a 4 } ~ ( 4 x J $C$Eƀ t If K$I $C$Eƀ t If 7 $K$L$If X 0 l6 3 4 a ~ k 4p 7 $K$L$If X 0 l6 3 4 a J $C$Eƀ t If K$I $C$Eƀ t If - v w x k 4 7 $K$L$If X 0 l6 3 4 a J $C$Eƀ t If K$I $C$Eƀ t If } 4 I $C$Eƀ t If 7 $K$L$If X 0 l6 3 4 a J $C$Eƀ t If K$ z * O $C$Eƀ t ]^a$ L C$Eƀ t ]^ 7 $$If - . .6 3 4 - a ^ _ m L C$Eƀ t ]^ D C$Eƀ t _ { | } Z Z L C$Eƀ t ]^X
& F
C$Eƀ t ]^` } 5 Z Z L C$Eƀ t ]^X
& F
C$Eƀ t ]^` 5 6 Z X
& F
C$Eƀ t ]^` L C$Eƀ t ]^ / G e e e e Q $C$Eƀ t If ]^ H C$Eƀ t ] G H m X X X X H X X X Q $C$Eƀ t If ]^ U $$If l F `' `' 4
l al
3 O P Q k Xt X U $$If l F `' `' 4
l al Q $C$Eƀ t If ]^ ) * Z Z X
& F
C$Eƀ t ]^` L C$Eƀ t ]^ * e 3 E F sX s8 st s B $$If l 0 ( T , ( 4
l a I $C$Eƀ t If
sH & M $C$Eƀ t If ] B $$If l 0 ( T , ( 4
l a I $C$Eƀ t If o o L C$Eƀ t ]^ B $$If l 0 ( T , ( 4
l a T U V W X ] ^ _ Z Z Z Z Z Z Z Z L C$Eƀ t ]^X
& F
C$Eƀ t ]^`
_ ^ H C$Eƀ t ]X
& F
C$Eƀ t ]^` & ' ( e e L C$Eƀ t ]^ L C$Eƀ t ]` ( J V P C$Eƀ t ]^` X
& F
C$Eƀ t ]^` Z Z X
& F
C$Eƀ t ]^` L C$Eƀ t ]^
e L C$Eƀ t ]^ L C$Eƀ t ]^ / a b d u 0 0 0 D C$Eƀ t D C$Eƀ t D C$Eƀ t % ' 7 8 r t r r r r G $C$Eƀ t a$ D C$Eƀ t
t k l m L pC$Eƀ t ^p` D C$Eƀ t l * , r r r r r r i g &`#$ D C$Eƀ t G $C$Eƀ t a$
D C$Eƀ t &`#$
j 0J U #
01h / =!"#$% D y K j l a z a r @ t o w s o n . e d u y K 2 m a i l t o : j l a z a r @ t o w s o n . e d u D y K a j o n e s 5 @ t o w s o n . e d u y K 4 m a i l t o : a j o n e s 5 @ t o w s o n . e d u D y K k a t i e b @ c m u . e d u y K , m a i l t o : k a t i e b @ c m u . e d u D y K i r i n a @ c s . u m d . e d u y K 0 m a i l t o : i r i n a @ c s . u m d . e d u D y K b e n @ c s . u m d . e d u y K , m a i l t o : b e n @ c s . u m d . e d u D d F$P J
C A ? " 2 f ~gfXM B # w`!: ~gfXM" k Y p> xZ]h\E>_CLbqi(B֤iB%i5Z|(
l5 $Ԣy!%X)Z}(&B B/
A "4wdv7ݠɞΙ3|f" J?
P
Y̐ lVx&>bcWn\7l[D4 ƴ]!ir_\%<ѺVhGP? n]k^FP} عg_+dfe*)8B]/r
]6hb/b
2y Y=v\
:!Q,x/Պ^
8d| ߠ~ެ k hxEi6d]vQ^ST%U{NN U;jcSնZfd=:rx>t:$ӉugsCE+;zx 弿'Ь>Ifϧ9>^.C=
q";@>D<,r]]k22`O6@G!FG.hg]\KF}m@%S˒_?gDf2A$FNSmR~FM0" k$e$.Ff(XY,3 edZOSaߧd2idr& *E~1]'ѷN#KD@8."^hK/#2d*
E@YmU`DwLF:Ta=V 0\F"ܚU c*dH#܈o@+ 'c#HZ'(Z0̯LFGuxނfM9 1gm~MaΊuw'aOq/k
?j~ɣdI1ډCsiJJ"'g$/x&Z mم]uaikua>7~ yQX63
;
-((+%VrN*,RcoÞJqW \952 R^kuۈvιr+hս uwz=[(Tnɚ ;Μ;}Et|r C
ZpHvs,qn}U62q5 qj6qu5W/jApպ
Z UO!~\\e \\?WmۀDZ W
qU-?\M;U"WGrqU)j':V1tq,__+#Ry!~ |->oGh2]#QI$*ķRqG? OwvwIDPQ_o 1I57(j
9<**w]GĎ9k&-RWJ^ ՝DXDbcF퉚|`O[ȍoĺfʈbg`I]Ath*̂JOQ 4) 4dv5"횿Vxj[ɪQx~;"aV3{zʞKآϳ83N^;s<5Tۗ˚{r䉭# 0
:v\v߇%$Vm:.X4;gǸW.s[Q)g^7OY%oYˠ\S&HoSEVptkzn]PF̈WN]9YˉY? u~:6PiLKJ?X]M1Wrq &;ܞdgw=sןۂ<Ɉ/ ?*`5 Ԯw.բdے}U.*V}yޛf7Z>-Y2Ƙ\sQ oj I6]&
! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~
! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~
A , @ ! " # $ % & ' ( ) * + F . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? G B C D E H I J K N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~ R o o t E n t r y
F 7[7 D a t a
W o r d D o c u m e n t % O b j e c t P o o l `[7 7[7 _ 1 1 1 3 6 5 4 8 2 2 ! F `[7`[7 O l e
P R I N T
$ C o m p O b j b
! " # $ % & ' ( ) * + , - . / 0 2
! F Microsoft Excel Chart Biff8 Excel.Chart.8 9q Oh +'0 @ H X h
ajones ajones Microsoft Excel @ r >@ s1? ?,
9
' ' Arial w@
HSw\w 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - "System
0 - ' -
- -
l
! ! - - - ' - - -
- - - ' - - - k
! ! - -
$ Cb*bCb - $ CbCZCb $ ZZ
- bC pC ~C ! ! C / / C > > C L L C Z Z - - - ' - - - k
! ! -
$ *bCb -
$ CbCZCb $ ZZ - - - ' - - - k
! ! - - ss - ss ss
$ E EE - MM - MM MM $ E E - $ MEMEME - ss - ss ss $ M - MM - MM MM $ EEEvv EE - $ EEEEE - ss - ss ss $ Ev/E - MM - MM MM $ E '' E - $ E EE - ss - ss ss $ ' - MM - MM MM $ E E - $ `E`E`E - ss - ss ss $ ` - MM - MM MM $ WEW_ WE - $ EWWEE - ss - ss ss $ W_B_W - MM - MM MM $ E99 E - $ EEE - ss - ss ss $ 9 - MM - MM MM $ E( E - $ rEr((ErE - ss - ss ss $ (r(( - MM - MM MM $ iEih iE - $ #E#iiE#E - ss - ss ss $ ihTh#i - - - bC C bC b# pC p# ~C ~# C # C # C # C # 2
2 0 .
2
@ 10. .
2
N 20. .
2
] 30. .
2
k 40. .
2
z 50. .
2
60. . bC b* bC C b b bU U b b bg g b b bz z b* * - - - - ' - - ,
8
N N Arial w@
Sw\w 0 - 2
1 Not Very Frustrating - - - - ' - - k
! ! - - - - ' - - L
N N Arial w@
Sw\w 0 - 2
20 - - - - ' - - k
! ! - - - - ' - - ~
N N Arial w@
Sw\w 0 - 2
K 30 - - - - ' - - k
! ! - - - - ' - - .
N N Arial w@
Sw\w 0 - 2
40 - - - - ' - - k
! ! - - - - ' - - ^
N N Arial w@
Sw\w 0 - 2
50 - - - - ' - - k
! ! - - - - ' - -
N N Arial w@
Sw\w 0 - 2
] 60 - - - - ' - - k
! ! - - - - ' - - A
N N Arial w@
Sw\w 0 - 2
70 - - - - ' - - k
! ! - - - - ' - - p
N N Arial w@
Sw\w 0 - 2
80 - - - - ' - - k
! ! - - - - ' - - R !
N N Arial w@
Sw\w 0 - 2
n 9 Very Frustrating - - - - ' - - k
! ! - - ' - -
- - ' - - k
! ! - - ' - -
- - - - ' - -
V
9 2
k ! Number of Frustrating Experiences < . G . . . 3 . * . . . 7 * . . . . * . * - - - ' - -
- - ' - -
-
- - - ' - - -
- - - ' - - -
- -
52
' 2
E Number of Frustrating< . G . . . 3 . * . . . 2
LE Experiences 7 * . . . . * . * - - - ' - - -
- - - ' - - -
- - - ' - - -
- -
l
! ! - - '
' ' O b j I n f o
W o r k b o o k - $ S u m m a r y I n f o r m a t i o n ( D o c u m e n t S u m m a r y I n f o r m a t i o n 8 T '@ \ p ajones B a = = $ < X@ " 1 A r i a l 1 A r i a l 1 A r i a l 1 A r i a l 1 . T i m e s N e w R o m a n 1 . T i m e s N e w R o m a n 1 A r i a l 1 A r i a l 1 A r i a l 1 A r i a l "$"#,##0_);\("$"#,##0\) ! "$"#,##0_);[Red]\("$"#,##0\) " "$"#,##0.00_);\("$"#,##0.00\) ' " "$"#,##0.00_);[Red]\("$"#,##0.00\) 7 * 2 _("$"* #,##0_);_("$"* \(#,##0\);_("$"* "-"_);_(@_) . ) ) _(* #,##0_);_(* \(#,##0\);_(* "-"_);_(@_) ? , : _("$"* #,##0.00_);_("$"* \(#,##0.00\);_("$"* "-"??_);_(@_) 6 + 1 _(* #,##0.00_);_(* \(#,##0.00\);_(* "-"??_);_(@_) "Yes";"Yes";"No" "True";"True";"False" "On";"On";"Off" + ) , * 8@ @ 8 @ 8@ 8 8@ 8 ` T Chart1 - Sheet1 " Sheet2 # Sheet3 `i Z R 3 @ @
q Level of Frustration! Number of Frustrating Experiences 1 Not Very Frustrating 9 Very Frustrating
'@ M \ \ K G R E E N I D G E \ H P L a s e r J e t 2 2 0 0 S ? d X X L e t t e r . HP LaserJet 2200 Series PCL 6 2 6xePM/CQs絗j!7Jʢ? em!̋nR$DbgcYu{Y9ܜܩmd "npt
2NCG!cA <='
una`y"&Qm*d^-Jcod}:&q Y
> s +$M/!'7i C9tB.Z-@w~p rjľIly6rR22klhgRI,ɸYM\Fp rޯ Gt@?N " d XX ? ? 3 `
# `
#
? D3 d 2 3 M
N M 4
3 Q :
F !N u m b e r o f F r u s t r a t i n g E x p e r i e n c e s Q ;
Q ;
Q 3 _ 4 E 4 D $ % M 3 O & Q 4 $ % M 3 O &
Q 4 F A B C g
3 O B C g
3 b P P PZR R
R R R R V
V d M !
O S S 4 3 * N@ $@ # M ! M ! M
N M 4 3 "
: , d d Q _ 3 O Q % M 3 O Q 4 4 4 S S 4 % / L B M < 3 O Q ' 4 S S
4
e 5 1 N o t V e r y F r u s t r a t i n g 2 3 4 5 6 7 8 - 9 V e r y F r u s t r a t i n g e @ @ *@ @ @ 7@ @@ I@ e >
'@
d MbP?_ * + % M \ \ K G R E E N I D G E \ H P L a s e r J e t 2 2 0 0 S ? d X X L e t t e r . HP LaserJet 2200 Series PCL 6 2 5xePM/CQsZD͢@-y
-ERXXH$65GQۘřsz:nNI&m&Ig s),t 2@{(Pp هm8I$
lڋf"rDM$5*Zj)t rK)RZe 4kN+'EM)Bp k$v[!cPk f# f@l跚?Tb$<9b)7Z1rx+t9K2n;WP3 Xȓf!] 7aW8N " d XX ? ? U } m' } 6 ; ; ; ; ; ; ; ; ; ;
;
~
@ @ @ @ @ *@ @ @ @ @ @ 7@ @ @@
~
I@
(
p
6 N M M ? ] P' K ] ` '@ " ? ? 3 `
# `
# 93 d 2 3 M
N M 4
3 Q :
F !N u m b e r o f F r u s t r a t i n g E x p e r i e n c e s Q ;
Q ;
Q 3 _ 4 E 4 D
$ % M 3 O & Q 4 $ % M 3 O & Q 4 F A B E ^
3 O B E ^
3 b # M !
O 4 3 * # M ! M ! M
N M 4 3 "
: d d k 3 O % M 3 O Q 4 4 4 4 % 3 N @ M < 3 O Q ' 4 4
e e e
> @
7
'@
d MbP?_ * + % " ? ?u U > @ 7
'@
d MbP?_ * + % " ? ?u U > @ 7
՜. +,0 $ P X
* Center for Applied Information Technology a Sheet1 Sheet2 Sheet3 Chart1 Worksheets Charts Oh +'0 [ pvEw }w1
,8yI<9
VGx$86zJg|6تیG\;߱Ÿv+X}jQ3kڬa렰uA 8n ǜQznƲ Qdlʄ*EcL)P1sFE {:s4dѓZ$䏯Dbr3,]q-5} ,]sB) 2^Kطo=U
]s*" 6
G#mp'h"NV,ND8&+Px00Fd(1b
@ e=#쓕r#z~]߃cfpܿ`ߪ7}r8&+P1|= {G6YBnQhRq&+Yƅ3
=z4:)p#>5\l Q
k a
bQ&!@D4)ڈ}%""IM6b_ciDDW""dѤh#88&FD4)ڈ}%""IM6b_ bEe'#ƁLK27yrՖ$uj m]f
EX "p!.>&#$ː8 O('# !"#2$("HIQј]!%E4eH D]FGd2DI Xt4RD}D&'s%2f"& CńBG:ef"zrF{Rd1'c,2@>䦺ZD>%>:p>4|\ߓZ
=N":ZV],B/zp;h`kN$_ .zK%kbLB@ %"ʤCOSRWcDDI ^&/D%}jADki{rfP͔C@9GK"Y, Ci(b\oOW
mNmmznhsCFФ&Ж qcМő ׂdE "P @DD2H @ E""q DDa #L5
6о2$N:eDa]T [GkaBQ#SSC'+L=ԦwM6,;SJ nb~+<6Zk4r&Q1߸|8u
ig7D"d, ,B>J92l"7XHQЩ$q
R}(g'Aoba od^*E [ODT7FH c$'4è7fDwEHGsVhD}ÐW " {F 19 V%0uR LJ0M#@D$v@" d8""07VJxB r 1E7x*Ge IENDB` D y K y K N h t t p : / / w w w . o p t a v i a . c o m / h f w e b / i n d e x . h t m D y K y K T h t t p : / / n e w s . c o m . c o m / 2 1 0 0 - 1 0 2 3 - 9 6 2 3 4 5 . h t m l D y K y K j h t t p : / / w w w . i n t e r n e t w k . c o m / s t o r y / I N W 2 0 0 1 1 1 2 8 S 2 0 0 1 0 0 0 4 1 T a b l e M 8 S u m m a r y I n f o r m a t i o n ( D o c u m e n t S u m m a r y I n f o r m a t i o n 8 C o m p O b j 1 j
< H T
` l t | THEORY HEO Katie Bessiere ati ati Normal.dot e Sheila Denn 2 ei Microsoft Word 9.0 @
@ , ;@ [7@ [7 F ՜. +,D ՜. +,8 h p
Home n [
THEORY Title @ 7 ? w _PID_HLINKS _AdHocReviewCycleID _EmailSubject
_AuthorEmail _AuthorEmailDisplayName A h < U 5 h t t p : / / w w w . i n t e r n e t w k . c o m / s t o r y / I N W 2 0 0 1 1 1 2 8 S 2 0 0 1 0 0 0 4 V * h t t p : / / n e w s . c o m . c o m / 2 1 0 0 - 1 0 2 3 - 9 6 2 3 4 5 . h t m l ; $ ' h t t p : / / w w w . o p t a v i a . c o m / h f w e b / i n d e x . h t m J h t t p : / / w w w . h c i r n . c o m / ; 0 h t t p : / / w w w . h c i b i b . o r g / h m a i l t o : b e n @ c s . u m d . e d u i m a i l t o : i r i n a @ c s . u m d . e d u S m a i l t o : k a t i e b @ c m u . e d u j m a i l t o : a j o n e s 5 @ t o w s o n . e d u * m a i l t o : j l a z a r @ t o w s o n . e d u [ - NEW draft of the workplace frustration paper orE ajones5@towson.edu r Adam Jones s
F Microsoft Word Document
MSWordDoc Word.Document.8 9q !
i 8 @ 8 N o r m a l CJ _HaJ mH sH tH 2 @ 2 H e a d i n g 1 $@& 5\2 @ 2 H e a d i n g 2 $@& 5\> > H e a d i n g 3 $$d @&a$ 5\F @ F H e a d i n g 4 $dh @& 5CJ W \ @ @ @ H e a d i n g 6 $$@&a$ 5PJ \aJ < A@ < D e f a u l t P a r a g r a p h F o n t B C@ B B o d y T e x t I n d e n t d ` J Y J D o c u m e n t M a p -D M
OJ QJ ^J , @ , F o o t e r
! & )@ !& P a g e N u m b e r 0 B@ 20 B o d y T e x t $a$ aJ 8 ^@ B8 N o r m a l ( W e b )
d d aJ ( "@ ( C a p t i o n 5\6 >@ b6 T i t l e $d d [$\$a$ CJ 6 '@ q6 C o m m e n t R e f e r e n c e CJ 0 @ 0 C o m m e n t T e x t CJ . P@ . B o d y T e x t 2 CJ F O F b o d y h e d 1 % 56CJ OJ QJ \]aJ o( ph " W@ " S t r o n g 5\H T@ H
B l o c k T e x t ]^ CJ OJ QJ ^J 4 J@ 4 S u b t i t l e $a$
5CJ \. U@ . H y p e r l i n k >*B*ph > V@ > F o l l o w e d H y p e r l i n k >*B* ph . Q@ . B o d y T e x t 3 5 I r i n a z I L= , / 3
6 7 ] x 3 h # $ -
b : % / / */ 4 = eG K Q U U U [ [ [ j ]o u z , - F Є k ̣ ͣ o p , X U V z { / 0 1 J K C D 9 : ; F = > 8
:
' 9 D E h [ \ q r ~ D ] ^ D E [ u v 6 8 9 G I J \ ^ _ t v w " " " " " " " " " " " " # # # # C# F# G# l# n# o# # # # # # # # # # # # # $ $ $ $ "$ #$ .$ 1$ 2$ Q$ S$ T$ U$ [$ _$ `$ a$ $ $ $ ) ) ) ) ) ) }* * + + ]+ ^+ - - !- "- #- $- Y- Z- [- - - - - . >. W. X. g. j. m. p. t. z. . . . . . 0 0 Y4 Z4 [4 5 5 5 5 5 6 '6 :6 [6 z6 6 6 6 6 6 6 6 6 6 6 6 6 (7 )7 M7 l7 7 7 7 7 7 7 7 8 8 8 8 8 8 !8 "8 #8 8 8 8 8 9 W9 9 9 9 9 9 9 9 9 : : : : : : : : : m: n: : : : : ; #; =; >; C; H; M; S; Z; a; h; i; j; k; ; ; ; ; ;
< < < !< "< #< $< ? ? ? A A A pB B B B B B B B B B B B C C C C C C C 2C 6C 9C =C >C NC SC VC [C \C jC lC nC rC sC C C C C C C C C C C C C C C C C C C C C D
D D D D *D .D 0D 6D 7D LD PD RD WD XD ^D bD eD jD kD qD vD zD D D D D L L L L L L L L L L L L L L M %M NM hM nM M M M M M M N N N KN kN qN N N N N N O O 3O MO SO TO qO O O O O O O O P P )P [P aP bP rP P P P P P P P P Q Q Q 7Q =Q _Q Q Q Q Q Q Q Q Q Q Q
R 4R :R ;R \ ?\ e\ w\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ] ] ] ?] @] A] B] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ] ^ ^ ^ ^ ^ ^ S^ T^ U^ V^ W^ X^ Y^ Z^ [^ \^ ]^ ^^ _^ `^ a^ b^ c^ d^ e^ f^ g^ h^ i^ j^ k^ l^ m^ ^ ^ ^ ^ ^ ^ ^ ^ a b b b b b b
c c #c &c 'c 6c 8c 9c Fc Ic Jc Wc Zc [c pc sc tc c c c c c c c c c c c c c c i i i i i i j k k l l n n n n o Eo Fo Vo io jo to ~o o o o o o o o "r #r $r %r v v ^v _v zv v v v v v v v v w w w w w %w &w 'w x x x } } } .~ /~ : < = J Œ ƒ ϒ ~ ֙ י # $ % & ' 8 C У g % ̦ t H ֨ n ͫ ܬ D ϭ J X y z $ % h L ( K ŷ _ ] a ) ս w Y h @ t . = > T i v = > x y z Q ) * + , - . 7 j k " V - . Q r s t u | } E s t u
t 4 } ~ - v w x ^ _ { | } 5 6 / G H m 3 O P Q k ) * e 3 E F T U V W X ] ^ _ & ' ( J / a b d % ' 7 8 r t k l * , 0 0 X 0 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 3 0 0 0 0 0 0 0 0 0 0 / 0 / 0 / 0 / 0 / 0 / 0 / 0 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 U 0 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 b 0 0 c 0 c 8 0 c 0 c 0 0 0 i 0 i 0 i 0 0 i 0 i 0 i 0 i 0 i 0 i 0 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 n 0 0 0 0 0 0 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 ' 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 @ 0 @ 0 @ 0
0 6 !W Dp \ g , # & e* - 0 %2 t4 [: !< < = > a? @ A F J P .T W Z ` Oa b *e 7g _h k n 4r 8u z { y 2 [ R Ŗ k @
! ' ) , / 4 7 @ A C H L Q U W a d k { | eK p X > h r u ! 8! ! ! & F' n' ( a( - - ]/ #1 Z1 W2 X2 2 2 4 : : : : ); ; ; < !< < = = > > m> =? >? h? i? ? ? @ !@ #@ E F G [G G G H WH zH H P P P nQ R KR SS qS [T rT T U U U U mV ~V V V DW [W ~W W e` ` ` ` ` ` ` ` ` ` a ?a a a b Sb ]b mb b f f &g Ig sg g g g m r Es s s $v ^z z z { &{ . = ϖ ם ϱ z L h = i * - j - s u } E
~ _ } 5 G * _ ( t l " # $ % & ( * + - . 0 1 2 3 5 6 8 9 : ; < = > ? B D E F G I J K M N O P R S T V X Y Z [ \ ] ^ _ ` b c e f g h i j l m n o p q r s t u v w x y z } ~ p # J [ ^
h $ _ c 0" B" y# % % M% & & ' o) ) #* + + + - - z0 >2 U2 ;3 4 4 u5 6 7 ]7 '9 >9
: ; ; Z< = = > QA hA 3B C C E ;G LG H |I I "M ~O O S `T nT 6V W X \ ^ '^ ^ ` ` a c c *d f 7f m n n p Dr Tr s Ou gu v x x ^{ } $} Ё A ~ b k ޑ } s 9 L N ù } " # M p 3 G > % # 9 T b e 0 = i V
@ DX eZ Z ^ ^ ^ $ 3 8 \ j } ڬ ! г I XXXXXQtQtQ4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4Q4QtQtQtQtQtQtQtQtQtQtQtQtQtQtQtQtQTQTQTQ4Q4Q4QtQQQQQQQtQtQQtQQQ:XX#tXXXĕ ! ! / ! A B O 2 $ _u&4GҐU ] &b $ 3?d&4W & $ &2 $ ?,u8 o &@ A 0 (
B
S ? _ T o c 2 5 7 5 6 0 1 8 _ T o c 2 6 4 4 4 4 1 0 _ T o c 2 6 4 4 4 6 7 4 _ T o c 2 6 4 4 4 8 2 7 _ T o c 2 5 7 5 6 0 1 9 _ T o c 2 6 4 4 4 4 1 1 _ T o c 2 6 4 4 4 6 7 5 _ T o c 2 6 4 4 4 8 2 8 _ T o c 2 5 7 5 1 2 9 7 _ T o c 2 5 7 5 1 3 5 0 _ T o c 2 5 7 5 1 3 9 8 _ T o c 2 5 7 5 4 3 7 9 _ T o c 2 5 7 5 4 4 7 3 _ T o c 2 5 7 5 4 7 0 5 _ T o c 2 5 7 5 6 0 2 0 _ T o c 2 6 4 4 4 4 1 2 _ T o c 2 6 4 4 4 6 7 6 _ T o c 2 6 4 4 4 8 2 9 _ T o c 2 5 7 5 6 0 2 1 _ T o c 2 6 4 4 4 4 1 3 _ T o c 2 6 4 4 4 6 7 7 _ T o c 2 6 4 4 4 8 3 0 _ 1 1 1 3 6 5 4 6 0 8 / / / / / / / / / / U U U U ^
@ )/ )/ )/ )/ )/ )/ )/ )/ )/ )/ U U U U ^
L=:z z p ! # \ !+ (+ P2 T2 2 2 O 3O P P nR ~R 7 ? ̂ { 4 5 7 ] k ) 1 = D У ף N W % - ̦ Ԧ
˧ ӧ H O ֨ ܨ n u 0 8 N Y n v D K J Q W [ s y ޮ j q z | % ( 3 = ? R _ c n v ] f l u a g # E K h s @ G t { R ] . 7 ? E h ( 6 S b nL zL j k dr r ! g$ $ * * * * 0 0
9 9 B B @H QH M "M NM RM M M M M M M KN NN N N N N O O O 2O 3O 6O qO tO O O O O P P )P ,P P P P P P P P P Q Q _Q bQ Q Q Q Q
R R JR MR nR }R ~R R R R
S S tS wS S S T T bn mn {v v Gz Qz E{ T{ 1 Ɔ z h i l S V V X C F ϣ 2 3 h i g { $ x ˦ f g r t G H ^ _ ը 6 m n ۩ ˫ C 0 1 W X j ͯ y # I 9 g q ʸ 0 \ ] չ ! 5 ` x º ^ _ ( P ü ļ V W * H I Z D E s l q N i Y e ? J = E ( * % 0 1 O V t % : H X _ q | , 0 g k H L } N S ? E 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 p ! # \ ^ w. x. ~. ~. . . 9 9 9 9 9 9 9 9 9 9 9 9 : : : : : : D D n n s s t t v v v v { } dž ʆ 4 8 k z ˫
j l a z a r } D : \ D o c u m e n t s a n d S e t t i n g s \ j l a z a r \ A p p l i c a t i o n D a t a \ M i c r o s o f t \ W o r d \ A u t o R e c o v e r y s a v e o f w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . a s d j l a z a r E C : \ l a z a r \ r e s e a r c h \ f r u s t r a t i o n \ w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . d o c j l a z a r E C : \ l a z a r \ r e s e a r c h \ f r u s t r a t i o n \ w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . d o c j l a z a r E C : \ l a z a r \ r e s e a r c h \ f r u s t r a t i o n \ w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . d o c j l a z a r } D : \ D o c u m e n t s a n d S e t t i n g s \ j l a z a r \ A p p l i c a t i o n D a t a \ M i c r o s o f t \ W o r d \ A u t o R e c o v e r y s a v e o f w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . a s d j l a z a r E C : \ l a z a r \ r e s e a r c h \ f r u s t r a t i o n \ w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . d o c j l a z a r E C : \ l a z a r \ r e s e a r c h \ f r u s t r a t i o n \ w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . d o c j l a z a r } D : \ D o c u m e n t s a n d S e t t i n g s \ j l a z a r \ A p p l i c a t i o n D a t a \ M i c r o s o f t \ W o r d \ A u t o R e c o v e r y s a v e o f w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . a s d j l a z a r E C : \ l a z a r \ r e s e a r c h \ f r u s t r a t i o n \ w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . d o c S h e i l a D e n n F D : \ D a t a \ g o v s t a t \ w e b p a g e \ p a p e r s \ w o r k p l a c e _ f r u s t r a t i o n _ f u l l _ p a p e r _ v 7 . d o c < @|a ut)Lk 6O=FP xULڔ ^`o( . ^` . pL p^p`L . @ @ ^@ ` . ^` . L ^`L . ^` . ^` . PL P^P`L . h h^h`o( . 8 8^8` . L ^`L . ^ ` . ^ ` . xL x^x`L . H H^H` . ^` . L ^`L . h h^h`o( . 8 8^8` . L ^`L . ^ ` . ^ ` . xL x^x`L . H H^H` . ^` . L ^`L . ^`o( . ^` . pL p^p`L . @ @ ^@ ` . ^` . L ^`L . ^` . ^` . PL P^P`L . xU < 6O=F ut) E h [ \ r D ] ^ D E [ u v 6 8 9 G I J \ ^ _ t v w " " # # C# F# G# l# n# o# # # # # # # # # # # # # $ $ $ $ "$ #$ .$ 1$ 2$ Q$ T$ U$ [$ `$ a$ Y- Z- [- - - - - . >. W. X. g. j. m. p. t. z. . . . 5 5 6 '6 :6 [6 z6 6 6 6 6 6 6 6 6 6 6 6 (7 )7 M7 l7 7 7 7 7 7 7 7 8 8 8 8 8 8 !8 8 8 8 W9 9 9 9 9 9 9 9 9 : : : : : : m: n: : : : : ; #; =; >; C; H; M; S; Z; a; h; i; ; ; ; ; ;
< < < !< pB B B B B B B B B B B B C C C C C C C 2C 6C 9C =C >C NC SC VC [C \C jC lC nC rC sC C C C C C C C C C C C C C C C C C C C C D
D D D D *D .D 0D 6D 7D LD PD RD WD XD ^D bD eD jD kD qD vD zD D D L L L L L L L %M nM M M N N qN N N O SO TO O O O P aP bP P P P P Q Q =Q Q Q Q Q Q Q :R ;R < E n a b l e d > 1 < / E n a b l e d > < S c a n U n f o r m a t t e d > 1 < / S c a n U n f o r m a t t e d > < S c a n C h a n g e s > 1 < / S c a n C h a n g e s > < / E N I n s t a n t F o r m a t > Y< E N L a y o u t > < S t y l e > A P A P u b l i s h e d < / S t y l e > < L e f t D e l i m > { < / L e f t D e l i m > < R i g h t D e l i m > } < / R i g h t D e l i m > < F o n t N a m e > T i m e s N e w R o m a n < / F o n t N a m e > < F o n t S i z e > 1 2 < / F o n t S i z e > < R e f l i s t T i t l e > < / R e f l i s t T i t l e > < S t a r t i n g R e f n u m > 1 < / S t a r t i n g R e f n u m > < F i r s t L i n e I n d e n t > 0 < / F i r s t L i n e I n d e n t > < H a n g i n g I n d e n t > 7 2 0 < / H a n g i n g I n d e n t > < L i n e S p a c i n g > 0 < / L i n e S p a c i n g > < S p a c e A f t e r > 0 < / S p a c e A f t e r > < / E N L a y o u t > N < E N L i b r a r i e s > < L i b r a r i e s > < i t e m > f r u s t r a t i o n . e n l < / i t e m > < / L i b r a r i e s > < / E N L i b r a r i e s > @ A ` @ U n k n o w n j l a z a r b e n . m e i s e l a j o n e s G z T i m e s N e w R o m a n 5 S y m b o l 3& z A r i a l I& ? ? A r i a l U n i c o d e M S 5& z a T a h o m a 7& V e r d a n a " 1 h ZZFZZF& F [ C 7 ! 0 d 3q H T H E O R Y K a t i e B e s s i e r e S h e i l a D e n n