alexa
Reach Us +44-1625-708989
Is it Possible to Assess an Organizationandrsquo;Preparedness for the Unforeseen? Development and Evaluation of a Methodology | OMICS International
ISSN: 2151-6200
Arts and Social Sciences Journal
Make the best use of Scientific Research and information from our 700+ peer reviewed, Open Access Journals that operates with the help of 50,000+ Editorial Board Members and esteemed reviewers and 1000+ Scientific associations in Medical, Clinical, Pharmaceutical, Engineering, Technology and Management Fields.
Meet Inspiring Speakers and Experts at our 3000+ Global Conferenceseries Events with over 600+ Conferences, 1200+ Symposiums and 1200+ Workshops on Medical, Pharma, Engineering, Science, Technology and Business
All submissions of the EM system will be redirected to Online Manuscript Submission System. Authors are requested to submit articles directly to Online Manuscript Submission System of respective journal.

Is it Possible to Assess an Organization’Preparedness for the Unforeseen? Development and Evaluation of a Methodology

Magnhild Kaarstad1* and Glenn-Egil Torgersen2

1Institute for Energy Technology IFE, Halden, Norway

2Division of Pedagogy and Methodology/Didactics, Norwegian Defense University College, Akershus fortress, Oslo mil/Akershus 0015 Oslo, Norway

*Corresponding Author:
Kaarstad M
Institute for Energy Technology IFE, P.O. Box 173
NO-1751 Halden, Norway
Tel: +47-69212200
E-mail: [email protected]

Received date: February 15, 2017; Accepted date: February 23, 2017; Published date: February 28, 2017

Citation: Kaarstad M, Torgersen G-E (2017) Is it Possible to Assess an Organization’s Preparedness for the Unforeseen? Development and Evaluation of a Methodology. Arts Social Sci J 8:254. doi:10.4172/2151-6200.1000254

Copyright: © 2017 Kaarstad M, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Visit for more related articles at Arts and Social Sciences Journal

Abstract

Several serious incidents are unforeseen to organizations, companies and actors when they occur. Organizations as well as individuals are challenged by continuous threats, accidents and unforeseen events. Unforeseen events have other characteristics than events that can easily be predicted based on historical data and experience. This paper describes the data collection concept Methodology for handling the unforeseen (UN-METH), developed within the Strategic Institute Initiative at IFE(Institute For Energy Technology), IO-EPO(Integrated Operations-Emergency Preparedness Organization), and uses the insight about the nature of the unforeseen developed through the Norwegian basic research and book project “Pedagogy for the unforeseen". UN-METH consists of two different approaches: UN-CAF (Unforeseen Competence Assurance Framework), where an organization’s preparedness plans are analyzed to determine to what extent they are considering the unforeseen, and UN-ORG (UNforeseen Organization questionnaire), which is a questionnaire that can be distributed to personnel in an organization, where individuals evaluate their organizations’ preparedness and ability to handle the unforeseen. The main purpose of this article is documenting the development and evaluation process of UN-ORG. This process was conducted to investigate the applicability, usefulness and relevance of the questionnaire directly with professionals with relevant experience in the area. The development and evaluation approach is based on methodological principles proposed by Stufflebeam. Interviews, a survey and a case study were used during the evaluation. The results indicated that the questionnaire is highly applicable, focuses on the unforeseen and that it covers an important area. Interviews further identified specific recommendations of items to improve and add. Publishing the findings from this development and evaluation process of the questionnaire, is a first step in making the method known for different organizations. By using UN-ORG, separately or in combination with UN-CAF, organizations can gain valuable insight into their own preparedness for the unforeseen, and the researchers can get useful input and gradually improve the methodology itself.

Keywords

The unforeseen; Preparedness; Quality assessment; Methodology

Introduction

Organizations and individuals are continuously challenged by threats, accidents and undesirable, surprising and sometimes incomprehensible events. Some models attempt to describe the development of risk situations and their barriers, such as the Bow-tie model [1-4] and Reasons' Swiss cheese model [5]. However, many events are unforeseen to organizations, which such models do not grasp in adequate way. Unforeseen events have different characteristics than events that can be predicted based on historical data and experiences. Depending on how well prepared organizations are, surprising and sudden events may in some situations form potentials and opportunities for learning and development. However, in other situations, such events may be disastrous. An unforeseen (UN) event can be defined as "a relatively unknown event or situation that occurs relatively unexpected and with relatively low probability or predictability to the individual, group or community that experience and handle the event" [2, p. 318]. Several studies point out that “These situations cannot be processed solely on the basis of fast associations and of easily applicable procedures” [6, p. 218].

An important question is how well prepared organizations are to meet unforeseen events, both in terms of anticipating and preventing the events, manage them, minimizing the consequences if they occur, and exploiting the potential of the event for the benefit of organizational learning. Emergency preparedness and response (EPR) requires large resources and extensive planning, particularly with regard to education, training and exercises. To acquire the best possible preparedness in relation to handling unforeseen events, insight into the nature of the unforeseen is needed.

A research project with a focus on developing knowledge within the area of the unforeseen resulted in a scientific anthology [2], which provides a foundation for further development of knowledge, perspectives and training programs in emergency preparedness and crisis management. A main idea in the book is that “the unforeseen” challenge our current experience and knowledge. Unforeseen situations are situations where what should not happen; do happen, despite rehearsed procedures and response patterns. EPR plans should thus, to a greater extent, also incorporate dimensions of creativity, improvisation, tacit knowledge and intuition.

There have been several attempts to evaluate organizational preparedness. Some assessments have focused on evaluating the resources and needs that exist in organizations more widely related to different kind of crisis [7-10]. Having the right equipment is important, however, this is not the only success factor for handling an event. Other efforts have looked into standards for preparedness plans [11,12]. In this context there exists fascinating suggestions for general mindset and mental approaches to plan, lead and make decisions related to complex and risky situations under unpredictable conditions [13]. However, such plans, standards and mindsets will not to a sufficient degree provide the needed outcome if the organizations and their infrastructure cannot deliver them. A plan to deliver supplies to an area that depends on one road and a single source of vehicles will be less likely to succeed than a plan that provides a variety of options. EPR plans should therefore also take into consideration aspects such as adaptation and redundancy.

In our research on emergency preparedness in organizations and principles for handling the unforeseen [1], we asked an overarching research question:

Is it possible to assess an organization’s preparedness, given that it does not know what it should be prepared for?

Supporting this research, a theoretical distinction between "General emergency preparedness" and "UN-preparedness" was introduced. "General emergency preparedness" encompasses preparedness for events that are considered known and predictable, while, "UNpreparedness" involves more specialized processes intended to prevent, master and exploit unforeseen events. We believe that organizations should focus on what unforeseen events may involve for them and develop plans and initiatives that focus on preparedness for unforeseen events in particular. These plans should be part of the organization's overall emergency preparedness plans.

Analyzing organizations’ preparedness plans will provide an impression as to which degree an organization is prepared to deal with the unforeseen or not. However, plans alone cannot provide a complete picture, as plans do not always correspond to the actual practice of the organization. Therefore, it is important to also examine to what extent and in what way the employees perceive that their organization is prepared to deal with the unforeseen.

This article documents the Methodology for handling the unforeseen (UN-METH), that has been developed within the Strategic Institute Initiative at IFE(Institute For Energy Technology), IOEPO( Integrated Operations-Emergency Preparedness Organization) [1], and is based on insight about the nature of the unforeseen established through the Norwegian basic research-and-book project “Pedagogy for the unforeseen" [2].

UN-METH consists of two different approaches (further explained in chapter 2 and illustrated in Figure 1):

arts-social-sciences-illustration

Figure 1: Illustration of UN-METH.

• UN-CAF (Unforeseen Competence Assurance Framework), where an organizations’ emergency plans are analyzed to determine to what extent they are considering the unforeseen, and

• UN-ORG (UNforeseen Organization questionnaire), which is an instrument that can be distributed to personnel in an organization, where individuals evaluate their organization’s preparedness and ability to handle unforeseen events.

UN-Methodology

The methodology for handling the unforeseen (UN-METH) is a holistic evaluation comprising UN-CAF and UN-ORG. The idea is that UN-CAF and UN-ORG can be used separately or in combination. The main purpose of the methodology is to collect central information about the organization's readiness, skills and ability to handle unforeseen events. Based on the results of the evaluation, an organization will be supported in assessing which areas they are well prepared, and which areas initiatives are needed to reinforce their preparedness further.

Figure 1 presents the Methodology for the Unforeseen, illustrating both UN-CAF, to the left, and UN-ORG, to the right.

The left side of the figure displays how unforeseen factors and the CAF method constitute the UN-CAF tool. Strategic emergency plans are the focus of the analysis. The right side shows how semi-structured interviews and self-assessment together constitutes the UN-ORG tool. Personnel are the focus of the analysis.

The UN-CAF and UN-ORG tools can be used separately or as a supplement to each other to consider whether there is an agreement between the analyzed emergency preparedness documents, and the opinion of individuals with regard to how they perceive their organizations’ emergency preparedness for handling the unforeseen. UN-CAF and UN-ORG does not have predefined interpretation keys that indicates the level of preparedness based on the total score. The results are instead used to assess the organization's quality or readiness level related to the organization's business area. This includes concrete proposals for the improvement potentials of the various UNcategories. Using the tools in combination may provide complementary information from different sources that contribute to support and strengthen the recommendations.

To carry out a study using UN-METH, a certain degree of contact with the organization is necessary - both in the data collection phase and possibly later related to implementation of recommendations. It is therefore essential that researchers and others who will work with this method approach the organization in a careful and competent manner and communicate in a language with examples that are understandable and relevant for the organization.

In order to accomplish this, a feasibility study should be performed to become familiar with the organization's culture, terminology, jargon and procedures. It is important to develop confidence so that the data collection and the dialogue between the various actors happen in the best possible way during the process. In a feasibility study, it is also possible to develop relevant examples for the organization at hand that later can be used in interviews and during the interpretation phase of the collected data.

It may be necessary to support the data collection with various forms of observations. Observations can be based on the UN-CAF factors defined, the feasibility study or topics that the organization itself find relevant and suggest that the researchers should focus on.

UN METH is a comprehensive methodological approach. A large amount of detailed data is collected. This data is compiled and will form the basis of a status assessment of the organization's emergency preparedness and response capacity for the unforeseen. The result can be used as a basis for developing specific improvements, if necessary. To evaluate whether an implemented improvement has had effect, UNMETH (or parts of UN-METH) can be used for a continuous quality assurance.

UN-CAF

UN-CAF analysis is based on the principles of Competence Assurance Framework (CAF), [14-16] which is a theoretical analysis method considering how good the relationship is between the levels in an organizations’ education and training. UN-CAF (UNforeseen Competence Assurance Framework) has been adjusted from the CAF methodology in order to help identifying to which degree there is a good correlation between an organization’s plans for general emergency preparedness, and its principles for dealing with the unforeseen. Further, the analysis can provide relevant input to adjustments of strategic preparedness documents in the organization at hand. Focus of the analysis is explicitly on competence in handling the unforeseen in emergency situations, and not on skills related to technical disciplines.

An UN-CAF analysis focuses on consistency between the organization's views and initiatives in relation to handling the unforeseen, as described in strategic principles, governing documentation and implemented emergency preparedness and response plans. Focus in the analysis are emergency preparedness and response plans and how these processes and emphasizes the unforeseen. An UN-CAF analysis will look at the extent to which an organization has expertise and emergency plans to be prepared for the unforeseen. The methodological phases of the UN-CAF analysis are:

• Definition of scope

• Selection of relevant documents

• Analysis of documents

• Prepare UN-CAF report with recommendations

• Implementation of advice

UN-CAF factors are selected during the definition of the scope of the analysis, based upon the specific focus of the organization at hand, and in which area it is considered that they will benefit most in developing. The factors are rooted in theoretical and practical research related to the unforeseen [1,2]. The UN-CAF methodology is already validated, and several studies in different organizations have been performed, using this method [14,15].

UN-ORG

As an additional source to identify organization´s preparedness for the unforeseen, a questionnaire was developed [1]. The purpose of the questionnaire is that employees with emergency experience in an organization assess the extent to which their organization has emergency preparedness and response plans and procedures that can handle the unforeseen.

The development and evaluation approach used for the UN-ORG questionnaire is based on methodological principles given in Stufflebeam and Shinkfield [17] and Stufflebeam [3], and used by e.g., Martz [18,19] and Fernandes, et al. [20]. The approach is called the Checklist Development Checklist (CDC), and follows the main principles for questionnaire development in general, although it is called checklist development. The approach consists of twelve main phases: (1) Focus the checklist task (2) Make a candidate list of checkpoints, (3) Classify and sort the checkpoints, (4) Define and flesh out the categories, (5) Determine the order of categories, (6) Obtain initial reviews of the checklist, (7) Revise the checklist content, (8) Delineate and format the checklist to serve the intended uses, (9) Evaluate the checklist, (10) Finalize the checklist, (11) Apply and disseminate the checklist, and (12) Periodically review and revise the checklist. The UN-ORG questionnaire has been developed and evaluated following these phases.

Chapter 3 of this article focuses on describing the development and evaluation process of UN-ORG. This process was conducted to verify the applicability, usefulness and relevance of this questionnaire directly with professionals with pertinent experience in the area.

Development and Evaluation of the UN-ORG Questionnaire

In phase (1), Focus the questionnaire, a knowledge base was developed regarding the Unforeseen, including review of relevant literature, and involvement of pertinent experts from the research project “Pedagogy for the unforeseen” [2]. The rationale for this phase is that checklist/questionnaire developers must establish a sound foundation for the intended questionnaire. Only then can the questionnaire be specifically targeted, coherent, possessing integrity, be valid, credible, and helpful to an identified population. Our starting point was to cover as many aspects of the unforeseen as possible to be able to identify how an organization in the best possible way can be prepared for the unforeseen.

Phase (2), Make a candidate list of items, was performed after the knowledge base in step (1) was established. What was done in this stage was to generate a working list of indicators with descriptors and associated definitions. Based on research related to the unforeseen [2], approximately 100 indicators were identified. Examples of indicators are:

• "In my organization, we train on observing relevant details during an incident," (classified into UN-ORG category 8, “Training”, in phase 3).

• "In my organization, we have thoroughly debrief during and directly after an incident" (classified into UN-ORG category 9, “Concurrent learning”, in phase 3).

• "In my organization we are trained in identify risks" (classified into UN-ORG category 7, “Identification of risks”, in phase 3).

• "In my organization we are able to deal with uncertainty in an unforeseen situation" (classified into UN-ORG category 4, “Capabilities for handling the unforeseen”, in phase 3).

The questionnaire is made in such a way that the respondent answer to each statement on a scale from one to ten.

Then, the (3) Indicators were classified and sorted. Similar items were grouped together. This helped in showing the particular items’ strength and weakness, and which categories that should be improved or reinforced. Grouping the indicators also helped to see gaps and areas of overlap, which is important for expanding and refining the questionnaire. The indicators were grouped into ten categories:

1. General preparedness: Coping and organization of basic skills.

2. Emergency plans for the unforeseen: Existing plans for the management and prevention of particular unforeseen events.

3. Understanding of “the unforeseen”: The employees and the organization's definition, description and perception to particular unforeseen events that is relevant for the organization.

4. Capabilities for handling the unforeseen: Coping and organization of basic skills particularly for unforeseen events.

5. Improvisation: Employees and the organization’s ability to improvise and find creative solutions during unforeseen conditions.

6. Flexibility: Employees and the organization’s ability and willingness to adapt their logistics and administrative system to the situation at hand.

7. Identification of risk: The organization´s procedure to identify and pursue warning signs.

8. Training: Continuous competence development and training programs for unforeseen events.

9. Concurrent learning: The organization's ability to emphasis observation and learning during events.

10. Interaction: Employees and the organization´s ability to collaborate internally and externally when events occur.

The philosophy is that each category provides an average score that can give input to the organization in determining in which category or which area they are well prepared, and in which area they need to improve.

The next phase was to (4) Define the categories. Once the indicators were categorized, the categories were defined.

Then, the (5) Order of the categories was determined, and we assured that the indicators in each category were sequenced logically and functionally. As our categories cover distinct topics, not sequentially dependent on each other the order is not as important as it may be in other types of questionnaires. However, we ensured that “general emergency preparedness” was the first category, and then the more specific categories followed.

Now, (6) an initial review was performed. Individual interviews with professionals were scheduled with a case-oriented approach. Interviews were conducted with two experts with experience from emergency preparedness and evaluation research. They were introduced to the anticipated use of the questionnaire, and its possible focus areas. The indicators in the questionnaire were individually analyzed, attending to format and content as well as the discussion of inclusion/removal of specific indicators. Each interview took about 2 hours. Some indicators were not clear enough and proposals for clarification were made. A few indicators were suggested added to the questionnaire, and a few were suggested deleted, as they did not have additional value, or were almost duplicated. Another important result of the interview was a suggestion to add 10 general sum-statements for each main category in the questionnaire, to be distributed to organizations as a pre-assessment before a more thorough evaluation is performed. Such a preassessment can help focus the main evaluation using UN-METH.

Based on the input in phase (6), we (7) revised the questionnaire. The acquisition of independent feedback is important for improving questionnaires, but such feedback is worthwhile only if it is seriously considered and appropriately applied. We made notes on whether and how to make improvements regarding each input; and then adjusted the questionnaire, taking into account the whole set of information, making sure its meaning and focus was not lost.

In phase (8), we discussed how to Delineate and format the questionnaire to serve the intended uses. An online evaluation survey was available in three weeks to participants asked to evaluate the questionnaire. The participants were asked to provide critical feedback, focusing on its strengths and weaknesses, if there were items that should be added or removed, if there were confusing items – and also give specific recommendations.

The survey included an introductory section where basic information on the participants was solicited, such as the area of expertise and years of experience.

Another set of questions consisted of a rating scale of nine specific criteria based on Stufflebeam’s [21] eight criteria for evaluating a checklist: 1) Applicability: The questionnaire is applicable to organizational preparedness for the unforeseen); 2) Clarity: The items in the questionnaire are clear; 3) Comprehensiveness: The questionnaire allows a comprehensive perspective of the unforeseen; 4) Concreteness: The items in the questionnaire are concrete/objective; 5) Ease of use: The questionnaire is easy to use; 6) Impartiality: The items in the questionnaire are impartial/unbiased; 7) Parsimony: The questionnaire covers an adequate amount of topics (shows parsimony); 8) Pertinence: The items included are pertinent/appropriate. In addition, we added, as recommended by Fernandes et al. [20] an additional category, 9) Usefulness: The questionnaire is useful for evaluating on organizations’ preparedness for the unforeseen. Each closed-ended item used an interval response format from 1 to 10, where 1=strongly disagree and 10=strongly agree. The rationale underlying the number of scale points used was that an expanded scale would allow respondents to more accurately express their positive or negative feelings about the questionnaire, thereby increasing the sensitivity to detect differences that may be less evident when using fewer scale points. The data was analyzed using descriptive statistics for the quantitative survey items, while the qualitative data was analyzed using summative content analysis [22].

Ten professionals completed this evaluation. Their average age was 54, 8 years, and they had background experience in strategic management (8), emergency preparedness (4), consultancy (3), project management (3), evaluating research (3) and logistics (2).

Most of the participants in the online evaluation of the questionnaire, mentioned that the questionnaire was thorough, covering a lot of different dimensions related to the unforeseen, and that it was clearly focusing on the unforeseen. However, some formulations needed to be made clearer, and the fact that the questionnaire is exhaustive was also signalized as a possible disadvantage or a potential issue regarding its applicability.

Participants mentioned that such a questionnaire is useful to the industry, that it may increase the understanding of the unforeseen, and that it triggers sound reflections related to preparedness for the unforeseen.

The ten participants rated whether the checklist complied with the nine criteria derived from Stufflebeam [3] and Fernandes et al. [20]. The results are presented in Table 1. The mean rating for the nine criteria ranged from 8.00 to 8.70. By comparison, Martz [18] validated a more general organization-oriented questionnaire and Fernandes et al. [20] validated a questionnaire related to the design of control rooms. Martz [8] reported a mean range of 6.30 to 7.80, and Fernandes et al. [20] reported mean values ranging from 6.20 to 8.30 for the same criteria (the numbers have been adjusted for different rating scales).

Criterion Mean (M) (N=13) Standard Deviation (SD)
Applicability 8.10 1.19
Clarity 8.00 1.33
Comprehensiveness 8.00 1.24
Concreteness 8.20 1.31
Ease of use 8.30 1.70
Impartiality (Unbiased) 8.40 1.42
Pertinence (Appropriate) 8.70 1.16
Parsimony (Adequate amount of topics) 8.10 1.91
Usefulness 8.20 1.32
Total 8.22 1.40

Table 1: Evaluators’ rating of compliance with checklist criteria.

This means that the UN-ORG questionnaire was rated quite high. The standard deviations for UN-ORG show that evaluators varied in their opinions, particularly with regard to parsimony (1.91) and ease of use (1.70). With the exception of the ratings for these two criteria, all individual ratings were 6 or higher, and all mean ratings were 8.00 or higher.

Our interpretation of these ratings suggests that the evaluators perceived the questionnaire in general as relevant for assessing preparedness for the unforeseen.

The next phases of the development of the questionnaire included (9) Evaluate the questionnaire. Before applying a checklist to its primary intended use and especially before disseminating it for widespread application, it is important to subject it to field-testing. The evaluations provide assurances regarding the questionnaire’s quality and direction for improvement. Just after an emergency exercise, the project team had the opportunity to subject the UN-ORG questionnaire to such a field-test. An UN-CAF analysis had already been performed in the organization.

An analysis of Cronbach’s alpha of UN-ORG resulted in a value of . 90, which indicates a rather high degree of internal consistency between the items [23,24]. Internal consistency depicts to which degree the items of a measure capture the same construct, and is expressed as a number between 0 and 1. Cronbach´s alpha is affected by the number of items in the test. In our case, the number of items is quite high (around 100), and may be part of the explanation of the high score. However, a value of .90 is a strong indication that the items in the questionnaire measure the same construct, or idea, which in this case is “the unforeseen”.

The document analysis, UN-CAF, gave several recommendations for improvement related to emergency plans and the supplementary UN-ORG study supported the main findings in the UN-CAF analysis. A “spider diagram” was made from the answers to UN-ORG (Figure 2), and illustrates in which areas the organization is well prepared, and in which area they may improve.

arts-social-sciences-spider-diagram-study

Figure 2: Spider diagram of UN-ORG factors in a case study.

Some examples of recommendations from the UN-METH case study analysis are:

• Consider developing a better understanding of the unforeseen, and include this in the general preparedness plans.

• Acquire a higher consciousness and training related to identifying risk and escalation of an event.

• Establish a better understanding for concurrent learning, during an event or an exercise.

The case study indicated that the UN-ORG and UN-CAF methods could be used as a supplement to each other to consider whether there is an agreement between the analyzed documents and the opinion of individuals.

The three final phases in Stufflebeam [3], is (10) Finalize the questionnaire, (11) Apply and disseminate the questionnaire, and (12) Periodically review and revise the questionnaire. The questionnaire was finalized and developed into a test battery of 100 statements. This test battery constitutes UN-ORG and can be used in organizations to assess organizations’ preparedness for handling unforeseen events. Phase (11) and (12) will be performed continuously, as it is desirable to invite users to provide critical feedback, since questionnaire development is an on-going process. We would like to invite case descriptions and critiques, systematically file the information, and periodically review and use the information for improving the questionnaire, as recommended by Stufflebeam.

Discussion

The main purpose of this article was to document the development and evaluation process of a questionnaire (UN-ORG) that has been developed in order to assess an organization’s preparedness and ability to handle unforeseen events.

The findings in this article suggest that the Methodology for Handling the Unforeseen, is one effort that may be valuable for organizations in assessing their preparedness for the unforeseen. The evaluation of UN-ORG, included a case study, a survey evaluation, and two separate interviews with experts.

The evaluation process showed some confusion in the wording of the questions, which potentially may reduce the respondents' understanding of the content and nuances in some questions. During the development and evaluation process, some questions have been altered to clarify their meaning.

Mean values on Stufflebeams [3] eight test criteria and the additional criteria (usefulness) introduced by Fernandes et al. [20], ranged from 8.00 to 8.70, and is compared to Martz’s [18] and Fernandes’ et al. [20] studies. Martz [18] reports a mean range of 6.30 to 7.80, and Fernandes et al. [20] reported mean values ranging from 6.20 to 8.30 for the same criteria (the numbers have been adjusted for different rating scales).

The two items with lowest mean values in our study were clarity and comprehensiveness 8.00 (SD=1.33) and 8.00 (SD=1.24). The respondents also provided the most qualitative feedback related to these two aspects of the questionnaire, and indicated that as they not completely understand an item, or felt that several questions were similar to each other.

The questions within the 10 UN-categories are made much nuanced, and several items contain concepts aimed at expressing processes based on recent research related to the unforeseen [2]. An example of such an item is (under category 8: Training):

Our organization has specific training on “side-winder effects” related to unforeseen event (“side-winder effects”: additional, unforeseen events related to the first one).

This may have caused that some respondents, despite their experience and academic competence within the discipline, did not fully comprehend the nuances included in the questionnaire. Such feedback was useful information for us, and we therefore adjusted and reformulated questions, without losing the underlying meaning.

The results from our evaluation suggests that UN-ORG can contribute to assess how well prepared an organization is to meet unforeseen events. UN-ORG collects data that specify the strengths and weaknesses in relation to the 10 UN-categories. Answers to these questions can form specific improvements and training needs of the organization with regard to the unforeseen. Seen in combination with data from UN-CAF, it will be possible also to develop specific EPR plans covering the unforeseen as well as programs for training, appropriate for the organization at hand.

UN-METH is not a fixed measurement instrument. There is a dynamic included in the concept, constantly forcing adjustments to the organization's development and the changes in the society. This dynamic makes the tool relevant as a contribution to the organizations’ economical sustainability. To exploit the full potentials of UN-METH, time, competence and initiative among the organizations’ management and employees is needed.

As with all questionnaires, a number of years of field-testing and use are required to thoroughly validate and refine the tool. The approach used in this article for developing and evaluating the questionnaire was an attempt to establish a reasonable level of validity of UN-ORG. The next steps of this research will be to identify organizations willing to participate in a study where UN-METH can be used. We believe that this will be useful for the organizations themselves, in that they may find areas where they have an adequate emergency preparedness, and areas where they may need to improve. Further, such a study may help the authors to further develop the method in general and the questionnaire in particular. Through the development process, we have also identified a need for a more refined and concise version of UNORG. A short version is therefore under development. This can be used in pre-studies and in more general data collection studies, before a full-scale UN-METH and/or UN-ORG analysis is performed.

Conclusion

In the introduction, we asked the following research question: Is it possible to assess an organization’s preparedness, given that it does not know what it should be prepared for? By using UN-MET, investigating an organizations’ strategic preparedness plans through an UN-CAF analysis and performing a survey with individual respondents through UN-ORG, we believe that we are able to provide a relatively representative overview of the organizations’ ability to handle the unforeseen.

In this article, we have described UN-METH, and the process of developing and evaluating UN-ORG. UN-ORG was developed and evaluated using Stufflebeam’s [3] twelve phases. Interviews, a survey and a case study were performed to evaluate the questionnaire. The validation of a questionnaire plays an important role in establishing the credibility and utility of the questionnaire. The use of subject matter experts and targeted users combined with an actual case study provide valuable perspectives for developing, refining, and validating UN-ORG.

Overall, the participants in the studies were positive to the development of a tool such as the UN-ORG questionnaire. The concept of a questionnaire for supporting the preparedness for the unforeseen seems to be accepted, relevant and adequate, and to be an industry need as indicated by the professionals participating in this evaluation process. The participants referred that the main advantages of this questionnaire were that UN-ORG focus on the unforeseen and covers many dimensions. On the other hand, one of the main identified weaknesses was related to the clarity of the language. Based on the direct feedback from the evaluators, the technical language and the overall construction of the items have been altered to ensure that each item is clear, reducing the chances of misinterpretation.

As noted earlier, critical feedback based on repeated use of the questionnaire offers a preferred method for validation and improvement. The next steps of this research will be to identify organizations willing to participate in studies where UN-METH can be used. We believe that this will be useful for the organizations themselves, in that they may find areas where their organization has an adequate emergency preparedness, and areas where they may need to improve. Further, such studies may help the researchers to further develop UN-METH in general and UN-ORG in particular. A short version of the questionnaire is underway, and both questionnaires will be translated to English. UN-METH seems promising, and we believe it covers a need in the industry and can be used to help organizations calibrate their general emergency preparedness towards their ability to handle the unforeseen in emergency situations, and making them more prepared to meet the continuous threats they are exposed to.

References

Select your language of interest to view the total content in your interested language
Post your comment

Share This Article

Article Usage

  • Total views: 1411
  • [From(publication date):
    February-2017 - Aug 25, 2019]
  • Breakdown by view type
  • HTML page views : 1316
  • PDF downloads : 95
Top