Expert evaluation: features, methods and results

Expert evaluation is the name of a whole system of diagnostic methods that are extremely widely used in management, economic analysis, psychology, marketing and other fields. These methods make it possible to characterize, classify, assign a certain rank or assessment to events and concepts that can not be quantified.

In what cases is an expert evaluation necessary?

During any research at any of its stages, an expert evaluation method can be applied . In management activities, it can come in handy:

• At the stage of defining the goals and objectives of the research process.
• During the construction or testing of the hypothesis.
• To clarify the problem situation. To interpret the processes and events that occur.
• To justify the adequacy of the tools used.
• To generate recommendations, as well as to implement many other goals.

The expert evaluation is justified in those cases when it is impossible to make a decision based on accurate calculations (for drawing up a psychological portrait, working characteristics, estimating economic uncertainty and risks).

Most often, the application of such estimates becomes important in the situation of choosing one or more variants from the proposed set:

• The launch of a production version of one of the developed products.
• Selection of astronauts from numerous contenders.
• Determination of the project of scientific work, which will be financed.
• The choice of an enterprise that will receive an environmental loan.
• Determination of the investment project for investment of financial resources.

Who are the experts and how do they work?

As follows from the name of the method, peer review involves the involvement of one or more expert experts competent to make assessments of individuals, as well as the processing of their opinions. The selection of experts is carried out taking into account the adequacy of their judgments and experience in this field.

Expert evaluation can be expressed both quantitatively and qualitatively. The data of expert studies are needed by managers, managers and employees of the management team as the basis for decision-making.

The development of an expert evaluation is most often done by creating a working group that organizes the activities of an expert (or several experts). If you have to use more than one person, they are united in an expert commission.

How many experts will it take?

Depending on the nature of the assignment and the capabilities of the enterprise, one or more experts may be invited to conduct the peer review. At the same time, expert evaluation is called individual or collective.

Individual is the assessment, which the teacher characterizes the depth of the student's knowledge. Also to this type carry the diagnosis, put by one doctor. However, in disputable or difficult situations (serious illness, stating the issue of expelling a student), they resort to a collective solution of the problem. Here we need symposiums of doctors and the organization of a commission of teachers.

The same algorithm works in the army: most often the decision is made by the commander alone, but if necessary, the military council is assembled.

Sequence of the evaluation procedure

The sequence of formation of actual and objective peer review is such stages:

1. Carry out an analysis of the situation that needs to be investigated.
2. Selection of experts for the procedure.
3. Examining the existing methods by which expert judgment will be measured.
4. Conducting the evaluation procedure itself.
5. Reduction and analysis of information obtained during the evaluation.

It may be necessary to perform an input validation, on which the expert evaluation will be based. In some cases, the working group has to change the composition of the expert group or resort to re-measuring the same questions (with a view to comparing the resulting evaluation with objective data from other sources).

Evaluation process: characteristics of stages

The competent decision of organizational questions is of great importance for successful carrying out of procedure:

• Planning costs for the event (payment of experts and specialists in analyzing the data received, costs for renting a room, buying office supplies).
• Preparation of necessary materials (compilation and printing of forms, provision of inventory).
• Choosing and instructing the moderator of the event.

In the process of work, experts should be guided by the specified rules, as additional time for decision-making does not affect its accuracy.

When the answers of all specialists are received, the expert opinion is evaluated . This takes into account the degree of coherence of all opinions. If there is no clear agreement, the working group should find out the reason for the disagreements, fix the formation of several groups of opinions and the lack of coherence as a result of expert evaluation. Then the study error is evaluated and the model is constructed based on the data that was obtained. This is necessary so that you can later carry out an analytical examination.

Methods used to conduct an individual peer review: what is an interview

Among the most effective and often used techniques are:

• Analytical method.
• Method of writing a script.
• Interview.

In accordance with the methodology of the interview, the forecaster talks with the expert, asking him questions. The subject of the conversation is the prospects for the development of the object or phenomenon in question. The program of the questionnaire is developed in advance.

The effectiveness and quality of the expert evaluation directly depend on whether the expert will be able to provide an opinion in a limited time.

Examination by analytical method

When choosing an analytical method for the assessment, the expert expert should prepare for the careful execution of independent work. He has to analyze trends, assess the state and possible ways of development of the object, in relation to which prediction is applied.

The system of expert assessments provides for the study of all information about the object that is available to the expert. The result is made out as a memorandum.

The main advantage of the analytical method is that a specialist can show all his individual abilities. True, this method is not suitable for analyzing large and complex systems, since the expert may not get knowledge from adjacent areas.

Execution of examination by writing scripts

Strictly speaking, this method should not be assigned only to the category of individual methods of assessment, since it is successfully applied to work in a group.

To use this method, the expert should determine the logic of the studied processes and phenomena with respect to time and different combinations of conditions. Then he can establish the supposed sequence of events (their development, the transition from the situation at the moment to the predicted state). The scenario reflects all stages of the solution of the problem, and also provides for possible obstacles.

Collective examination: the method of "brainstorming"

To evaluate complex, large-scale, multilevel systems, one can not do without attracting several expert experts.

They can perform the assigned task by applying one of the following methods:

• Collective generation of ideas ("brainstorm").
• Method "635".
• The Delphi Method.
• Evaluation of commissions.

Thanks to collective efforts and a special organization, experts can effectively conduct the most complex procedures, such as expert risk assessment for an investment project or forecasting the performance of various systems.

"Brainstorming" allows to fully disclose the creative data of experts. At the first stage, experts actively generate ideas, then apply destructuring (subject them to criticism, destroy), put forward the counterdies and develop a coordinated point of view.

The main condition is the lack of criticism at first and the statement of all spontaneously emerging ideas.

Specificity of the "635" method

This method was named because of the reception that experts use when using it: each of the six experts writes on a sheet of paper three spontaneously emerging ideas for a period equal to five minutes.

Next, the sheet moves to the next participant. The duration of the procedure is half an hour. Thus, 108 sentences will be recorded.

What is the peculiarity of the Delphi method

The purpose of developing this method of peer review was the need for a more rigorous and justified procedure that could provide an objective and maximally useful result.

It is used by experts invited to scientific and technical institutions, investment and insurance companies, as well as in a number of other cases.

The essence of the method is that they conduct multitask individual surveys (often with the help of questionnaires). Then, a computer analysis of expert assessments is conducted to form a collective opinion. In doing so, they identify and generalize the arguments to protect each judgment.

At the next stage, the results are passed to experts for adjustments. Their disagreement with collective judgment must be in writing justified. As a result of the multiple return of the assessment to the adjustment, the working group achieves a narrowing of the range and the development of an agreed judgment concerning the prospects for the development of the object under study.

Than the method is good:

1. Experts involved in the evaluation do not know each other and do not communicate. Thus, their interaction is excluded.
2. The results of previous rounds are also of interest and value to the working group.
3. It is possible to obtain a statistical characteristic of the group opinion.

Despite the relatively high cost and duration, this method becomes the best way to predetermine the development of long-term situations of a problem nature.

Often, the evaluation is carried out by a specially organized commission (method of commissions), which at the "round table" consider all aspects of the problem and make an agreed decision. The disadvantage is the influence of the participants on each other and the distortion of the results. An example is the expert evaluation of the activities of teachers and doctors.

Other methods

Above, the most common methods of performing the examination were listed, but in the practice of industrial, scientific and research organizations, others are also used.

Depending on the specifics of the situation to be forecasted, as well as on the resources and capabilities of the enterprise, the following can be applied:

• Business game. It allows you to model the necessary number of situations to study the features of the control system or other processes.
• "Court" is a staged trial, in which some experts defend decisions, others try to refute them.
• Method of report - after the analysis, the expert expresses his opinion in the form of an analytical note or a report. This is relevant if you need to carry out relatively simple work (for example, expert evaluation of the car for insurance, taxation or damage compensation).

As a result, it can be noted that the existence of a large number of methods and methods for conducting peer review allows the enterprise manager and working group to choose the most effective option for solving a particular problem.