Evaluation Research

Dr. David Garson of North Carolina State University provides an informative website at ncsu.edu. One of the topics he included in Multivariate Analysis is the Evaluation Research. He started with an overview stating that, “Evaluation research seeks to provide objective assessments of past, present, or proposed programs of action”. Here is a summary of Dr. Garson’s article.

There are three basic types of question asked in evaluation research – descriptive, normative, and impact. Descriptive studies are used to describe the goals, objectives, start-up procedures, implementation processes, and anticipated outcomes of a program. Normative studies evaluate programs by multiple values while impact studies evaluate in terms of outcomes.

The four major types of evaluation designs are survey research, case study, field experiment, and secondary analysis of archival data. Both survey and case studies are primarily used in descriptive and normative questions. Field experiments, which are used primarily around impact questions, have three main types – true, nonequivalent comparison, and before-and-after studies. Secondary data analysis is useful for all three types of research questions.

Some research examples cited are pilot studies, cost-benefit analysis, performance reference model, program assessment rating tool, needs assessment, and feasibility studies. Pilot studies are “reality check” trial runs of a full blown research. Cost-benefit analysis deals with measuring costs or benefits. It may sound straight-forward and simple, but a lot of questions arise from this analysis. Benefits may be tangible or intangible. The return of investment is traditionally used for the cost-benefit analysis of tangible benefits. On the other hand, intangible benefits cannot be adequately measured. There is also performance reference model, which is a framework for evaluation developed by the Office of Management. It uses four types of evaluation measure – mission and business results, customer results, processes and activities, and technology.

Another approach to evaluation developed by the Office of Management is Program Assessment Rating Tool, which is designed to assess and improve program performance through a formal review process which was meant to identify a program's strengths and weaknesses.

Needs assessment serves to identify organizational problems (both internal and arising from the organization's environment), to investigate opportunities for change, and to document and defend projects which are proposed to address problems. Needs assessments assure that projects are directed to real problems faced by the organization rather than wasting scarce resources. Needs assessment process is implemented in four stages – collecting information, identifying and prioritizing problems researching alternative possible solutions, and seeking consensus on a proposed solution.

Lastly, there is feasibility study. It is normally undertaken only after the organization has identified and defined a proposed project, and has a written requirements analysis and a general design statement for it. It has three dimensions – operational, economic, and technical.


In general, the article provides substantial information on evaluation research especially for the business sector. It provides specific examples such as questions regarding measurement of benefits and questions regarding measurement of costs. Furthermore, it also compares and contrasts one type of evaluation research from the other. For example, aside from describing feasibility studies, it makes a distinction between feasibility studies from needs assessments.

There are technical business terms used such as cost-benefit, potential value, and present value that might hinder a non-business oriented person from understanding the text. Nevertheless, educators can relate to some of the evaluation research designs such as pilot studies, survey research, and field experiment. An educator, however, will find the article more helpful if the examples given are centered on education research.

Furthermore some organizations cited are not found locally; thus, their standards may not be applicable in our field. It would be beneficial if there is an academic site featuring evaluation research standards in the Philippines.

On the bright side, the article opens a new window for us to expand our knowledge on research. Dr. Garson has done a commendable website, which is informative not only in North Carolina State University but in the research communities worldwide.

Reference:
Garson, G. David (n.d.). "Evaluation Research", from Statnotes: Topics in
Multivariate Analysis. Retrieved 12/28/2008 from
http://www2.chass.ncsu.edu/garson/pa765/statnote.htm.

Making Numbers Make Sense

My research beliefs completely changed when I took up qualitative research under a professor who is quite passionate and well-versed with the subject. I was fascinated with how qualitative researchers collect and interpret data. The “in-depth” analysis of data somewhat made quantitative research less credible. Example, a quantitative researcher might say the result is not significant if there is only one rape case per 10,000; however, the qualitative researcher will consider that lone case and declare the results to be “significant”. This example cited by my teacher keeps reverberating in my mind. After the end of the semester, I developed my utmost respect to qualitative researchers and I stopped cultivating my passion for quantitative research. I even started to think that qualitative research is better.

How about marrying the two types of research? I bow my head to the one who first thought of this idea. This would definitely settle the issue regarding the differences of quantitative research and qualitative research. With the “marriage” of the two researches becoming a success, researchers should be keen enough to polish their skills in analysis of narrative and numerical data.

For those who are still groping in the dark on how to interpret numerical data the PowerPoint presentation Making Sense of Numbers by Dr. Jim Adams-Berger would definitely help. Those who have learned the ropes in data analysis could use the presentation to recall the basic considerations on interpreting “numbers”.
The presentation encompasses using data to identify community problems, to explore problem relationships and to choose responsive programming. As a whole, Making Sense of Numbers is anchored on the framework of research (perhaps a research program particularly designed for OMNI Research and Training, Inc.). Berger points out that values serve as a filter of the other major analytic elements, which are beliefs, data and research. There is also a detailed explanation on modeling problem relationships. The use of graphic models makes the presentation more substantial and interesting especially for those who have just started doing research.

A wide range of examples could have been presented in surfacing and identifying problems; however, the presentation focused on the differences of numbers and rates in a particular case. In a certain example, the numbers of aggravated assaults for 4 consecutive years are not enough for us to conclude what year has the worst record. Calculating the rate is more meaningful. The recommended procedure in calculating the rate is number of cases divided by the population and multiplied by a large number (“per” number). That would be acceptable if the number to be multiplied is smaller than the population. What if it is larger than the population? One murder case in a population of 500 may be interpreted as 20 murder cases per 10,000. Is it not misleading? I think it would be better to stick to the usual per 100 in solving rates.

It is also best to consider different viewpoints before making conclusions from a set of data. There are instances when rates are favored than numbers, but there are cases when rates give misleading information.
In the example in calculating rates, the data in year 1997 shows population of 10,000 and number of aggravated assaults of 70. However, the rate per 10,000 is 7. Let us take a moment to ponder if it is correct. Perhaps, a calculator could help.
Aside from faulty calculations, another pitfall of “numbers” is the method of gathering data. There are a lot to consider – from the structure of questions to the respondents’ willingness to answer. That is, we have to ensure validity and reliability.

I consider myself a neophyte in the research arena. I realize there are still a lot to learn. Starting with Making Numbers Make Sense I hope to fill my brain with the skills in numerical data analysis that are useful in research.