PRE-TEST POST-TEST Analysis

The acquisition of knowledge is one  of the  most important components of training. To test the  effectiveness  of  a  competency-based  training program this component should be evaluated.

Pre-Test Post Test

Tests administered upon an agreed upon “entry  point”  and  “exit point.” These tests can be standardized or locally-developed.

Pre-test is a preliminary test administered to determine a student’s baseline knowledge or preparedness for an educational experience, course or qualification.

Posttest is a test given to trainees after completion of the course or qualification and used in conjunction with a pre-test to measure their achivement and the effectiveness of the training program.

Pre-Test/Post test is not limited to written test. Performance test can also be used. But for our purpose, we will use the written test as a measure of knowledge.

To serve our purpose, we shall be using a trainer-made test that is item-analyzed. True or False and Multiple Choice Type of written test is therefore most appropriate.

Advantages:

• Useful method for measuring the “value-added” by a program of study
• The “after-only” design of documenting learning is a weak approach because positive change cannot necessarily be attributed to the effectiveness of a program.

• Pre-tests serve several purposes: knowledge  of  the  current  status of a group may provide guidance for  future  activities as well as the basis of comparison for a post-test results; administering a test of entry behavior can determine whether assumed prerequisites have been achieved.

Disadvantages:

• Hard to discern if the positive change charted in a pre-post test is due to learning in the workshop or simply natural maturation.
  • Due to trainee’s dropping out, the post-test results may be higher because those who remain are more successful or persistent.
  • Problems with statistics: if the control group scored so low that they can only go up, or the control group that  scored  so  high little improvement will be indicated in the post-test scores.
  • If using the same test for both the pre- and post-test, some argue that trainees will absorb knowledge just from taking the test and will attend more readily to the content. To avoid this, we shall be using other evaluation instruments to back-up our analysis.
  • Concentrates on value-added rather than outcomes assessment.
  • Tendency to teach to the post-test

Statistical Analysis

The statistical tool which is most appropriate to analyze data  on pre-test and post test shall be paired t-test.

T-test is an inferential test that determines if there is a significant difference between the means of two data sets.

t-test (For Paired Samples) – Use this test to compare two small sets of quantitative data when data in each sample set are related in a special way.

Criteria

• The number of points in each data set must be the same, and they must be organized in pairs, in which there is a definite relationship between each pair of data points
• If the data were taken as random samples, you must use the independent test even if the number of data points in each set is the same

• Even if data are related in pairs, sometimes the paired t is still inappropriate
• Here’s a simple rule to determine if the paired t must not be used – if a given data point in group one could be paired with any data point in group two, you cannot use a paired t test

Since statistical software are now ready available,  we  shall  be  using the computer to compute for the p-value which shall  be  basis  of  our  analysis.

Since majority of the trainers are using Microsoft Office Applications,  we shall be using Microsoft Excel Analysis Toolpak to analyze our data.

1. Click the Microsoft Office Button                ,

2. then click Excel Options.

Note: These steps are for a Microsoft Office 2007 version, if you are using another version, please consult your trainer.

3. Click Add-ins, and then in the Manage box, select Excel Add-ins.

4. Click Go.

5. In the Add-Ins available box,  select the Analysis ToolPak check box, and then click OK.

Tip  If  Analysis  ToolPak   is  not  listed  in the Add-Ins available box, click Browse to locate it.

6. If you are prompted that the Analysis ToolPak is not currently installed on your computer, click Yes to install it.

After installing Analysis Toolpak you will see the data analysis icon on your Data Toolbar

The Statistical Table

The analysis table of the pre-test and posttest using paired t-test will look like the table below:

t-Test: Paired Two Sample for Means

If your statistic is higher than the critical value from the table:

• Your finding is significant.
• You reject the null hypothesis.
• The probability is small that the difference or relationship happened by chance, and p is less than the critical alpha level (p < alpha ).

If your statistic is lower than the critical value from the table:

• Your finding is not significant.
• You fail to reject the null hypothesis.
• The probability is high that the difference or relationship happened by chance, and p is greater than the critical alpha level (p > alpha ).

In our example above, the average for post test is equal to 60.37 and 53.74 for pre-test. The t-statistic is equal to 4.24 and t-crical value = 2.55. Since t-statistic is greater than t-critical value, we reject the null hypothesis.

In this case the null hypothesis is “the are no significant differences between pre-test and posttest scores” and the alternate hypothesis  is  “there are significant differences between pre-test and post test scores”. Stated in layman’s language the  null hypothesis means that pre-test score  is equal to the posttest scores or the  scores did not increase. The  alternate  hypothesis  on the other hand, means that pre-test scores are not equal to the post test scores or the scores increased.

Analyzing the result of the above example, since the null hypothesis is rejected, we accept the alternate hypothesis which is “there are significant differences between the pre-test and post test scores”. Conclusion is  the  scores increased.

Instead of comparing the t-critical and t-statistical values to determine significant difference, you may also compare the alpha level and p-values. In our example, because the p-value is less than the alpha level, the alternate hypothesis is accepted. However, if the p-value was greater than the alpha level, p>α, the null hypothesis would be retained. If your alpha is .01 and your p-value = 0.00025 there are significant differences between treatment means at .01 level of significance.

An alpha level represents the number of times out of 100 you are willing to be incorrect if you reject the null hypothesis. If you choose an alpha level of 0.05, 5 times out of 100 you will be incorrect if you reject the null hypothesis. Those five times, both means would equal, but that’s about it. 95 times out of 100, you will be correct because it is more likely that the means are not equal.

Sample Analysis

Presented below is a hypothetical data of the pretest and  posttest  scores which will be the basis of our  discussions  and example  for analyzing the results of a pretest and post test.

Student	pretest score	posttest score
1	23	27
2	23	28
3	15	20
4	24	30
5	26	32
6	22	24
7	20	25
8	28	26
9	30	35
10	27	27
11	25	30
12	28	33
13	32	32
14	30	35
15	33	33
16	27	27
17	25	25
18	20	30
19	30	35
20	31	27
21	22	40
22	34	40
23	26	30
24	31	34
25	29	34

Graph of the Pretest and Post Test Scores

From the pretest and post test scores above, the graph is generated. The blue line represents the pretest scores of the trainees and the red line represents posttest scores. Generally, the red line is higher that the blue  line which means that post test scores are generally higher than the pretest scores although some students did not show an increase in score like in the case of trainee 15, 16 and 17.

To establish that there is an increase in scores statistically, we  test the null hypothesis “there are no significant differences between the pre and post test scores” using paired t-test.

Note: An average, say 25 and 26 may obviously be different but may not differ statistically. There is a need to statistically analyze data, to test whether test scores are statistically different.

Microsoft Excel provides statistical tools to do this

Using Excel to analyze data

1. Type your data in excel

To perform paired t-test, select Data

1. Click on the icon Data Analysis (the dialog box on Analysis Tools will appear)

2. Click t-Test: paired Two sample for Means Then click OK

3. Click on the red arrow beside the blank area for Variable 1 Range, select the range of scores for posttest,   then   click   on the arrow again.

4. Click on the red arrow beside the blank area for Variable 2 Range, select the range for pretest scores, then click  on the red arrow to go back to the dialog box.

5. On the blank area for Hypothesized Mean Difference type 0.

6. Select the output range( or the range where the table will appear)

7. Then click OK

Note: In case you cannot do this on your own ask the assistance of your trainer. A sample analysis is provided in your CD.