Assessing Software Cost Estimation Models: criteria for accuracy, consistency and regression

Abstract

One of the problems in software cost estimation is how to evaluate estimation models. Estimation models are usually evaluated against two attributes: estimation accuracy and estimation consistency. A number of measures are reported in the literature, but they have shortcomings. There is no generally accepted standard to evaluate estimation models and the existing measures sometimes are not consistent among themselves. This paper examines existing measures of estimation accuracy and consistency and proposes two new ones: the weighted mean of quartiles of relative errors (WMQ) as a measure of accuracy and the standard deviation of the ratios of the estimate to actual observation (SDR) as a measure of consistency. Besides, a new regression criterion is proposed to determine model parameters. The proposed measures and criterion were tested with a data set from real world software projects. Results obtained show that these new measures and criterion overcome many of the difficulties of the existing ones.