What makes a good model?

June 14, 2017.

by – Posted in Predictive Analytics

There are four main characteristics that can be used to determine the degree of how good a model is. These are

1. Accuracy

The accuracy measures how well the model predicts the outcome. When the predictions are close to the actual values (on some validation dataset), the model is deemed to be accurate. Some measures are more objective than others. Objective measures, such as the c-statistic, do not depend on the composition of the dataset used to fit the model. The value of non-objective measures, such as the misclassification rate, does in fact depend on the distribution of the values of the dependent variable in the training data.

2. Robustness

The robustness in predictive modeling can be defined as the requirement that the performance of the model – in terms of prediction accuracy – does not depend on the data with which this model is fitted. Robustness is usually checked by scoring some data for which the outcome is known and comparing the accuracy of the model predictions on both data sets, the fitting dataset and the validation dataset. When the performance is the same (or very close), the model is deemed robust. There are many variations on this scheme. A popular one is called K-fold validation. In this scheme, the validation is done K-times (K>1) using samples taken from the validation data and the results are combined in a way to reflect the overall performance. This way the effect of the choice of the validation dataset is minimized on the assertion of robustness. Robustness is usually at odds with accuracy. Achieving high accuracy during fitting the model often results in reduction in robustness. Certain models are more susceptible to this behavior than others.

3. Simplicity

Keeping the model simple is basically an attempt to comply with the Occam Razor – also known as the law of “briefness”. In a relevant version, the Occam Razor states that we should not use more than necessary to explain the problem. In our world of modeling, we should keep the model as simple as possible. A typical example of how simplicity can be so powerful is the equation that explains the universe: E=MC2. For years, statisticians promoted building parsimonious models, i.e. models that have the smallest number of parameters (the simplest models). Keeping the models simple usually impacts the accuracy. Therefore, a balance between accuracy, robustness and simplicity is always on the mind of the modeller.

4. Explainable

The requirement that the model be explainable may sound contradictory to the promise that models can discover hidden knowledge. But in fact, the majority of the model predictions or behaviour should be explainable and should make sense. There is a common myth that models will always produce surprising results. This will be the case when all the wisdom of the experts running the business is false. This should be a rare event. Therefore, for the model to make sense or can be explained is a necessary requirement to accept the model and use its predictions.

The above four requirements can be quantified using different levels of objectivity. In future articles, we will explore some of the measures used to assess some of these characteristics.

Leave a Reply

Your email address will not be published. Required fields are marked *