Comparing Model Results

 

Selecting Tools -> Compare Analyses from the Menu bar will allow the comparison of various analyses of the same data.

 

The user is prompted to select any number of Cognalysis Multirate analysis files (.cgmr extension) to include in the comparison.

 

High level statistics are summarized for each analysis in a table:

Significance

Relative Net Lift

Root Mean Squared Error (RMSE)

Average Absolute Error

Fuzzy K

 

The analysis with the highest ratio of Relative Net Lift to Average Absolute Error (or the "best fit") is shown in bold.

 

The analysis with the lowest Fuzzy K within the probability threshold of best fit will be colored yellow. The probability threshold selected determines at what level of probability from a Kolmogorov-Smirnov Test (K-S Test) will two analyses be considered to be the same. In the example below, CG_2013_02_12_1 is the "best fit". However, the K-S Test determined that the observed difference between CG_2013_02_12_1 and CG_2013_02_12_2 would occur randomly with a probability higher than the 10% probability threshold. Because of this, the two models are considered to be essentially the same, so the one with the lowest Fuzzy K value is colored yellow. The default probability threshold is 0.1, but another threshold can be selected in the box at the top of the window.

 

 

Analyses will be considered suboptimal if there is another analysis with all of the following:

Higher Significance

Lower RMSE

Lower Fuzzy K

 

If an analysis is suboptimal its row in the table will be colored gray, as demonstrated in 5 of the analyses below (please note that there are more analyses being compared than are shown in the image):

 

To be able to compare RMSE between the analyses, it is necessary that they are calculated at the same level of detail, typically forced by the selection of a Detail Key, and must be applied to the same data. Analyses for which this is not true will also be identified as gray rows, and additional columns for Cell Count, Exposure, and Target will be added to the table to illustrate the discrepancy:

 

 

Double clicking on two analysis names will force a direct comparison of those two analyses. There are several comparison tools to evaluate the differences:

Top right - Difference in structure between the two models.

Bottom right - The top 100 factor differences between the two models.

Bottom left - Detailed comparison of the statistics of the two models, both with and without the Curve Modification and for both Training and Test data. Below this box is the result of a Kolmogorov-Smirnov Test on the errors of the test data, which gives the probability that the observed difference between the two models would occur randomly.

 

At any time the two analyses being compared may be changed by double-clicking the desired analysis name header.

 

The results of the compare analyses tool may be printed using the print button. The amount of detail printed will depend on the amount of detail currently displayed in the tool.