Authors: Jieting Wang, Yuhua Qian, Feijiang Li

Abstract:

Human beings may make random guesses in decision-making. Occasionally, their guesses  may generate consistency with the real situation. This kind of consistency is termed random consistency. In the area of machine leaning, the randomness is unavoidable and ubiquitous in learning algorithms. However, the accuracy (A), which is a fundamental performance measure for machine learning, does not recognize the random consistency. This  causes that the classifers learnt by A contain the random consistency. The random consistency may cause an unreliable evaluation and harm the generalization performance. To  solve this problem, the pure accuracy (PA) is defned to eliminate the random consistency  from the A. In this paper, we mainly study the necessity, learning consistency and leaning  method of the PA. We show that the PA is insensitive to the class distribution of classifer  and is more fair to the majority and the minority than A. Subsequently, some novel generalization bounds on the PA and A are given. Furthermore, we show that the PA is Bayesrisk consistent in fnite and infnite hypothesis space. We design a plug-in rule that maximizes the PA, and the experiments on twenty benchmark data sets demonstrate that the  proposed method performs statistically better than the kernel logistic regression in terms of  PA and comparable performance in terms of A. Compared with the other plug-in rules, the  proposed method obtains much better performance.

Keywords： Random consistency; Accuracy; Pure accuracy; Bayes-risk consistent

Learning with mitigating random consistency from the accuracy measure.pdf

查看详情

Fri Dec 25 15:50:00 CST 2020