> Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly intelligent behavior. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for.  Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.

Lapuschkin, et al. Unmasking Clever Hans predictors and assessing what machines really learn.
Received 06 December 2017.  Published 11 March 2019

* comment: Note the lag time between submission and publication. Actually,p eople have been raising red flags about the validity of machine intelligence results for decades, but folks tend to turn a blind eye.

> These abstract machines are mathematical fictions rather than physical objects. By definition they are incapable of errors of functioning. In this sense we can truly say that "machines can never make mistakes." Errors of conclusion can only arise when some meaning is attached to the output signals from the machine. The machine might, for instance, type out mathematical equations, or sentences in English. When a false proposition is typed we say that the machine has committed an error of conclusion. There is clearly no reason at all for saying that a machine cannot make this kind of mistake. For example, it might have some method for drawing conclusions by scientific induction. We must expect such a method to lead occasionally to erroneous results.

Turing. Computing machinery and intelligence. October, 1950

PLB

thinking, Can one that makes the error, detect the error? My interactions with machine intelligence reveal that when taken to task over outright errors—it tends to apologize, and let it go at that.

             ***********************************************
The BEE-L mailing list is powered by L-Soft's renowned
LISTSERV(R) list management software.  For more information, go to:
http://www.lsoft.com/LISTSERV-powered.html