Combining the facial recognition decisions of humans and computers can prevent costly mistakes database builder

After a series of bank robberies that took place in the US in 2014, police arrested Steve Talley. He was beaten during the arrest and held in maximum security detention for almost two months. His estranged ex-wife identified him as the robber in CCTV footage and an FBI facial examiner later backed up her claims.

It turned out Talley was not the perpetrator. Unfortunately, his arrest left him with extensive injuries, and led to him losing his job and a period of homelessness. Talley has now become an example of what can go wrong with facial identification.

My research focuses on how to improve the accuracy of these decisions. This can make society safer by protecting against terrorism, organised crime and identity fraud. And make them fairer by ensuring that errors in these decisions do not lead to people being wrongly accused of crimes.


To understand just how challenging this task can be, try it for your self: are the images below of the same person or different people? Same or different person? The correct answer is provided at the end of this article. Humans versus machines

We recruited two groups of professional facial identification experts. One group were international experts that produce forensic analysis reports for court (Examiners). Another group were face identification specialists that made quicker decisions, for example when reviewing the validity of visa applications or in forensic investigation (Reviewers). We also recruited a group of “ super-recognisers” who have a natural ability to identify faces, similar to groups that have been deployed as face identification specialists in the London Metropolitan Police.

Interestingly, the super-recognisers also performed extremely well, with three out of 12 attaining the maximum possible score. These people had no specialist training or experience in performing face identification decisions, suggesting that selecting people based on natural ability is also a promising solution.

Performance of the algorithms is shown by the red dots on the right of the graph. We tested three iterations of the same algorithm as the algorithm was improved over the last two years. There is a clear improvement of this algorithm with each iteration, demonstrating the major advances that Deep Convolutional Neural Network technology have made over the past few years.

Our study provides a solution to this problem. By averaging the responses of groups of humans, using what is known as a “ wisdom of crowds” approach, we were able to attain near-perfect levels of accuracy. Group performance was also more predictable than individual accuracy.

Importantly, this application of face recognition technology is not automatic – like automated border control systems are. Rather, the technology generates “candidate lists” like the one shown below. For the systems to be of any use, humans must review these candidate lists to decide if the target identity is present. A ‘candidate list’ returned by face recognition software performing a database search. Humans must adjudicate the output of these systems by deciding whether the person in the ‘probe’ image – the image at the top – is pictured in the array below, and if so to select the matching face. The correct answer is provided at the end of this article.

banner