A Review of the Benefits and Issues of Speaker Verification Evaluation Campaigns
| Type of publication: | Inproceedings |
| Citation: | elha08:elra |
| Booktitle: | Proceedings of the ELRA Workshop on Evaluation at LREC 08, Marrakech, Morocco |
| Year: | 2008 |
| Pages: | 29--34 |
| Note: | http://www.lrec-conf.org/proceedings/lrec2008/ |
| URL: | http://www.hennebert.org/downl... |
| Abstract: | Evaluating speaker verification algorithms on relevant speech corpora is a key issue for measuring the progress and discovering the remaining difficulties of speaker verification systems. A common evaluation framework is also a key point when comparing systems produced by different labs. The speech group of the National Institute of Standards and Technology (NIST) has been organizing evaluations of text-independent telephony speaker verification technologies since 1997, with an increasing success and number of participants over the years. These NIST evaluations have been recognized by the speaker verification scientific community as a key factor for the improvement of the algorithms over the last decade. However, these evaluations measure exclusively the effectiveness in term of performance of the systems, assuming some conditions of use that are sometimes far away from any real-life commercial context for telephony applications. Other important aspects of speaker verification systems are also ignored by such evaluations, such as the efficiency, the usability and the robustness of the systems against impostor attacks. In this paper we present a review of the current NIST speaker verification evaluation methods, trying to put objectively into evidence their current benefits and limitations. We also propose some concrete solutions for going beyond these limitations. |
| Keywords: | Benchmarking, Biometrics, Speaker Verification |
| Authors | |
| Added by: | [] |
| Total mark: | 0 |
|
Attachments
|
|
|
Notes
|
|
|
|
|
|
Topics
|
|
|
|
|