Acknowledgements
This work is supported by the EPSRC(Engineering and Physical Research Council) programme grant EP/T028572/1: Visual AI project.
Welcome to the 2023 VoxCeleb Speaker Recognition Challenge! The goal of this challenge is to probe how well current methods can recognise speakers from speech obtained 'in the wild'. The data is obtained from YouTube videos of celebrity interviews, as well as news shows, talk shows, and debates - consisting of audio from both professionally edited videos as well as more casual conversational audio in which background noise, laughter, and other artefacts are observed in a range of recording environments.
| Development set for verification tracks released. | |
| Development set for diarisation tracks released. | |
| Test set released and evaluation server open. | |
| Deadline for submission of results. | |
| Deadline for technical report. | |
| Challenge workshop |
VoxSRC 2023 will feature four tracks. All four tracks are identical to previous year. Track 1, 2 and 3 are speaker verification tracks, where the task is to determine whether two samples of speech are from the same person. Track 4 is a speaker diarisation track, where the task is to break up multi-speaker audio into homogenous single speaker segments, effectively solving ‘who spoke when’.
# |
Description |
|---|---|
Track 1 |
Fully supervised speaker verification (closed train set)
|
Track 2 |
Fully supervised speaker verification (open train set)
|
Track 3 |
Semi-supervised domain adaptation (closed train set)
|
Track 4 |
Speaker diarisation (open train set)
|
Please read the Tracks section carefully and choose the data for training your own models. Below are some hyperlinks that may be helpful to you.
| VoxCeleb | CNCeleb | VoxMovies | VoxConverse |
We have included some of the utterances from VoxCeleb1, VoxMovies and VoxConverse dataset in validation set. You are allowed to use these datasets for training in Track 2. Validation data for Track 3 is same as last year.
For verification validation set, please download all three files using wget. Then, run zip -F VoxSRC2023_val.zip --out VoxSRC2023_val_total.zip, before unzipping VoxSRC2023_val_total.zip. Specifically, run the commands in your terminal:
wget https://www.robots.ox.ac.uk/~vgg/data/voxceleb/data_workshop_2023/VoxSRC2023_val.z01
wget https://www.robots.ox.ac.uk/~vgg/data/voxceleb/data_workshop_2023/VoxSRC2023_val.z02
wget https://www.robots.ox.ac.uk/~vgg/data/voxceleb/data_workshop_2023/VoxSRC2023_val.zip
zip -F VoxSRC2023_val.zip --out VoxSRC2023_val_total.zip
unzip VoxSRC2023_val_total.zip
| File | MD5 Checksum | |
| Track 1 & 2 validation wavfiles | See the instruction above. | |
| Track 1 & 2 validation trial pairs | Download | 8c3476802e14682f11ea356954ceab8e |
| Track 3 unsupervised target domain data | Download | b0157d5cb961ecb1f5f617625fb843a1 |
| Track 3 supervised target domain data | Download | 57170ba6c8c5223be0cefc6ab1b43e5f |
| Track 3 validation wavfiles | Download | 50fccc3315cf7b18d6575350d8fb043d |
| Track 3 validation trial pairs | Download | 97f71af121f620363f86070089adad02 |
For verification validation set, please download all four files using wget. Then, run zip -F VoxSRC2023_test.zip --out VoxSRC2023_test_total.zip, before unzipping VoxSRC2023_test_total.zip. Specifically, run the commands in your terminal:
wget https://www.robots.ox.ac.uk/~vgg/data/voxceleb/data_workshop_2023/VoxSRC2023_test.z01
wget https://www.robots.ox.ac.uk/~vgg/data/voxceleb/data_workshop_2023/VoxSRC2023_test.z02
wget https://www.robots.ox.ac.uk/~vgg/data/voxceleb/data_workshop_2023/VoxSRC2023_test.z03
wget https://www.robots.ox.ac.uk/~vgg/data/voxceleb/data_workshop_2023/VoxSRC2023_test.zip
zip -F VoxSRC2023_test.zip --out VoxSRC2023_test_total.zip
unzip VoxSRC2023_test_total.zip
| File | MD5 Checksum | |
| Track 1 & 2 test wavfiles | See the instruction above. | |
| Track 1 & 2 test trial pairs | Download | e42df439e75da53fcdfdf0821999ee36 |
| Track 3 test wavfiles | Download | 05b7bbba05bb80f94ed192b4833e6bda |
| Track 3 test trial pairs | Download | c055d52b3aa043fc1419462af13c55db |
| File | MD5 Checksum | |
| Track 4 test wavfiles | Download | 2c7b562df1eb3b52d39b57e9eb267890 |
For the Speaker Verification tracks, we will display both the Equal Error Rate (EER) and the Minimum Detection Cost (CDet). For tracks 1 and 2, the primary metric for the challenge will be the Detection Cost, and the final ranking of the leaderboard will be determined using this score alone. For track 3, the primary metric is EER, as this is a more forgiving metric.
For the Speaker Diarisation track, we will display both the Diarisation Error Rate (DER) and the Jaccard Error Rate (JER), but the leaderboard will be ranked using the Diarisation Error Rate (DER) only.
We use a collar of 0.25 seconds and include overlapping speech in the scoring. For more details, consult section 6.1 of the NIST RT-09 evaluation plan.
Four tracks will be held via Codalab platform. You need a Codalab account for registration, so please make it if you don't have one. Any researchers, whether in academia or industry, can participate in our challenge, but we only accept institutional emails to register. Please follow the instructions on each challenge website for submission.
CodaLab evaluation server are active now. Please visit the links below for participation.| Challenge | Links |
|---|---|
| VoxSRC-19 | challenge / workshop |
| VoxSRC-20 | challenge / workshop |
| VoxSRC-21 | challenge / workshop |
| VoxSRC-22 | challenge / workshop |
All teams are required to submit a brief technical report describing their method.
All reports must be a minimum of 1 page and a maximum of 4 pages excluding references.
You can combine descriptions for multiple tracks into one report.
Reports must be written in English.
See
here,
here and
here
for examples of reports.
Q. Who is allowed to participate? A. Any researcher, whether in academia or industry, is invited to participate in VoxSRC . We only request a valid official email address, associated with an institution for registration, once the registration system opens. This ensures we limit the number of submissions per team. Q: Do I need to use the name of my institution or my real name as the team name for a submission? A: No, you do not have to. The name of the CodaLab user (or the Team name, if you have set up one in CodaLab) that uploads the submission will be used in the public leaderboard. Hence if you do not want your details to be public, you should anonymise if appropriate. You must select a team name before the server's closing time. Q: For the semi-supervised track 3, can I use my model that I trained for the closed track 1? A: Yes. For track 3, participants are allowed to train on the VoxCeleb2 dev dataset. So participants can use their model that was trained for track 1. Q: For the semi-supervised track 3, can I use the CnCeleb validation set? A: No. For track 3, participants can only use the provided validation set. Q. Can I participate in only some tracks? A. Yes, you can participate in as many tracks as you like and be considered for each one independantly. Q: How many submissions can I make? A: You can only make 1 submission per day. In total, you can make only 10 submissions to the test set for each track. Q: Can I train on other external datasets (public, or not)? A: Only for the OPEN tracks. Not for the CLOSED tracks. Q: Can I use data augmentation? A: Yes, you can use any kind of noise or music, as long as you are not training on additional speech data, for the CLOSED tracks. You may also use the MUSAN noise dataset as augmentation for the CLOSED tracks. For the OPEN track, you can train on any data you see fit. Q. Can I participate in the challenge but not submit a report describing my method? A. We do not allow that option. Entries to the challenge will only be considered if a technical report is submitted on time. This should not affect later publications of your method if you restrict your report to 2 pages including references. You can still submit to the leaderboard, however, even if you do not submit a technical report. Q. Will the technical report submitted to this workshop be archived by Interspeech 2023? A. No. We shall use the papers to select some authors to present their work at the workshop. Q. Will there be prizes for the winners? A. Yes, there will be cash prizes for the top 3 on the leaderboard for each track. Q. For the CLOSED condition, can I use the validation set for training anything, eg. the PLDA parameters? A. No, for the CLOSED condition you can use the validation set only to tune user-defined hyperparameters, eg. for example selecting which convolutional model to use. Q. For the CLOSED conditions, what can I use as the validation set? A. For the closed conditions, participants may only use the provided pairs for this year's challenge, or the VoxCeleb1 pairs. These must strictly NOT be used for training. It is beneficial for participants to use this year's provided validation pairs, as their distribution matches that of the hidden test pairs. Q. What kind of supervision can I use when training without labels in the semi-supervised track? A. Self-supervision is an increasingly popular field of machine learning which does not use manually labelled training data for a particular task. The supervision for training instead comes from the data itself, for example from the future frames of a video or from another modality, such as faces. Q. For the semi-supervised track, when I am training on the large set of target domain data without labels, can I use the total number of speakers in the CnCeleb2 dev set as a hyperparameter? A. No, you cannot use any speaker identity information at all. You cannot use the number of speakers in any way, e.g. to determine the number of clusters for a clustering algorithm. Q. What if I have an additional question about the competition? A. If you are registered in the CodaLab competition, please post your question in the competition forum (rather than contact the organisers directly by e-mail) and we will answer it as soon as possible. The reason for this approach is that others may have similar questions: use of the forum ensures that the question can be useful for everyone. If you rather make your question before registering, please follow the procedure in the Organisers section below.
Jaesung Huh, VGG, University of Oxford
Jee-weon Jung, Carnegie Mellon University
Andrew Brown, Facebook AI Research
Arsha Nagrani, Google Research,
Joon Son Chung, KAIST, South Korea,
Andrew Zisserman, VGG, University of Oxford,
Daniel Garcia-Romero, AWS AI
Mitchell McLaren, Speech Technology and Research
Laboratory, SRI International, CA,
Douglas A Reynolds, Lincoln Laboratory, MIT.
Please contact jaesung[at]robots[dot]ox[dot]ac[dot]uk if you have any queries, or if you would be interested in sponsoring this challenge.