Technical Manual

---

• What is Speaker Verification?
• Why this topic?
• Chosen Implementation languages
• System Design
  • System Architecture
  • Word Model Design
  • Acceptance Criteria
  • Code Structure
• Problems Encountered
• Future Extensions to the Project

What is Speaker Verification?

Speaker Verification is the process of verifying the claimed identify of a registered speaker, using his/her voice characteristics. Therefore, users of a Speaker Verification system must first record a (large) set of recorded speech, upon which statistical models (Hidden Markov Model's in our case) are built to represent the speaker's voice.

There are four possible strategies for the implementation of a Speaker Verification system:

Fixed Phase Verification
This is little more than a comparison between recorded training data and a test sample. Although many DTW (Dynamic Time Warping) systems use fixed phrase verification, at a low level, it is nothing more than wav-on-wav comparison.

Fixed Vocabulary Verification
This technique makes verification more flexible and practical. All training and testing data is based on a fixed vocabulary (a grammar). In our case, as seems to be quite common, we have used a grammar that contains digits. With fixed vocabulary verification, models are built based on complete grammar components, this means that word models are built up for each word in the grammar for each speaker. Once these are in place, testing can be carried out on any sequence of grammar components.

Flexible Vocabulary Verification
If a flexible vocabulary system is implemented training and testing can be carried out on a wider range of samples. Sub-word models are built; therefore, any word or phrase can be modeled. For testing, the speaker would be presented with a certain string, which (when recorded) would have various sub-words and lexicons extracted and compared to the models.

Text Independent Verification
If a text independent system is implemented training and testing can be carried out on any sample. This is because samples are broken down into their phonetic components (by comparing the transcripts to a phone-level dictionary, such as "beep"). The Hidden Markov Models are then built up on these phones rather than the complete words. Speed becomes an issue with this method of Speaker Verification, as additional sample labeling and phone-level dictionary querying is required. So testing takes slightly longer, but training takes considerably longer.


As our project will be implemented across a network, we have chosen to implement Speaker Verification using a Fixed Vocabulary. Our grammar consists of digits ONE through NINE and DOE, RAY, ME, FA, SO, LA, TEE.



Why this topic?
Human Computer Interaction and especially a spoken interface to computers are seen as major milestones in the development of computers. So while there are products such as ViaVoice and XVoice that capture and interpret spoken words from an audio user input into on-screen text, there are no main-stream (off-the-shelf) products that do Speaker Verification. Our project will implement Speaker Verification as a security method.
Chosen Implementation languages
• HTK
• Java
• Perl
HTK(HTK Home page)
HTK is the main tool that we use in this Project. HTK stands for Hidden Markov Model Toolkit, which is a freely available toolkit for building and manipulating hidden Markov models. HTK is primarily used for speech recognition research, but when used in specialised ways, it can be used as the backbone to a Speaker Verification system.
HTK consists of a set of library modules and tools available in C source form. These library modules are designed for UNIX systems, therefore in the early days of the project, we had to build HTK under the Windows NT environment, which proved quite challenging. Once built though, we had a comprehensive set of tools which provided sophisticated facilities for speech analysis, HMM training, testing and results analysis. Our approach to the project was to implement HMM's (*Link to HMM explanation*) in our system design and so HTK contained the perfect set of tools.
Microsoft bought HTK (Entropic) in 1999, but licensed it back to Cambridge University Engineering Department (where it was developed), where it is now available as an open-source set of Speech tools. Microsoft retains copyright to the existing HTK code, but modifications and improvements to the code are welcomed.
The open-source license allows products to be built using HTK, but not redistributed (i.e. you can't ship shrink-wrap boxes with products that contain HTK code.)
In our project we used the following HTK tools:

HParse	The HParse program generates word level lattice files (for use with HVite), from a grammar file.
HLEd	This program is a simple editor for manipulating label files. We use it to make word (or phone) level transcriptions of training data.
HCopy	This program is used for generation of MFCC (Mel Frequency Cepstral Coefficients), which are required by HTK for model building, from recorded WAV files.
HCompV	This Program calculates the global mean and covariance of a set of training data. We use this tool to create initial (HMM)values from the training data.
HERest	This program is used to perform a single re-estimation of the parameters of a set of HMMs using an embedded training version of the Baum-Welch algorithm.
HHEd	This tools is used in manipulating sets of HMM definitions. We used it to generate new states from combinations of existing ones. Especially silence models.
HVite	HVite is a general-purpose Viterbi word recogniser. We use HVite to force-align testing data, and to calculate log-likelihoods when testing.
HResults	HResults is the HTK performance analysis tool. It can be used for result statistics, such as word recognised percentages.



JAVA
We chose Java for a number of reasons. We wanted to implement a simple Client-Server protocol in the project. Java allows easy manipulation of sockets and so was ideal for this.
We also required a GUI on the client side. We needed one that allows the creation of new users and allows current users to log on. The Java Swing libraries give the capability of creating such GUI's.
The third major factor in our decision to use Java was our requirement to record WAV files. We were able to use a Java audio package called 'SoundBite'. This package provides a SoundBite Class which can be used to record audio files, and also set sampling rate.
Java also lets you run System calls. This is required to run the perl scripts needed to manipulate the data for HTK. All the HTK functionality we needed is done from the command line and so this was necessary for us.

Perl
Perl was used in this project to glue the whole system together. Building and comparing speaker models involved many HTK commands that could all be grouped together using Perl scripts. There is also a lot of string manipulation involved when using HTK and Perl is ideal for dealing with strings of data. Parts of the the model creation involved copying sections out of some files and sticking into other parts of other files. Perl is very easy to use to do this.

System Design

We got some valued assistance from our supervisor Dr. John McKenna. He advised us on many issues, but we especially appreciated his assistance in the areas of building word models and deriving threshold values.


System Architecture

Client Architecture
Server Architecture

Building the word model
We decided on using a word model based on a fixed grammar. One of the reasons for this is that for accuracy, the more test data supplied, the more accurate the model. We restricted our model to use a fixed amount of words so that each model would be well trained for each word in the system. We decided on using the numbers 'ONE - NINE' and 'DOE RAY ME FA SO LA TEE'. This meant that we had sixteen different words in our model. Whenever a new user is to be added to our system, they must utter each word at least ten times during training, which is considered a good amount of training. An alternative would have been to build phone models. There are 42 Phonemes in the english language. However when using phonemes, more training is required as combination of phones must also be taken into account. This is not good if you want to get people to use speaker verification as training is often found quite tedious by users.

In building a word model you need to create a common framework from which every model will be created. Although HTK builds these models for you, they need to be set up correctly or HTK can never work out values. These values (Hidden Markov Model values) are transition probabilities. That is HTK works out the probabilities of transitions from one state to another. The diagram below shows an example of this. It was important that each word was assigned the right number of states (a word is made up of a number of states). The number of states was derived from the number of phonemes in a word. As well as a state for each phone in a word, there is a silence at the start of a word, followed by a lead up state to the first phone. there are also states between each of the phones and then two states at the end to follow the final phone in a word. The following table contains a list of the the words we used, there phones, and the number of states we assigned. NOTE, we did not use phones in this project. They were used only at this stage to derive the number of transition states in a word.



Word	Phones	Num states
Doe	d ow	7
Eight	ey t	7
Fa	f aa	7
Five	f ay v	9
Four	f ao r	9
La	l aa	7
Me	m iy	7
Nine	n ay n	9
One	w ah n	9
Ray	r ey	7
Seven	s eh v n	11
Six	s ih k s	11
So	s ow	7
Tee	t iy	7
Three	th r iy	9
Two	t uw	7


Once the number of states had been decided, the next step was to assign an initial set of values for the Markov models. The models are a square matrix which is dependent on the number of states in the word. For example, any word with 7 states was given initial Markov vales of the following:
These can be found in any users 'hmm0\hmmdefs' file. Every user in the system will have one of these models for each of the words in the grammar. These values are then altered/re-estimated by the HTK functions HCompV and HERest, where they are used in conjunction with the training data. HTK must be supplied with transcriptions of the data being used so that it can build the models from it. These transcriptions must be in a form that HTK can use. This transcription is an MLF (Master Label File) called 'words.mlf'. The recorded WAV files need to be converted to a format which HTK can manipulate. This is an 'MFC' format (Mel Frequency Cepstral Coefficients). This is done using HCopy.

HERest is used at a number of times and stages during the development of a users word model. HERest performs re-estimation over the full set of word models simultaneously. After we have run HERest a few times on our initial model, we add an 'sp'(short pause) model to a users model. We insert these into the transcription files as well. These are used to indicate the short pauses that occur between the words as they are spoken. HHEd is then used to manipulate the hmm's and further re-estimation occurs again using HERest. HVite is then used to force align the data. It is later used to calculate the log likelihood values needed for testing. Then we use HERest twice more for further re-estimation. We now have our final word model built for a user.

We stored all this is a perl script called build.pl which can be viewed in the section for source code.

Acceptance Criteria

With the models built, we now needed to decide upon some criteria for a which a user must comply to be accepted into the system. We accumulated as much data as possible to look for trends in the output of users test data against given models. We built an antimodel, which was like a user model, except it was made up of speech from users that were never going to use the system. This was made up of ten people. The training data from these ten people was all combined into one model. Whenever a user tried to log on, their speech was compared against their own word model as well as against the anti-speaker model. When you compare the test data with a built model you get an mlf file that looks like the following
#!MLF!#
"ver.rec"
0 2900000 FOUR -2184.767090
2900000 3400000 ME -435.572388
3400000 56400000 SIX -36530.304688
56400000 57600000 EIGHT -846.197083
57600000 58300000 RAY -538.054749
58300000 58600000 TEE -265.183105
58600000 60100000 NINE -1074.599243
60100000 61200000 TWO -813.827332
61200000 100000000 SO -26589.082031
.

The first two columns are to do with time alignment. It is the fourth column that we are interested in. This column contains the log-likelihood values for a word spoken. We sum these values using a perl script against both the users model and also against the anti-speaker model. It is the difference between both of these sums that interests us.

To calculate the threshold value, we get the user to record 5 more samples. Each of these samples is compared against the users model as well as against 5 other speaker models in the system(that are not in the anti-speaker model). We plotted these values in 'matlab' as can be seen below. They are represented by a normal distribution of the values supplied. In the example below, the red curve represents 25 anti-speaker model values. The green curve represents the 5 correct speaker model values and the blue line represents the threshold where the threshold is the mean of the incorrect vales plus twice the standard deviation of the incorrect.
Unfortunately the results did not always show up as clearly as the above. In the next example, the distributions overlap and the threshold in this example is set as the mean of the incorrect values + 1.2817 the standard deviation of the incorrect values. Sometimes there can be even more of an overlap then in the next example. To set the threshold, we calculated it as being "meanX + (n * stdX)" where meanX is the mean value of the anti-speakers and stdX is the standard deviation of the anti-speakers. The n value above decides the severity of the threshold. The higher the n value, the less chance there is of an anti-user getting into the system as the given user. 'n' above represents a Z value, so a Z value of 1.64 for example would mean that there was only a 5% chance of an anti user getting in. To get the probability of the user getting rejected on their own log in, we used the formula "(T - meanY) / stdY" where T is the set threshold value, meanY is the mean of the speaker values and stdY is the standard deviation of the speaker values. This gives a z value which we convert to a percentage format using a summarised lookup table. We used the following conversion table choosing the closest match

%	zvalue
25 %	0.6742
20 %	0.8415
15 %	1.0364
10 %	1.2817
5 %	1.6452
2 %	2.0542
1 %	2.3268
0.5 %	2.5762
0.1 %	3.0905
0.05 %	3.2908
0.01 %	3.7191


A user then has the opportunity to alter there threshold value. They choose what percentage chance they wanted anti-users to have of logging in as them. Obviously, the lower this percentage the larger the chance that the user will be rejected on their own log on. When they pick the percentage chance they want for anti-users being able to log on as them, the system then calculates the chance of them the user getting rejected on their own log-in. Each user has a file called 't.txt' stored in their directory that stores four values. The first value is their threshold. The second value is the z value for allowing anti-users log in as them. The third value is the percentage chance of anti-user getting in, and the fourth value is the percentage chance of the user getting rejected.

Code Structure
Client Code Structure

Client.java	Basic Functions Construct GUI Connect to Server Logon Process Sends Username to Server Receives Random Text Calls Record.java with Text New User Process Sends Username to Server Calls NewUser.java with Username Set Threshold Process Sends Username to Server If no threshold already exists, Call Threshold.java with Useranme
Record.java	Paint GUI showing random text and Record Button Record a 10 second WAV (Using SoundBite Class) Save WAV to disk (Using SoundBite Class) Send WAV to Server Receive threshold and sum of likelihoods from server Compare received vales and grant access/reject
NewUser.java	Paint GUI showing 17 text fields and 17 record buttons + Finish button Record 17 10 second WAVs (Using SoundBite Class) Save WAVs to disk (Using SoundBite Class) Send WAVs to Server
Threshold.java	Paint GUI showing 5 text fields and 5 record buttons + Finish button Record 5 10 second WAVs (Using SoundBite Class) Save WAVs to disk (Using SoundBite Class) Send WAVs to Server Receive user's security level Call setSecurity.java with security values
setSecurity.java	Paint GUI a JTextArea and a JList Send JList value to server, to change security level
AboutBox.java	Shows Project about box (Credits and Version information)

Server Code Structure

Server.java	Basic Functions Construct GUI Connect to Client Logon Process Receive Username from Client Call generateText.java Send result to Client Receive WAV from Client Call verify perl script Send result to Client Send user threshold to Client New User Process Receive Username from Client Calls a perl script to create a new user directory on Server Receive 17 WAV files from Client Call a perl script to move files into user's folder Call a perl script to build the user model Set Threshold Process Receive Username from Client Receive 5 WAV files from Client Call a perl script to move files into user's folder Call a perl script to build the user threshold Call calcThreshold.java with initial value of 1.2817 Send lines 3 and 4 of t.txt to Client Receive new security level from Client and call calcThreshold again with this value
generateText.java	Using system current time in Milliseconds get a random number Using this pick out 9 words from the dictionary of digits and Doe, Ray, Me..... Generate "testref.mlf" from these values Generate the grammar file "gram" from these values
calcThreshold.java	Receives a username and a z-value for possibility of anti-user getting in. Calculate mean and standard deviation of user and anti user values. Calculate Threshold value for a user. Calculate % chance of anti-user getting accepted and % chance of user getting rejected. Store threshold value, z-value of anti-user getting in and the two percentage values for the user.

Problems Encountered

We came across a number of difficulties during the development of this project.


• HTK is designed for Unix use. The version they supply is for Unix use and so, we had to port it to NT so it could be used for our requirements.
• Getting used to Using HTK proved very tough. Neither of us had ever done modules in speech or pattern matching so it was a completely new field to both of us. When we started using HTK we weren't even sure what we were looking for out of it. We ended up spending the first three weeks of the project trying to build up a simple phone model for one user.
• We found that HTK was very reliant on the microphone used. We found that values for the same user varied greatly depending on which microphone was used. Therefore we found it to be quite important to train users on the same microphone that was going to be used for testing.
• The lab that we were working in was not ideal, with much background noise, which quite often interfered with the samples being recorded. Predominant background voices were often present on recordings (espacially in anti-speaker model, where we did not have much time to organise people to record).
• Ronan's computer used a different hub to the rest of the computers in the lab. This led to many transfer errors when the client tried to send data over. When a new user recorded their training data, 17 WAV files had to be sent to the server. This was frequently stopped by a transfer error.

Future Extensions to the Project

Possible usage

In our program, the X: Drive is mapped to a "secure" folder on the server, as an example to an application of this Project. In place of this, any other code could be placed: Sending file(s) from Server to Client, Printing Special Information to the User's screen or opening applications/URL's. Any information that would normally be behind a password or PIN Number would be well suited to this project. When security the level is set to 0.01%, this gives a 1 in 10000 chance of the incorrect person logging in, which is equal to the probability of guessing the correct PIN number from an ATM card on first attempt. The system can also be set to a lower level of security, so as to decrease the probability of the correct user not being granted access, this would be useful in environments where security was not a huge issue and when a user tries to log in they want to be granted access as quickly as possible. This would be quite typical in small private companies.

Possible Further development

Although we covered everything we had set out to cover at the start with our technical specification, there are a few things we would have liked to have added if we had had the time. We may possibly have added a database to the project. At the time of design and even up till the end of coding there had been no reason to have a database. HTK stores all its information in files and from the start we were using files because of HTK. There still isn't much need for a database with this project, but it may be nicer to use a database then to store information in files. If we had had the time we would also like to have added some cryptography. It wasn't necessary as our project was aiming to use voice as a form of security and were not concerned about the lack of security in the network.