Etienne Barnard

Prof

Prof Etienne Barnard holds a PhD degree in Electrical and Computer Engineering from Carnegie Mellon University, and has been active in research and development in pattern recognition and speech processing for many years. He has held a number of academic and industry positions, holds international patents in speech processing, and is a past Associate Editor of the IEEE Transactions on Neural Networks. Barnard’s research contributions have been recognized in various ways, including Google Research Awards in 2009, 2011 and 2014. He is currently an extraordinary professor at the NWU.

Publications

2020

Davel MH, Theunissen MW, Pretorius AP, Barnard E. DNNs as layers of cooperating classifiers. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20). New York; 2020.

Abstract BibTex Entry BibTeX Download PDF

Published (shortened) version

A robust theoretical framework that can describe and predict the generalization ability of deep neural networks (DNNs) in general circumstances remains elusive. Classical attempts have produced complexity metrics that rely heavily on global measures of compactness and capacity with little investigation into the effects of sub-component collaboration. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. By tracing the origin of these patterns, we show how such networks can be viewed as the combination of two information processing systems: one continuous and one discrete. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration. This perspective on DNN classification offers a novel way to think about generalization, in which different subsets of the training data are used to train distinct classifiers; those classifiers are then combined to perform the classification task, and their consistency is crucial for accurate classification.

@{236,
  author = {Marelie Davel and Marthinus Theunissen and Arnold Pretorius and Etienne Barnard},
  title = {DNNs as layers of cooperating classifiers},
  abstract = {A robust theoretical framework that can describe and predict the generalization ability of deep neural networks (DNNs) in general circumstances remains elusive. Classical attempts have produced complexity metrics that rely heavily on global measures of compactness and capacity with little investigation into the effects of sub-component collaboration. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. By tracing the origin of these patterns, we show how such networks can be viewed as the combination of two information processing systems: one continuous and one discrete. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration. This perspective on DNN classification offers a novel way to think about generalization, in which different subsets of the training data are used to train distinct classifiers; those classifiers are then combined to perform the classification task, and their consistency is crucial for accurate classification.},
  year = {2020},
  journal = {The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)},
  pages = {3725 - 3732},
  month = {07/02-12/02/2020},
  address = {New York},
}

2019

Theunissen MW, Davel MH, Barnard E. Insights regarding overfitting on noise in deep learning. In: South African Forum for Artificial Intelligence Research (FAIR). Cape Town, South Africa; 2019.

Abstract BibTex Entry BibTeX Download PDF

Insights regarding overfitting on noise in deep learning

The understanding of generalization in machine learning is in a state of flux. This is partly due to the elatively recent revelation that deep learning models are able to completely memorize training data and still perform appropriately on out-of-sample data, thereby contradicting long-held intuitions about generalization. The phenomenon was brought to light and discussed in a seminal paper by Zhang et al. [24]. We expand upon this work by discussing local attributes of neural network training within the context of a relatively simple and generalizable framework. We describe how various types of noise can be compensated for within the proposed framework in order to allow the global deep learning model to generalize in spite of interpolating spurious function descriptors. Empirically, we support our postulates with experiments involving overparameterized multilayer perceptrons and controlled noise in the training data. The main insights are that deep learning models are optimized for training data modularly, with different regions in the function space dedicated to fitting distinct kinds of sample information. Detrimental overfitting is largely prevented by the fact that different regions in the function space are used for prediction based on the similarity between new input data and that which has been optimized for.

@{284,
  author = {Marthinus Theunissen and Marelie Davel and Etienne Barnard},
  title = {Insights regarding overfitting on noise in deep learning},
  abstract = {The understanding of generalization in machine learning is in a state of flux. This is partly due to the elatively recent revelation that deep learning models are able to completely memorize training data and still perform appropriately on out-of-sample data, thereby contradicting long-held intuitions about generalization. The phenomenon was brought to light and discussed in a seminal paper by Zhang et al. [24]. We expand upon this work by discussing local attributes of neural network training within the context of a relatively simple and generalizable framework. We describe how various types of noise can be compensated for within the proposed framework in order to allow the global deep learning model to generalize in spite of interpolating spurious function descriptors. Empirically, we support our postulates with experiments involving overparameterized multilayer perceptrons and controlled noise in the training data. The main insights are that deep learning models are optimized for training data modularly, with different regions in the function space dedicated to fitting distinct kinds of sample information. Detrimental overfitting is largely prevented by the fact that different regions in the function space are used for prediction based on the similarity between new input data and that which has been optimized for.},
  year = {2019},
  journal = {South African Forum for Artificial Intelligence Research (FAIR)},
  pages = {49-63},
  address = {Cape Town, South Africa},
}

Pretorius AP, Barnard E, Davel MH. ReLU and sigmoidal activation functions. In: South African Forum for Artificial Intelligence Research (FAIR). Cape Town, South Africa: CEUR Workshop Proceedings; 2019.

Abstract BibTex Entry BibTeX Download PDF

ReLU and sigmoidal activation functions

The generalization capabilities of deep neural networks are not well understood, and in particular, the influence of activation functions on generalization has received little theoretical attention. Phenomena such as vanishing gradients, node saturation and network sparsity have been identified as possible factors when comparing different activation functions [1]. We investigate these factors using fully connected feedforward networks on two standard benchmark problems, and find that the most salient differences between networks with sigmoidal and ReLU activations relate to the way that class-distinctive information is propagated through a network.

@{279,
  author = {Arnold Pretorius and Etienne Barnard and Marelie Davel},
  title = {ReLU and sigmoidal activation functions},
  abstract = {The generalization capabilities of deep neural networks are not well understood, and in particular, the influence of activation functions on generalization has received little theoretical attention. Phenomena such as vanishing gradients, node saturation and network sparsity have been identified as possible factors when comparing different activation functions [1]. We investigate these factors using fully connected feedforward networks on two standard benchmark problems, and find that the most salient differences between networks with sigmoidal and ReLU activations relate to the way that class-distinctive information is propagated through a network.},
  year = {2019},
  journal = {South African Forum for Artificial Intelligence Research (FAIR)},
  pages = {37-48},
  month = {04/12-07/12},
  publisher = {CEUR Workshop Proceedings},
  address = {Cape Town, South Africa},
}

2017

Van der Walt C, Barnard E. Variable Kernel Density Estimation in High-dimensional Feature Spaces. In: AAAI Conf. on Artificial Intelligence (AAAI-17). ; 2017.

Abstract BibTex Entry BibTeX Download PDF

Variable Kernel Density Estimation in High-dimensional Feature Spaces

Estimating the joint probability density function of a dataset is a central task in many machine learning applications. In this work we address the fundamental problem of kernel bandwidth estimation for variable kernel density estimation in high-dimensional feature spaces. We derive a variable kernel bandwidth estimator by minimizing the leave-one-out entropy objective function and show that this estimator is capable of performing estimation in high-dimensional feature spaces with great success. We compare the performance of this estimator to state-of-the art maximum likelihood estimators on a number of representative high-dimensional machine learning tasks and show that the newly introduced minimum leave-one-out entropy estimator performs optimally on a number of high-dimensional datasets considered.

@{275,
  author = {Christiaan Van der Walt and Etienne Barnard},
  title = {Variable Kernel Density Estimation in High-dimensional Feature Spaces},
  abstract = {Estimating the joint probability density function of a dataset is a central task in many machine learning applications. In this work we address the fundamental problem of kernel bandwidth estimation for variable kernel density estimation in high-dimensional feature spaces. We derive a variable kernel bandwidth estimator by minimizing the leave-one-out entropy objective function and show that this estimator is capable of performing estimation in high-dimensional feature spaces with great success. We compare the performance of this estimator to state-of-the art maximum likelihood estimators on a number of representative high-dimensional machine learning tasks and show that the newly introduced minimum leave-one-out entropy estimator performs optimally on a number of high-dimensional datasets considered.},
  year = {2017},
  journal = {AAAI Conf. on Artificial Intelligence (AAAI-17)},
  pages = {2674-2680},
  month = {04/02-09/04},
}

2015

Kleynhans N, Barnard E. Efficient data selection for ASR. Language Resources and Evaluation. 2015;49(2). doi:10.1007/s10579-014-9285-0.

Abstract BibTex Entry BibTeX Download PDF

Efficient data selection for ASR

Automatic speech recognition (ASR) technology has matured over the past few decades and has made significant impacts in a variety of fields, from assistive technologies to commercial products. However, ASR system development is a resource intensive activity and requires language resources in the form of text annotated audio recordings and pronunciation dictionaries. Unfortunately, many languages found in the developing world fall into the resource-scarce category and due to this resource scarcity the deployment of ASR systems in the developing world is severely inhibited. One approach to assist with resource-scarce ASR system development, is to select 'useful' training samples which could reduce the resources needed to collect new corpora. In this work, we propose a new data selection framework which can be used to design a speech recognition corpus. We show for limited data sets, independent of language and bandwidth, the most effective strategy for data selection is frequency-matched selection and that the widely-used maximum entropy methods generally produced the least promising results. In our model, the frequency-matched selection method corresponds to a logarithmic relationship between accuracy and corpus size; we also investigated other model relationships, and found that a hyperbolic relationship (as suggested from simple asymptotic arguments in learning theory) may lead to somewhat better performance under certain conditions.

@article{291,
  author = {Neil Kleynhans and Etienne Barnard},
  title = {Efficient data selection for ASR},
  abstract = {Automatic speech recognition (ASR) technology has matured over the past few decades and has made significant impacts in a variety of fields, from assistive technologies to commercial products. However, ASR system development is a resource intensive activity and requires language resources in the form of text annotated audio recordings and pronunciation dictionaries. Unfortunately, many languages found in the developing world fall into the resource-scarce category and due to this resource scarcity the deployment of ASR systems in the developing world is severely inhibited. One approach to assist with resource-scarce ASR system development, is to select &#039;useful&#039; training samples which could reduce the resources needed to collect new corpora. In this work, we propose a new data selection framework which can be used to design a speech recognition corpus. We show for limited data sets, independent of language and bandwidth, the most effective strategy for data selection is frequency-matched selection and that the widely-used maximum entropy methods generally produced the least promising results. In our model, the frequency-matched selection method corresponds to a logarithmic relationship between accuracy and corpus size; we also investigated other model relationships, and found that a hyperbolic relationship (as suggested from simple asymptotic arguments in learning theory) may lead to somewhat better performance under certain conditions.},
  year = {2015},
  journal = {Language Resources and Evaluation},
  volume = {49},
  pages = {327-353},
  issue = {2},
  publisher = {Springer Science+Business Media},
  address = {Dordrecht},
  doi = {10.1007/s10579-014-9285-0},
}

Kleynhans N, De Wet F, Barnard E. Unsupervised acoustic model training: comparing South African English and isiZulu. In: Pattern Recognition Association of South Africa (PRASA). Port Elizabeth, South Africa; 2015. doi: 10.1109/RoboMech.2015.7359512.

Abstract BibTex Entry BibTeX Download PDF

Unsupervised acoustic model training: comparing South African English and isiZulu

Large amounts of untranscribed audio data are generated every day. These audio resources can be used to develop robust acoustic models that can be used in a variety of speech-based systems. Manually transcribing this data is resource intensive and requires funding, time and expertise. Lightly-supervised training techniques, however, provide a means to rapidly transcribe audio, thus reducing the initial resource investment to begin the modelling process. Our findings suggest that the lightly-supervised training technique works well for English but when moving to an agglutinative language, such as isiZulu, the process fails to achieve the performance seen for English. Additionally, phone-based performances are significantly worse when compared to an approach using word-based language models. These results indicate a strong dependence on large or well-matched text resources for lightly-supervised training techniques.

@{290,
  author = {Neil Kleynhans and Febe De Wet and Etienne Barnard},
  title = {Unsupervised acoustic model training: comparing South African English and isiZulu},
  abstract = {Large amounts of untranscribed audio data are generated every day. These audio resources can be used to develop robust acoustic models that can be used in a variety of speech-based systems. Manually transcribing this data is resource intensive and requires funding, time and expertise. Lightly-supervised training techniques, however, provide a means to rapidly transcribe audio, thus reducing the initial resource investment to begin the modelling process. Our findings suggest that the lightly-supervised training technique works well for English but when moving to an agglutinative language, such as isiZulu, the process fails to achieve the performance seen for English. Additionally, phone-based performances are significantly worse when compared to an approach using word-based language models. These results indicate a strong dependence on large or well-matched text resources for lightly-supervised training techniques.},
  year = {2015},
  journal = {Pattern Recognition Association of South Africa (PRASA)},
  pages = {136-141},
  address = {Port Elizabeth, South Africa},
  isbn = {978-1-4673-7450-7, 978-1-4673-7449-1},
  doi = {10.1109/RoboMech.2015.7359512},
}

Davel MH, Barnard E, Van Heerden CJ, Hartman W, Karakos D, Schwartz R, Tsakalidis S. Exploring minimal pronunciation modeling for low resource languages. In: Interspeech. Dresden, Germany; 2015.

Abstract BibTex Entry BibTeX Download PDF

Exploring minimal pronunciation modeling for low resource languages

Pronunciation lexicons can range from fully graphemic (modeling each word using the orthography directly) to fully phonemic (first mapping each word to a phoneme string). Between these two options lies a continuum of modeling options. We analyze techniques that can improve the accuracy of a graphemic system without requiring significant effort to design or implement. The analysis is performed in the context of the IARPA Babel project, which aims to develop spoken term detection systems for previously unseen languages rapidly, and with minimal human effort. We consider techniques related to letter-to-sound mapping and language-independent syllabification of primarily graphemic systems, and discuss results obtained for six languages: Cebuano, Kazakh, Kurmanji Kurdish, Lithuanian, Telugu and Tok Pisin.

@{288,
  author = {Marelie Davel and Etienne Barnard and Charl Van Heerden and William Hartman and Damianos Karakos and Richard Schwartz and Stavros Tsakalidis},
  title = {Exploring minimal pronunciation modeling for low resource languages},
  abstract = {Pronunciation lexicons can range from fully graphemic (modeling each word using the orthography directly) to fully phonemic (first mapping each word to a phoneme string). Between these two options lies a continuum of modeling options. We analyze techniques that can improve the accuracy of a graphemic system without requiring significant effort to design or implement. The analysis is performed in the context of the IARPA Babel project, which aims to develop spoken term detection systems for previously unseen languages rapidly, and with minimal human effort. We consider techniques related to letter-to-sound mapping and language-independent syllabification of primarily graphemic systems, and discuss results obtained for six languages: Cebuano, Kazakh, Kurmanji Kurdish, Lithuanian, Telugu and Tok Pisin.},
  year = {2015},
  journal = {Interspeech},
  pages = {538-542},
  address = {Dresden, Germany},
}