CAIR Deep Learning Research Publications

2024

Mouton, C., Rabe, R., Haasbroek, D. G., Theunissen, M. W., Potgieter, H., & Davel, M. H. (2024). Is network fragmentation a useful complexity measure? NeurIPS 2024 Workshop SciForDL.

It has been observed that the input space of deep neural network classifiers can exhibit ‘fragmentation’, where the model function rapidly changes class as the input space is traversed. The severity of this fragmentation tends to follow the double descent curve, achieving a maximum at the interpolation regime. We study this phenomenon in the context of image classification and ask whether fragmentation could be predictive of generalization performance. Using a fragmentation-based complexity measure, we show this to be possible by achieving good performance on the PGDL (Predicting Generalization in Deep Learning) benchmark. In addition, we report on new observations related to fragmentation, namely (i) fragmentation is not limited to the input space but occurs in the hidden representations as well, (ii) fragmentation follows the trends in the validation error throughout training, and (iii) fragmentation is not a direct result of increased weight norms. Together, this indicates that fragmentation is a phenomenon worth investigating further when studying the generalization ability of deep neural networks.

@misc{514,
  author = {Coenraad Mouton and Randle Rabe and Daniël Haasbroek and Marthinus Theunissen and Harmen Potgieter and Marelie Davel},
  title = {Is network fragmentation a useful complexity measure?},
  abstract = {It has been observed that the input space of deep neural network classifiers can exhibit ‘fragmentation’, where the model function rapidly changes class as the input space is traversed. The severity of this fragmentation tends to follow the double descent curve, achieving a maximum at the interpolation regime. We study this phenomenon in the context of image classification and ask whether fragmentation could be predictive of generalization performance. Using a fragmentation-based complexity measure, we show this to be possible by achieving good performance on the PGDL (Predicting Generalization in Deep Learning) benchmark. In addition, we report on new observations related to fragmentation, namely (i) fragmentation is not limited to the input space but occurs in the hidden representations as well, (ii) fragmentation follows the trends in the validation error throughout training, and (iii) fragmentation is not a direct result of increased weight norms. Together, this indicates that fragmentation is a phenomenon worth investigating further when studying the generalization ability of deep neural networks.},
  year = {2024},
  journal = {NeurIPS 2024 Workshop SciForDL},
  month = {12/2024},
}
Le Roux, V., Davel, M. H., & Bosman, J. (2024). Parsimonious airfoil Parameterisation: A deep learning framework with Bidirectional LSTM and Gaussian Mixture models. Expert Systems With Applications, 255. http://doi.org/https://doi.org/10.1016/j.eswa.2024.124726

The choice of airfoil parameterisation method significantly influences the overall wing optimisation performance by affecting the flexibility and computational efficiency of the process. Ideally, one should be able to intuitively constrain airfoil shape and structural characteristics as input to the optimisation process. Current parameterisation techniques lack the flexibility to generate airfoils efficiently by specifying parsimonious shape and structural features. To address this limitation, a deep learning framework is proposed, enabling conditional airfoil generation from an airfoil’s shape and structural feature definition. Specifically, we demonstrate the application of Bidirectional Long Short Term Memory models and Bayesian Gaussian Mixture models to derive airfoil coordinates from a compact set of shape and structural characteristics that we define. The proposed framework is shown to achieve favorable airfoil performance optimisation due to improved exploration and exploitation of the design space, compared to traditional approaches. Overall, the proposed optimisation framework is able to realise a 9.04% performance improvement over an airfoil design optimised with traditional parameterisation techniques.

@article{513,
  author = {Vincent Le Roux and Marelie Davel and Johan Bosman},
  title = {Parsimonious airfoil Parameterisation: A deep learning framework with Bidirectional LSTM and Gaussian Mixture models},
  abstract = {The choice of airfoil parameterisation method significantly influences the overall wing optimisation performance by affecting the flexibility and computational efficiency of the process. Ideally, one should be able to intuitively constrain airfoil shape and structural characteristics as input to the optimisation process. Current parameterisation techniques lack the flexibility to generate airfoils efficiently by specifying parsimonious shape and structural features. To address this limitation, a deep learning framework is proposed, enabling conditional airfoil generation from an airfoil’s shape and structural feature definition. Specifically, we demonstrate the application of Bidirectional Long Short Term Memory models and Bayesian Gaussian Mixture models to derive airfoil coordinates from a compact set of shape and structural characteristics that we define. The proposed framework is shown to achieve favorable airfoil performance optimisation due to improved exploration and exploitation of the design space, compared to traditional approaches. Overall, the proposed optimisation framework is able to realise a 9.04% performance improvement over an airfoil design optimised with traditional parameterisation techniques.},
  year = {2024},
  journal = {Expert Systems With Applications},
  volume = {255},
  month = {10 July 2024},
  doi = {https://doi.org/10.1016/j.eswa.2024.124726},
}
Mouton, C., Theunissen, M. W., & Davel, M. H. (2024). Input margins can predict generalization too. In In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’24/IAAI’24/E. AAAI Conference on Artificial Intelligence (AAAI).

Understanding generalization in deep neural networks is an active area of research. A promising avenue of exploration has been that of margin measurements: the shortest distance to the decision boundary for a given sample or its representation internal to the network. While margins have been shown to be correlated with the generalization ability of a model when measured at its hidden representations (hidden margins), no such link between large margins and generalization has been established for input margins. We show that while input margins are not generally predictive of generalization, they can be if the search space is appropriately constrained. We develop such a measure based on input margins, which we refer to as ‘constrained margins’. The predictive power of this new measure is demonstrated on the ‘Predicting Generalization in Deep Learning’ (PGDL) dataset and contrasted with hidden representation margins. We find that constrained margins achieve highly competitive scores and outperform other margin measurements in general. This provides a novel insight on the relationship between generalization and classification margins, and highlights the importance of considering the data manifold for investigations of generalization in DNNs

@inbook{512,
  author = {Coenraad Mouton and Marthinus Theunissen and Marelie Davel},
  title = {Input margins can predict generalization too},
  abstract = {Understanding generalization in deep neural networks is an active area of research. A promising avenue of exploration has been that of margin measurements: the shortest distance to the decision boundary for a given sample or its representation internal to the network. While margins have been shown to be correlated with the generalization ability of a model when measured at its hidden representations (hidden margins), no such link between large margins and generalization has been established for input margins. We show that while input margins are not generally predictive of generalization, they can be if the search space is appropriately constrained. We develop such a measure based on input margins, which we refer to as ‘constrained margins’. The predictive power of this new measure is demonstrated on the ‘Predicting Generalization in Deep Learning’ (PGDL) dataset and contrasted with hidden representation margins. We find that constrained margins achieve highly competitive scores and outperform other margin measurements in general. This provides a novel insight on the relationship between generalization and classification margins, and highlights the importance of considering the data manifold for investigations of generalization in DNNs},
  year = {2024},
  journal = {In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence (AAAI'24/IAAI'24/E},
  pages = {14379 - 14387},
  month = {20 February 2024},
  publisher = {AAAI Conference on Artificial Intelligence (AAAI)},
}

2023

Davel, M. H., Lotz, S., Theunissen, M. W., De Villiers, A., Grant, C., Rabe, R., & Conacher, C. (2023). Knowledge Discovery in Time Series Data. In Deep Learning Indaba 2023.

• Complex time series data often encountered in scientific and engineering domains. • Deep learning (DL) is particularly successful here: – large data sets, multivariate input and/or ouput, – highly complex sequences of interactions. • Model interpretability: – Ability to understand a model’s decisions in a given context [1]. – Techniques typically not originally developed for time series data. – Time series interpretations themselves become uninterpretable. • Knowledge Discovery: – DL has potential to reveal interesting patterns in large data sets. – Potential to produce novel insights about the task itself [2, 3]. • ‘know-it’: Collaborative project that studies knowledge discovery in time series data.

@{507,
  author = {Marelie Davel and Stefan Lotz and Marthinus Theunissen and Almaro De Villiers and Chara Grant and Randle Rabe and Cleo Conacher},
  title = {Knowledge Discovery in Time Series Data},
  abstract = {• Complex time series data often encountered in scientific and engineering domains.
• Deep learning (DL) is particularly successful here:
– large data sets, multivariate input and/or ouput,
– highly complex sequences of interactions.
• Model interpretability:
– Ability to understand a model’s decisions in a given context [1].
– Techniques typically not originally developed for time series data.
– Time series interpretations themselves become uninterpretable.
• Knowledge Discovery:
– DL has potential to reveal interesting patterns in large data sets.
– Potential to produce novel insights about the task itself [2, 3].
• ‘know-it’: Collaborative project that studies knowledge discovery in
time series data.},
  year = {2023},
  journal = {Deep Learning Indaba 2023},
  month = {September 2023},
}
Olivier, J. C., & Barnard, E. (2023). Minimum phase finite impulse response filter design. The Institute of Engineering and Technology, 17. http://doi.org/ https://doi.org/10.1049/sil2.12166

The design of minimum phase finite impulse response (FIR) filters is considered. The study demonstrates that the residual errors achieved by current state-of-the-art design methods are nowhere near the smallest error possible on a finite resolution digital computer. This is shown to be due to conceptual errors in the literature pertaining to what constitutes a factorable linear phase filter. This study shows that factorisation is possible with a zero residual error (in the absence of machine finite resolution error) if the linear operator or matrix representing the linear phase filter is positive definite. Methodology is proposed able to design a minimum phase filter that is optimal—in the sense that the residual error is limited only by the finite precision of the digital computer, with no systematic error. The study presents practical application of the proposed methodology by designing two minimum phase Chebyshev FIR filters. Results are compared to state-of-the-art methods from the literature, and it is shown that the proposed methodology is able to reduce currently achievable residual errors by several orders of magnitude.

@article{506,
  author = {Jan Olivier and Etienne Barnard},
  title = {Minimum phase finite impulse response filter design},
  abstract = {The design of minimum phase finite impulse response (FIR) filters is considered. The study demonstrates that the residual errors achieved by current state-of-the-art design methods are nowhere near the smallest error possible on a finite resolution digital computer. This is shown to be due to conceptual errors in the literature pertaining to what constitutes a factorable linear phase filter. This study shows that factorisation is possible with a zero residual error (in the absence of machine finite resolution error) if the linear operator or matrix representing the linear phase filter is positive definite. Methodology is proposed able to design a minimum phase filter that is optimal—in the sense that the residual error is limited only by the finite precision of the digital computer, with no systematic error. The study presents practical application of the proposed methodology by designing two minimum phase Chebyshev FIR filters. Results are compared to state-of-the-art methods from the literature, and it is shown that the proposed methodology is able to reduce currently achievable residual errors by several orders of magnitude.},
  year = {2023},
  journal = {The Institute of Engineering and Technology},
  volume = {17},
  edition = {7},
  month = {July 2023},
  doi = {https://doi.org/10.1049/sil2.12166},
}
Ngorima, A., Helberg, A., & Davel, M. H. (2023). Sequence Based Deep Neural Networks for Channel Estimation in Vehicular Communication Systems. In Artificial Intelligence Research. SACAIR 2023. Communications in Computer and Information Science (Vol. 1976). Springer, Cham. http://doi.org/https://doi.org/10.1007/978-3-031-49002-6_12

Channel estimation is a critical component of vehicular communications systems, especially in high-mobility scenarios. The IEEE 802.11p standard uses preamble-based channel estimation, which is not sufficient in these situations. Recent work has proposed using deep neural networks for channel estimation in IEEE 802.11p. While these methods improved on earlier baselines they still can perform poorly, especially in very high mobility scenarios. This study proposes a novel approach that uses two independent LSTM cells in parallel and averages their outputs to update cell states. The proposed approach improves normalised mean square error, surpassing existing deep learning approaches in very high mobility scenarios.

@inbook{504,
  author = {Aldrin Ngorima and Albert Helberg and Marelie Davel},
  title = {Sequence Based Deep Neural Networks for Channel Estimation in Vehicular Communication Systems},
  abstract = {Channel estimation is a critical component of vehicular communications systems, especially in high-mobility scenarios. The IEEE 802.11p standard uses preamble-based channel estimation, which is not sufficient in these situations. Recent work has proposed using deep neural networks for channel estimation in IEEE 802.11p. While these methods improved on earlier baselines they still can perform poorly, especially in very high mobility scenarios. This study proposes a novel approach that uses two independent LSTM cells in parallel and averages their outputs to update cell states. The proposed approach improves normalised mean square error, surpassing existing deep learning approaches in very high mobility scenarios.},
  year = {2023},
  journal = {Artificial Intelligence Research. SACAIR 2023. Communications in Computer and Information Science},
  volume = {1976},
  pages = {176 - 186},
  month = {29 November 2023},
  publisher = {Springer, Cham},
  isbn = {978-3-031-49001-9},
  doi = {https://doi.org/10.1007/978-3-031-49002-6_12},
}
Lotz, S., Nel, A., Wicks, R., Roberts, O., Engelbrecht, N., Strauss, R., … Bale, S. (2023). The Radial Variation of the Solar Wind Turbulence Spectra near the Kinetic Break Scale from Parker Solar Probe Measurements. In The Astrophysical Journal (2nd ed., Vol. 942). The American Astronomical Society. http://doi.org/10.3847/1538-4357/aca903

In this study we examine the radial dependence of the inertial and dissipation range indices, as well as the spectral break separating the inertial and dissipation range in power density spectra of interplanetary magnetic field fluctuations using Parker Solar Probe data from the fifth solar encounter between ∼0.1 and ∼0.7 au. The derived break wavenumber compares reasonably well with previous estimates at larger radial distances and is consistent with gyro-resonant damping of Alfvénic fluctuations by thermal protons. We find that the inertial scale power-law index varies between approximately −1.65 and −1.45. This is consistent with either the Kolmogorov (−5/3) or Iroshnikov–Kraichnan (−3/2) values, and has a very weak radial dependence with a possible hint that the spectrum becomes steeper closer to the Sun. The dissipation range power-law index, however, has a clear dependence on radial distance (and turbulence age), decreasing from −3 near 0.7 au (4 days) to −4 [±0.3] at 0.1 au (0.75 days) closer to the Sun.

@inbook{503,
  author = {Stefan Lotz and Amore Nel and Robert Wicks and Owen Roberts and Nicholas Engelbrecht and Roelf Strauss and Gert Botha and Eduard Kontar and Alexander Pitňa and Stuart Bale},
  title = {The Radial Variation of the Solar Wind Turbulence Spectra near the Kinetic Break Scale from Parker Solar Probe Measurements},
  abstract = {In this study we examine the radial dependence of the inertial and dissipation range indices, as well as the spectral break separating the inertial and dissipation range in power density spectra of interplanetary magnetic field fluctuations using Parker Solar Probe data from the fifth solar encounter between ∼0.1 and ∼0.7 au. The derived break wavenumber compares reasonably well with previous estimates at larger radial distances and is consistent with gyro-resonant damping of Alfvénic fluctuations by thermal protons. We find that the inertial scale power-law
index varies between approximately −1.65 and −1.45. This is consistent with either the Kolmogorov (−5/3) or Iroshnikov–Kraichnan (−3/2) values, and has a very weak radial dependence with a possible hint that the spectrum becomes steeper closer to the Sun. The dissipation range power-law index, however, has a clear dependence on radial distance (and turbulence age), decreasing from −3 near 0.7 au (4 days) to −4 [±0.3] at 0.1 au (0.75 days) closer to the Sun.},
  year = {2023},
  journal = {The Astrophysical Journal},
  volume = {942},
  edition = {2},
  month = {01/2023},
  publisher = {The American Astronomical Society},
  doi = {10.3847/1538-4357/aca903},
}
Ramalepe, S., Modipa, T., & Davel, M. H. (2023). The Analysis of the Sepedi-English Code-switched Radio News Corpus. Journal of the Digital Humanities Association of Southern Africa, 4(Vol. 4 No. 01 (2022): Proceedings of the 3rd workshop on Resources for African Indigenous Languages (RAIL). http://doi.org/https://doi.org/10.55492/dhasa.v4i01.4444

Code-switching is a phenomenon that occurs mostly in multilingual countries where multilingual speakers often switch between languages in their conversations. The unavailability of large scale code-switched corpora hampers the development and training of language models for the generation of code-switched text. In this study, we explore the initial phase of collecting and creating Sepedi-English code-switched corpus for generating synthetic news. Radio news and the frequency of code-switching on read news were considered and analysed. We developed and trained a Transformer-based language model using the collected code-switched dataset. We observed that the frequency of code-switched data in the dataset was very low at 1.1%. We complemented our dataset with the news headlines dataset to create a new dataset. Although the frequency was still low, the model obtained the optimal loss rate of 2,361 with an accuracy of 66%.

@article{502,
  author = {Simon Ramalepe and Thipe Modipa and Marelie Davel},
  title = {The Analysis of the Sepedi-English Code-switched Radio News Corpus},
  abstract = {Code-switching is a phenomenon that occurs mostly in multilingual countries where multilingual speakers often switch between languages in
their conversations. The unavailability of large scale code-switched corpora hampers the development and training of language models for the generation of code-switched text. In this study, we explore the initial phase of collecting and creating Sepedi-English code-switched corpus for generating synthetic news. Radio news and the frequency of code-switching on read news were considered and analysed. We developed and trained a Transformer-based language model using the collected code-switched dataset. We observed that the frequency of code-switched data in the dataset was very low at 1.1%. We complemented our dataset with the news headlines dataset to create a new dataset.
Although the frequency was still low, the model obtained the optimal loss rate of 2,361 with an accuracy of 66%.},
  year = {2023},
  journal = {Journal of the Digital Humanities Association of Southern Africa},
  volume = {4},
  edition = {1},
  month = {2023-01-25},
  issue = {Vol. 4 No. 01 (2022): Proceedings of the 3rd workshop on Resources for African Indigenous Languages (RAIL)},
  doi = {https://doi.org/10.55492/dhasa.v4i01.4444},
}
Ramalepe, S., Modipa, T., & Davel, M. H. (2023). Transformer-based text generation for code-switched Sepedi-English news. In Southern African Conference for Artificial Intelligence Research (SACAIR).

Code-switched data is rarely available in written form and this makes the development of large datasets required to train codeswitched language models difficult. Currently, available Sepedi-English code-switched corpora are not large enough to train a Transformer-based model for this language pair. In prior work, larger synthetic datasets have been constructed using a combination of a monolingual and a parallel corpus to approximate authentic code-switched text. In this study, we develop and analyse a new Sepedi-English news dataset (SepEnews). We collect and curate data from local radio news bulletins and use this to augment two existing sources collected from Sepedi newspapers and news headlines, respectively. We then develop and train a Transformer-based model for generating historic code-switched news, and demonstrate and analyse the system’s performance.

@{501,
  author = {Simon Ramalepe and Thipe Modipa and Marelie Davel},
  title = {Transformer-based text generation for code-switched Sepedi-English news},
  abstract = {Code-switched data is rarely available in written form and this makes the development of large datasets required to train codeswitched language models difficult. Currently, available Sepedi-English code-switched corpora are not large enough to train a Transformer-based
model for this language pair. In prior work, larger synthetic datasets have been constructed using a combination of a monolingual and a parallel
corpus to approximate authentic code-switched text. In this study, we develop and analyse a new Sepedi-English news dataset (SepEnews). We collect and curate data from local radio news bulletins and use this to augment two existing sources collected from Sepedi newspapers and news headlines, respectively. We then develop and train a Transformer-based model for generating historic code-switched news, and demonstrate and analyse the system’s performance.},
  year = {2023},
  journal = {Southern African Conference for Artificial Intelligence Research (SACAIR)},
  pages = {84 - 97},
  month = {December 2023},
}
Middel, C., & Davel, M. H. (2023). Comparing Transformer-based and GBDT models on tabular data: A Rossmann Store Sales case study. In Southern African Conference for Artificial Intelligence Research (SACAIR).

Heterogeneous tabular data is a common and important data format. This empirical study investigates how the performance of deep transformer models compares against benchmark gradient boosting decision tree (GBDT) methods, the more typical modelling approach. All models are optimised using a Bayesian hyperparameter optimisation protocol, which provides a stronger comparison than the random grid search hyperparameter optimisation utilized in earlier work. Since feature skewness is typically handled differently for GBDT and transformer-based models, we investigate the effect of a pre-processing step that normalises feature distribution on the model comparison process. Our analysis is based on the Rossmann Store Sales dataset, a widely recognized benchmark for regression tasks.

@{500,
  author = {Coenraad Middel and Marelie Davel},
  title = {Comparing Transformer-based and GBDT models on tabular data: A Rossmann Store Sales case study},
  abstract = {Heterogeneous tabular data is a common and important data format. This empirical study investigates how the performance of deep transformer models compares against benchmark gradient boosting decision tree (GBDT) methods, the more typical modelling approach. All models are optimised using a Bayesian hyperparameter optimisation protocol, which provides a stronger comparison than the random grid search hyperparameter optimisation utilized in earlier work. Since feature skewness is typically handled differently for GBDT and transformer-based models, we investigate the effect of a pre-processing step that normalises feature distribution on the model comparison process. Our analysis is based on the Rossmann Store Sales dataset, a widely recognized benchmark for regression tasks.},
  year = {2023},
  journal = {Southern African Conference for Artificial Intelligence Research (SACAIR)},
  pages = {115 - 129},
  month = {December 2023},
}

2022

Ramalepe, S., Modipa, T., & Davel, M. H. (2022). The development of a Sepedi text generation model using transformers. In Southern Africa Telecommunication Networks and Applications Conference (SATNAC).

Text generation is one of the important sub-tasks of natural language generation (NLG), and aims to produce humanly readable text given some input text. Deep learning approaches based on neural networks have been proposed to solve text generation tasks. Although these models can generate text, they do not necessarily capture long-term dependencies accurately, making it difficult to coherently generate longer sentences. Transformer-based models have shown significant improvement in text generation. However, these models are computationally expensive and data hungry. In this study, we develop a Sepedi text generation model using a Transformerbased approach and explore its performance. The developed model has one Transformer block with causal masking on the attention layers and two separate embedding layers. To train the model, we use the National Centre for Human Language Technology (NCHLT) Sepedi text corpus. Our experimental setup varied the model embedding size, batch size and the sequence length. The final model was able to reconstruct unseen test data with 75% accuracy: the highest accuracy achieved to date, using a Sepedi corpus.

@{511,
  author = {Simon Ramalepe and Thipe Modipa and Marelie Davel},
  title = {The development of a Sepedi text generation model using transformers},
  abstract = {Text generation is one of the important sub-tasks of natural language generation (NLG), and aims to produce humanly readable text given some input text. Deep learning approaches based on neural networks have been proposed to solve text generation tasks. Although these models can generate text, they do not necessarily capture long-term dependencies accurately, making it difficult to coherently generate longer sentences. Transformer-based models have shown significant improvement in text generation. However, these models are computationally expensive and data hungry. In this study, we develop a Sepedi text generation model using a Transformerbased approach and explore its performance. The developed model has one Transformer block with causal masking on the attention layers and two separate embedding layers. To train the model, we use the National Centre for Human Language Technology (NCHLT) Sepedi text corpus. Our experimental setup varied the model embedding size, batch size and the sequence length. The final model was able to reconstruct unseen test data with 75% accuracy: the highest accuracy achieved to date, using a Sepedi corpus.},
  year = {2022},
  journal = {Southern Africa Telecommunication Networks and Applications Conference (SATNAC)},
  pages = {51 - 56},
  month = {August 2022},
}
Oosthuizen, M., Oosthuizen, M., & Davel, M. H. (2022). A Comparative Study of Graph Neural Network Speed Prediction during Periods of Congestion. In In Proceedings of the 14th International Joint Conference on Computational Intelligence (IJCCI 2022) - NCTA (Vol. 1). http://doi.org/10.5220/0011374100003332

Traffic speed prediction using deep learning has been the topic of many studies. In this paper, we analyse the performance of Graph Neural Network-based techniques during periods of traffic congestion. We first compare a selection of recently proposed techniques that claim to achieve good results using the METR-LA and PeMS-BAY data sets. We then investigate the performance of three of these approaches – Graph WaveNet, Spacetime Neural Network (STNN) and Spatio-Temporal Attention Wavenet (STAWnet) – during congested periods, using recurrent congestion patterns to set a threshold for general congestion through the entire traffic network. Our results show that performance deteriorates significantly during congested time periods, which is concerning, as traffic speed prediction is usually of most value during times of congestion. We also found that, while the above approaches perform almost equally in the absence of congestion, there are much bigger differences in performance during peri ods of congestion.

@{510,
  author = {Marko Oosthuizen and Marko Oosthuizen and Marelie Davel},
  title = {A Comparative Study of Graph Neural Network Speed Prediction during Periods of Congestion},
  abstract = {Traffic speed prediction using deep learning has been the topic of many studies. In this paper, we analyse the performance of Graph Neural Network-based techniques during periods of traffic congestion. We first compare a selection of recently proposed techniques that claim to achieve good results using the METR-LA and PeMS-BAY data sets. We then investigate the performance of three of these approaches – Graph WaveNet, Spacetime Neural Network (STNN) and Spatio-Temporal Attention Wavenet (STAWnet) – during congested periods, using recurrent congestion patterns to set a threshold for general congestion through the entire traffic network. Our results show that performance deteriorates significantly during congested time periods, which is concerning, as traffic speed prediction is usually of most value during times of congestion. We also found that, while the above approaches perform almost equally in the absence of congestion, there are much bigger differences in performance during peri ods of congestion.},
  year = {2022},
  journal = {In Proceedings of the 14th International Joint Conference on Computational Intelligence (IJCCI 2022) - NCTA},
  volume = {1},
  pages = {331 - 338},
  month = {October 2022},
  isbn = {978-989-758-611-8},
  doi = {10.5220/0011374100003332},
}
Oosthuizen, A. J., Helberg, A., & Davel, M. H. (2022). Adversarial training for channel state information estimation in LTE multi-antenna systems. In Southern African Conference for Artificial Intelligence Research (Vol. 1734). Springer, Cham. http://doi.org/https://doi.org/10.1007/978-3-031-22321-1_1

Deep neural networks can be utilised for channel state information (CSI) estimation in wireless communications. We aim to decrease the bit error rate of such networks without increasing their complexity, since the wireless environment requires solutions with high performance while constraining implementation cost. For this reason, we investigate the use of adversarial training, which has been successfully applied to image super-resolution tasks that share similarities with CSI estimation tasks. CSI estimators are usually trained in a Single-In Single-Out (SISO) configuration to estimate the channel between two specific antennas and then applied to multi-antenna configurations. We show that the performance of neural networks in the SISO training environment is not necessarily indicative of their performance in multi-antenna systems. The analysis shows that adversarial training does not provide advantages in the SISO environment, however, adversarially trained models can outperform non-adversarially trained models when applying antenna diversity to Long-Term Evolution systems. The use of a feature extractor network is also investigated in this study and is found to have the potential to enhance the performance of Multiple-In Multiple-Out antenna configurations at higher SNRs. This study emphasises the importance of testing neural networks in the context of use while also showing possible advantages of adversarial training in multi-antenna systems without necessarily increasing network complexity.

@{509,
  author = {Andrew Oosthuizen and Albert Helberg and Marelie Davel},
  title = {Adversarial training for channel state information estimation in LTE multi-antenna systems},
  abstract = {Deep neural networks can be utilised for channel state information (CSI) estimation in wireless communications. We aim to decrease the bit error rate of such networks without increasing their complexity, since the wireless environment requires solutions with high performance while constraining implementation cost. For this reason, we investigate the use of adversarial training, which has been successfully applied to image super-resolution tasks that share similarities with CSI estimation tasks. CSI estimators are usually trained in a Single-In Single-Out (SISO) configuration to estimate the channel between two specific antennas and then applied to multi-antenna configurations. We show that the performance of neural networks in the SISO training environment is not necessarily indicative of their performance in multi-antenna systems. The analysis shows that adversarial training does not provide advantages in the SISO environment, however, adversarially trained models can outperform non-adversarially trained models when applying antenna diversity to Long-Term Evolution systems. The use of a feature extractor network is also investigated in this study and is found to have the potential to enhance the performance of Multiple-In Multiple-Out antenna configurations at higher SNRs. This study emphasises the importance of testing neural networks in the context of use while also showing possible advantages of adversarial training in multi-antenna systems without necessarily increasing network complexity.},
  year = {2022},
  journal = {Southern African Conference for Artificial Intelligence Research},
  volume = {1734},
  pages = {3 - 17},
  month = {November 2022},
  publisher = {Springer, Cham},
  isbn = {978-3-031-22320-4},
  doi = {https://doi.org/10.1007/978-3-031-22321-1_1},
}
Fourie, E., Davel, M. H., & Versfeld, J. (2022). Neural speech processing for whale call detection. In Southern African Conference for AI Research (SACAIR) (Vol. 1734). Springer, Cham. http://doi.org/https://doi.org/10.1007/978-3-031-22321-1_19

Passive acoustic monitoring with hydrophones makes it possible to detect the presence of marine animals over large areas. For monitoring to be cost-effective, this process should be fully automated. We explore a new approach to detecting whale calls, using an end-to-end neural architecture and traditional speech features. We compare the results of the new approach with a convolutional neural network (CNN) applied to spectrograms, currently the standard approach to whale call detection. Experiments are conducted using the “Acoustic trends for the blue and fin whale library” from the Australian Antarctic Data Centre (AADC). We experiment with different types of speech features (mel frequency cepstral coefficients and filter banks) and different ways of framing the task. We demonstrate that a time delay neural network is a viable solution for whale call detection, with the additional benefit that spectrogram tuning – required to obtain high-quality spectrograms in challenging acoustic conditions – is no longer necessary. While the initial speech feature-based system (accuracy 96%) did not outperform the CNN (accuracy 98%) when trained on exactly the same dataset, it presents a viable approach to explore further.

@{508,
  author = {Edrich Fourie and Marelie Davel and Jaco Versfeld},
  title = {Neural speech processing for whale call detection},
  abstract = {Passive acoustic monitoring with hydrophones makes it possible to detect the presence of marine animals over large areas. For monitoring to be cost-effective, this process should be fully automated. We explore a new approach to detecting whale calls, using an end-to-end neural architecture and traditional speech features. We compare the results of the new approach with a convolutional neural network (CNN) applied to spectrograms, currently the standard approach to whale call detection. Experiments are conducted using the “Acoustic trends for the blue and fin whale library” from the Australian Antarctic Data Centre (AADC). We experiment with different types of speech features (mel frequency cepstral coefficients and filter banks) and different ways of framing the task. We demonstrate that a time delay neural network is a viable solution for whale call detection, with the additional benefit that spectrogram tuning – required to obtain high-quality spectrograms in challenging acoustic conditions – is no longer necessary. While the initial speech feature-based system (accuracy 96%) did not outperform the CNN (accuracy 98%) when trained on exactly the same dataset, it presents a viable approach to explore further.},
  year = {2022},
  journal = {Southern African Conference for AI Research (SACAIR)},
  volume = {1734},
  pages = {276 - 290},
  month = {November 2022},
  publisher = {Springer, Cham},
  doi = {https://doi.org/10.1007/978-3-031-22321-1_19},
}
Theunissen, M. W., Mouton, C., & Davel, M. H. (2022). The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs. In Artificial Intelligence Research (SACAIR 2022), Communications in Computer and Information Science (Vol. 1734). Springer, Cham. http://doi.org/ https://doi.org/10.48550/arXiv.2302.06925

Classification margins are commonly used to estimate the generalization ability of machine learning models. We present an empirical study of these margins in artificial neural networks. A global estimate of margin size is usually used in the literature. In this work, we point out seldom considered nuances regarding classification margins. Notably, we demonstrate that some types of training samples are modelled with consistently small margins while affecting generalization in different ways. By showing a link with the minimum distance to a different-target sample and the remoteness of samples from one another, we provide a plausible explanation for this observation. We support our findings with an analysis of fully-connected networks trained on noise-corrupted MNIST data, as well as convolutional networks trained on noise-corrupted CIFAR10 data.

@inbook{505,
  author = {Marthinus Theunissen and Coenraad Mouton and Marelie Davel},
  title = {The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs},
  abstract = {Classification margins are commonly used to estimate the generalization ability of machine learning models. We present an empirical study of these margins in artificial neural networks. A global estimate of margin size is usually used in the literature. In this work, we point out seldom considered nuances regarding classification margins. Notably, we demonstrate that some types of training samples are modelled with consistently small margins while affecting generalization in different ways. By showing a link with the minimum distance to a different-target sample and the remoteness of samples from one another, we provide a plausible explanation for this observation. We support our findings with an analysis of fully-connected networks trained on noise-corrupted MNIST data, as well as convolutional networks trained on noise-corrupted CIFAR10 data.},
  year = {2022},
  journal = {Artificial Intelligence Research (SACAIR 2022), Communications in Computer and Information Science},
  volume = {1734},
  pages = {78 - 92},
  month = {November 2022},
  publisher = {Springer, Cham},
  doi = {https://doi.org/10.48550/arXiv.2302.06925},
}
  • DSI
  • Covid-19