Temporal representation in the brain has been recently acknowledged as a fundamental mechanism underlying short-term memory (STM). Nonetheless, the existing body of research presents conflicting results on the extent of this relationship. Some studies propose that rhythmic disruption adversely affects perception and comprehension, and regular rhythm has been associated with the facilitative impact on STM tasks, while others suggest that its influence on STM tasks might not be as substantial. This thesis delves into rhythmic irregularity's impact on STM, particularly when an internally established rhythm is disrupted. Through two verbal STM tasks – the jabberwocky memory task and the Urdu memory task – conducted with thirty participants (31 females), involving the repetition of sentences in native English or foreign Urdu, this experiment investigates whether manipulating the speed of target sentences in relation to prime phrases affects the accuracy of sentence repetition in the assigned tasks. Our hypothesis posited that STM for sentences presented at altered speeds, either slower or faster, would be compromised compared to sentences at a normal pace. However, the outcomes of our study did not reveal any significant differences in repetition accuracy across the three speed conditions within the two STM tasks. Interestingly, our investigation uncovers two noteworthy findings. Firstly, variations in repetition accuracy among the three speed conditions appear to be influenced by participants' bilingual or multilingual backgrounds. Bilingual and multilingual individuals exhibited better performance under slow speed conditions at the syllable level, while monolingual participants displayed enhanced recall accuracy for whole words in the normal speed condition. Secondly, a discrepancy emerges between participants' self-perceived performance across the three speech conditions and their actual performance. These findings emphasize the potential roles of linguistic background and metacognition in shaping both temporal representation and STM performance, thereby prompting further exploration of these intricate interactions. / Thesis / Master of Science (MSc) / Our ability to remember spoken language, or phonological content, is closely tied to how we perceive its rhythm and timing. When we hear speech, our brain processes not just the words but also the temporal structure of the context. This temporal organization helps to retain spoken information, and our ability to remember speech in turn predicts how well we can learn new words in a foreign language. This suggests that the rhythmic patterns of words and their sentence contexts might play a crucial role in how we organize and remember linguistic content. The brain’s rhythms naturally synchronize with the rhythms of speech, influencing comprehension. However, when this synchronization is disrupted – for instance, when a speaker changes their pace suddenly – comprehension may be impeded. On the other hand, while studies have shown that memory for digits is not affected by their presentations at different rhythmic regularities, the way the brain processes disruption of internally driven rhythms might differ from how it processes externally driven rhythms. This thesis explores these rhythmic influences on memory by conducting tasks involving repeating sentences in native English and the foreign Urdu language to see how sudden changes in rhythm impact memory. This sheds light on the mechanisms by which the brain handles different time-related aspects of language and how this affects the memory retention, which ultimately shapes language skills and learning abilities.

This thesis explores the treatment of temporal information in Automated Speech Recognition. It reviews the study of time in speech perception and concludes that while some temporal information in the speech signal is of crucial value in the speech decoding process not all temporal information is relevant to decoding. We then review the representation of temporal information in the main automated recognition techniques: Hidden Markov Models and Artificial Neural Networks. We find that both techniques have difficulty representing the type of temporal information that is phonetically or phonologically significant in the speech signal. In an attempt to improve this situation we explore the problem of representation of temporal information in the acoustic vectors commonly used to encode the speech acoustic signal in the front-ends of speech recognition systems. We attempt, where possible, to let the signal provide the temporal structure rather than imposing a fixed, clock-based timing framework. We develop a novel acoustic temporal parameter (the Parameter Similarity Length), a measure of temporal stability, that is tested against the time derivatives of acoustic parameters conventionally used in acoustic vectors.

This report elucidates the linguistic representation of temporal relations among events. This involves examining sentences that contain two clauses connected by words like once, by the time, when, and before. Specifically, the effect of the tenses of the connected clauses on the acceptability of sentences are examined. For example, Rachel disappeared once Jon had fallen asleep is fine, but *Rachel had disappeared once Jon fell asleep is unacceptable. A theory of acceptability is developed and its implications for interpretation discussed. Factoring of the linguisitic knowledge into a general, syntactic component and a lexical component clarifies the interpretation problem. Finally, a computer model of the theory is demonstrated.

Automatic recognition of human emotion in speech aims at recognizing the underlying emotional state of a speaker from the speech signal. The area has received rapidly increasing research interest over the past few years. However, designing powerful spectral features for high-performance speech emotion recognition (SER) remains an open challenge. Most spectral features employed in current SER techniques convey short-term spectral properties only while omitting useful long-term temporal modulation information. In this thesis, modulation spectral features (MSFs) are proposed for SER, with support vector machines used for machine learning. By employing an auditory filterbank and a modulation filterbank for speech analysis, an auditory-inspired long-term spectro-temporal (ST) representation is obtained, which captures both acoustic frequency and temporal modulation frequency components. The MSFs are then extracted from the ST representation, thereby conveying information important for human speech perception but missing from conventional short-term spectral features (STSFs). Experiments show that the proposed features outperform features based on mel-frequency cepstral coefficients and perceptual linear predictive coefficients, two commonly used STSFs. The MSFs further render a substantial improvement in recognition performance when used to augment the extensively used prosodic features, and recognition accuracy above 90% is accomplished for classifying seven emotion categories. Moreover, the proposed features in combination with prosodic features attain estimation performance comparable to human evaluation for recognizing continuous emotions. / Thesis (Master, Electrical & Computer Engineering) -- Queen's University, 2009-09-08 13:01:54.941

Facial expressions convey a wealth of information about our feelings, personality and mental state. In this thesis we seek efficient ways of representing and analysing facial expressions of varying intensities. Firstly, we analyse state-of-the-art systems by decomposing them into their fundamental components, in an effort to understand what are the useful practices common to successful systems. Secondly, we address the problem of sequence registration, which emerged as an open issue in our analysis. The encoding of the (non-rigid) motions generated by facial expressions is facilitated when the rigid motions caused by irrelevant factors, such as camera movement, are eliminated. We propose a sequence registration framework that is based on pre-trained regressors of Gabor motion energy. Comprehensive experiments show that the proposed method achieves very high registration accuracy even under difficult illumination variations. Finally, we propose an unsupervised representation learning framework for encoding the spatio-temporal evolution of facial expressions. The proposed framework is inspired by the Facial Action Coding System (FACS), which predates computer-based analysis. FACS encodes an expression in terms of localised facial movements and assigns an intensity score for each movement. The framework we propose mimics those two properties of FACS. Specifically, we propose to learn from data a linear transformation that approximates the facial expression variation in a sequence as a weighted sum of localised basis functions, where the weight of each basis function relates to movement intensity. We show that the proposed framework provides a plausible description of facial expressions, and leads to state-of-the-art performance in recognising expressions across intensities; from fully blown expressions to micro-expressions.

La recherche développée dans cette thèse introduit une approche qualitative pour représenter et raisonner à partir d'entités spatiales dans un espace géographique à deux dimensions. Les patrons de mouvements entre entités dynamiques sont catégorisés à partir d'un modèle qualitatif de relations topologiques entre une ligne orientée et une région, et de relations d'orientation entre deux lignes orientées, respectivement. Les mouvements qualitatifs sont dérivés à partir de relations spatio-temporelles qui caractérisent des entités dynamiques conceptualisées comme des points ou des régions dans un espace à deux dimensions. Cette architecture de raisonnement permet de dériver des configurations de mouvements basiques dérivées à partir d'entités statiques et dynamiques. L'approche est complétée par une qualification de ces configurations à partir d'expressions du langage naturel. Les compositions de mouvements sont étudiées tout comme les transitions possibles dans des cas de données incomplètes. Les tables de compositions sont également explorées et permettent d'étendre les possibilités de raisonnement. Le modèle est expérimenté dans le contexte de l'analyse de trajectoires aériennes et maritimes. / The research developed in this thesis introduces a qualitative approach for representing and reasoning on moving entities in a two-dimensional geographical space. Movement patterns of moving entities are categorized based on a series of qualitative spatial models of topological relations between a directed line and a region, and orientation relations between two directed lines, respectively. Qualitative movements are derived from the spatio-temporal relations that characterize moving entities conceptualized as either points or regions in a two-dimensional space. Such a spatio-temporal framework supports the derivation of the basic movement configurations inferred from moving and static entities. The approach is complemented by a tentative qualification of the possible natural language expressions of the primitive movements identified. Complex movements can be represented by a composition of these primitive movements. The notion of conceptual transition that favors the exploration of possible trajectories in the case of incomplete knowledge configurations is introduced and explored.Composition tables are also studied and provide additional reasoning capabilities. The whole approach is applied to the analysis of flight patterns and maritime trajectories.

O presente trabalho propõe um novo modelo de rede neural artificial voltado a aplicações robóticas, em especial a tarefas de natureza espaço-temporal e de horizonte infinito. Este modelo apresenta três características que o tornam único e que foram tomadas como guia para a sua concepção: auto-organização, representação temporal e aprendizado construtivo. O algoritmo de aprendizagem auto-organizada incorpora todos os mecanismos que são básicos para a auto-organização: competição global, cooperação local e auto-amplificação seletiva. A rede neural é suprida com propriedades dinâmicas através de uma memória de curto prazo. A memória de curto prazo é inserida na estrutura da rede por meio de integradores e diferenciadores, os quais são implementados na camada de entrada da rede. Nesta abordagem existe uma evidente separação de papéis: a rede é responsável pela não-linearidade e a memória é responsável pelo tempo. A construção automática da arquitetura da rede neural é realizada de acordo com uma unidade de habituação. A unidade de habituação regula o crescimento e a poda de neurônios. O procedimento de inclusão, adaptação e remoção de conexões sinápticas é realizado conforme o método de aprendizado hebbiano competitivo. Em muitos problemas práticos, como os existentes na área da robótica, a auto-organização, a representação temporal e o aprendizado construtivo são fatores imprescindíveis para o sucesso da tarefa. A grande dificuldade e, ao mesmo tempo, a principal contribuição deste trabalho consiste em integrar tais tecnologias em uma arquitetura de rede neural artificial de maneira eficiente. Estudos de caso foram elaborados para validar e, principalmente, determinar as potencialidades e as limitações do modelo neural proposto. Os cenários abrangeram tarefas simples de classificação de padrões e segmentação temporal. Os resultados preliminares obtidos demonstraram a eficiência do modelo neural proposto frente às arquiteturas conexionistas existentes e foram considerados bastante satisfatórios com relação aos parâmetros avaliados. No texto são apresentados, também, alguns aspectos teóricos das ciências cognitivas, os fundamentos de redes neurais artificiais, o detalhamento de uma ferramenta de simulação robótica, conclusões, limitações e possíveis trabalhos futuros. / The present work proposes a new artificial neural network model suitable for robotic applications, in special to spatiotemporal tasks and infinite horizon tasks. This model has three characteristics which make it unique and are taken as means to guide its conception: self-organization, temporal representation and constructive learning. The algorithm of self-organizing learning incorporates all the mechanisms that are basic to the self-organization: global competition, local cooperation and selective self-amplification. The neural network is supplied with dynamic properties through a short-term memory. The short-term memory is added in the network structure by means of integrators and differentiators, which are implemented in the input layer of the network. In this approach exists an evident separation of roles: the network is responsible for the non-linearity and the memory is responsible for the time. The automatic construction of the neural network architecture is carried out taking into account habituation units. The habituation unit regulates the growing and the pruning of neurons. The procedure of inclusion, adaptation and removal of synaptic connections is carried out in accordance with competitive hebbian learning technique. In many practical problems, as the ones in the robotic area, self-organization, temporal representation and constructive learning are essential factors to the success of the task. The great difficulty and, at the same time, the main contribution of this work consists in the integration of these technologies in a neural network architecture in an efficient way. Some case studies have been elaborated to validate and, mainly, to determine the potentialities and the limitations of the proposed neural model. The experiments comprised simple tasks of pattern classification and temporal segmentation. Preliminary results have shown the good efficiency of the neural model compared to existing connectionist architectures and they have been considered sufficiently satisfactory with regard to the evaluated parameters. This text also presents some theoretical aspects of the cognitive science area, the fundamentals of artificial neural networks, the details of a robotic simulation tool, the conclusions, limitations and possible future works.

O presente trabalho propõe um novo modelo de rede neural artificial voltado a aplicações robóticas, em especial a tarefas de natureza espaço-temporal e de horizonte infinito. Este modelo apresenta três características que o tornam único e que foram tomadas como guia para a sua concepção: auto-organização, representação temporal e aprendizado construtivo. O algoritmo de aprendizagem auto-organizada incorpora todos os mecanismos que são básicos para a auto-organização: competição global, cooperação local e auto-amplificação seletiva. A rede neural é suprida com propriedades dinâmicas através de uma memória de curto prazo. A memória de curto prazo é inserida na estrutura da rede por meio de integradores e diferenciadores, os quais são implementados na camada de entrada da rede. Nesta abordagem existe uma evidente separação de papéis: a rede é responsável pela não-linearidade e a memória é responsável pelo tempo. A construção automática da arquitetura da rede neural é realizada de acordo com uma unidade de habituação. A unidade de habituação regula o crescimento e a poda de neurônios. O procedimento de inclusão, adaptação e remoção de conexões sinápticas é realizado conforme o método de aprendizado hebbiano competitivo. Em muitos problemas práticos, como os existentes na área da robótica, a auto-organização, a representação temporal e o aprendizado construtivo são fatores imprescindíveis para o sucesso da tarefa. A grande dificuldade e, ao mesmo tempo, a principal contribuição deste trabalho consiste em integrar tais tecnologias em uma arquitetura de rede neural artificial de maneira eficiente. Estudos de caso foram elaborados para validar e, principalmente, determinar as potencialidades e as limitações do modelo neural proposto. Os cenários abrangeram tarefas simples de classificação de padrões e segmentação temporal. Os resultados preliminares obtidos demonstraram a eficiência do modelo neural proposto frente às arquiteturas conexionistas existentes e foram considerados bastante satisfatórios com relação aos parâmetros avaliados. No texto são apresentados, também, alguns aspectos teóricos das ciências cognitivas, os fundamentos de redes neurais artificiais, o detalhamento de uma ferramenta de simulação robótica, conclusões, limitações e possíveis trabalhos futuros. / The present work proposes a new artificial neural network model suitable for robotic applications, in special to spatiotemporal tasks and infinite horizon tasks. This model has three characteristics which make it unique and are taken as means to guide its conception: self-organization, temporal representation and constructive learning. The algorithm of self-organizing learning incorporates all the mechanisms that are basic to the self-organization: global competition, local cooperation and selective self-amplification. The neural network is supplied with dynamic properties through a short-term memory. The short-term memory is added in the network structure by means of integrators and differentiators, which are implemented in the input layer of the network. In this approach exists an evident separation of roles: the network is responsible for the non-linearity and the memory is responsible for the time. The automatic construction of the neural network architecture is carried out taking into account habituation units. The habituation unit regulates the growing and the pruning of neurons. The procedure of inclusion, adaptation and removal of synaptic connections is carried out in accordance with competitive hebbian learning technique. In many practical problems, as the ones in the robotic area, self-organization, temporal representation and constructive learning are essential factors to the success of the task. The great difficulty and, at the same time, the main contribution of this work consists in the integration of these technologies in a neural network architecture in an efficient way. Some case studies have been elaborated to validate and, mainly, to determine the potentialities and the limitations of the proposed neural model. The experiments comprised simple tasks of pattern classification and temporal segmentation. Preliminary results have shown the good efficiency of the neural model compared to existing connectionist architectures and they have been considered sufficiently satisfactory with regard to the evaluated parameters. This text also presents some theoretical aspects of the cognitive science area, the fundamentals of artificial neural networks, the details of a robotic simulation tool, the conclusions, limitations and possible future works.

O presente trabalho propõe um novo modelo de rede neural artificial voltado a aplicações robóticas, em especial a tarefas de natureza espaço-temporal e de horizonte infinito. Este modelo apresenta três características que o tornam único e que foram tomadas como guia para a sua concepção: auto-organização, representação temporal e aprendizado construtivo. O algoritmo de aprendizagem auto-organizada incorpora todos os mecanismos que são básicos para a auto-organização: competição global, cooperação local e auto-amplificação seletiva. A rede neural é suprida com propriedades dinâmicas através de uma memória de curto prazo. A memória de curto prazo é inserida na estrutura da rede por meio de integradores e diferenciadores, os quais são implementados na camada de entrada da rede. Nesta abordagem existe uma evidente separação de papéis: a rede é responsável pela não-linearidade e a memória é responsável pelo tempo. A construção automática da arquitetura da rede neural é realizada de acordo com uma unidade de habituação. A unidade de habituação regula o crescimento e a poda de neurônios. O procedimento de inclusão, adaptação e remoção de conexões sinápticas é realizado conforme o método de aprendizado hebbiano competitivo. Em muitos problemas práticos, como os existentes na área da robótica, a auto-organização, a representação temporal e o aprendizado construtivo são fatores imprescindíveis para o sucesso da tarefa. A grande dificuldade e, ao mesmo tempo, a principal contribuição deste trabalho consiste em integrar tais tecnologias em uma arquitetura de rede neural artificial de maneira eficiente. Estudos de caso foram elaborados para validar e, principalmente, determinar as potencialidades e as limitações do modelo neural proposto. Os cenários abrangeram tarefas simples de classificação de padrões e segmentação temporal. Os resultados preliminares obtidos demonstraram a eficiência do modelo neural proposto frente às arquiteturas conexionistas existentes e foram considerados bastante satisfatórios com relação aos parâmetros avaliados. No texto são apresentados, também, alguns aspectos teóricos das ciências cognitivas, os fundamentos de redes neurais artificiais, o detalhamento de uma ferramenta de simulação robótica, conclusões, limitações e possíveis trabalhos futuros. / The present work proposes a new artificial neural network model suitable for robotic applications, in special to spatiotemporal tasks and infinite horizon tasks. This model has three characteristics which make it unique and are taken as means to guide its conception: self-organization, temporal representation and constructive learning. The algorithm of self-organizing learning incorporates all the mechanisms that are basic to the self-organization: global competition, local cooperation and selective self-amplification. The neural network is supplied with dynamic properties through a short-term memory. The short-term memory is added in the network structure by means of integrators and differentiators, which are implemented in the input layer of the network. In this approach exists an evident separation of roles: the network is responsible for the non-linearity and the memory is responsible for the time. The automatic construction of the neural network architecture is carried out taking into account habituation units. The habituation unit regulates the growing and the pruning of neurons. The procedure of inclusion, adaptation and removal of synaptic connections is carried out in accordance with competitive hebbian learning technique. In many practical problems, as the ones in the robotic area, self-organization, temporal representation and constructive learning are essential factors to the success of the task. The great difficulty and, at the same time, the main contribution of this work consists in the integration of these technologies in a neural network architecture in an efficient way. Some case studies have been elaborated to validate and, mainly, to determine the potentialities and the limitations of the proposed neural model. The experiments comprised simple tasks of pattern classification and temporal segmentation. Preliminary results have shown the good efficiency of the neural model compared to existing connectionist architectures and they have been considered sufficiently satisfactory with regard to the evaluated parameters. This text also presents some theoretical aspects of the cognitive science area, the fundamentals of artificial neural networks, the details of a robotic simulation tool, the conclusions, limitations and possible future works.

Nous présentons dans cette thèse un nouveau paradigme pour caractériser la maladie d’Alzheimer à travers l’écriture manuscrite acquise sur tablette graphique. L’état de l’art est dominé par des méthodes qui supposent un comportement unique ou homogène au sein de chaque profil cognitif. Ces travaux exploitent des paramètres cinématiques globaux, sur lesquels ils appliquent des tests statistiques ou des algorithmes de classification pour discriminer les différents profils cognitifs (les patients Alzheimer, les troubles cognitifs légers (« Mild Cognitive impairment » : MCI) et les sujets Contrôle (HC)). Notre travail aborde ces deux limites de la littérature de la façon suivante : premièrement au lieu de considérer un comportement homogène au sein de chaque profil cognitif ou classe (HC, MCI, ES-AD : « Early-Stage Alzheimer Disease »), nous nous sommes affranchis de cette hypothèse (ou contrainte) forte de la littérature. Nous considérons qu’il peut y avoir plusieurs comportements au sein de chaque profil cognitif. Ainsi, nous proposons un apprentissage semi-supervisé pour trouver des groupes homogènes de sujets et analysons l’information contenue dans ces clusters ou groupes sur les profils cognitifs. Deuxièmement, au lieu d’exploiter les paramètres cinématiques globaux (ex : vitesse moyenne, pression moyenne, etc.), nous avons défini deux paramétrisations ou codages : une paramétrisation semi-globale, puis locale en modélisant la dynamique complète de chaque paramètre. L’un de nos résultats importants met en évidence deux clusters majeurs qui sont découverts, l’un dominé par les sujets HC et MCI et l’autre par les MCI et ES-AD, révélant ainsi que les patients atteints de MCI ont une motricité fine qui est proche soit des sujets HC, soit des patients ES-AD. Notre travail montre également que la vitesse prise localement regroupe un ensemble riche des caractéristiques telles que la taille, l’inclinaison, la fluidité et la régularité, et révèle comment ces paramètres spatiotemporels peuvent conjointement caractériser les profils cognitifs. / We present, in this thesis, a novel paradigm for assessing Alzheimer’s disease by analyzing impairment of handwriting (HW) on tablets, a challenging problem that is still in its infancy. The state of the art is dominated by methods that assume a unique behavioral trend for each cognitive profile, and that extract global kinematic parameters, assessed by standard statistical tests or classification models, for discriminating the neuropathological disorders (Alzheimer’s (AD), Mild Cognitive Impairment (MCI)) from Healthy Controls (HC). Our work tackles these two major limitations as follows. First, instead of considering a unique behavioral pattern for each cognitive profile, we relax this heavy constraint by allowing the emergence of multimodal behavioral patterns. We achieve this by performing semi-supervised learning to uncover homogeneous clusters of subjects, and then we analyze how much information these clusters carry on the cognitive profiles. Second, instead of relying on global kinematic parameters, mostly consisting of their average, we refine the encoding either by a semi-global parameterization, or by modeling the full dynamics of each parameter, harnessing thereby the rich temporal information inherently characterizing online HW. Thanks to our modeling, we obtain new findings that are the first of their kind on this research field. A striking finding is revealed: two major clusters are unveiled, one dominated by HC and MCI subjects, and one by MCI and ES-AD, thus revealing that MCI patients have fine motor skills leaning towards either HC’s or ES-AD’s. This thesis introduces also a new finding from HW trajectories that uncovers a rich set of features simultaneously like the full velocity profile, size and slant, fluidity, and shakiness, and reveals, in a naturally explainable way, how these HW features conjointly characterize, with fine and subtle details, the cognitive profiles.