  Topic models such as LDA, DocNADE, iDocNADEe have been popular in document
analysis. However, the traditional topic models have several limitations
including: (1) Bag-of-words (BoW) assumption, where they ignore word ordering,
(2) Data sparsity, where the application of topic models is challenging due to
limited word co-occurrences, leading to incoherent topics and (3) No Continuous
Learning framework for topic learning in lifelong fashion, exploiting
historical knowledge (or latent topics) and minimizing catastrophic forgetting.
This thesis focuses on addressing the above challenges within neural topic
modeling framework. We propose: (1) Contextualized topic model that combines a
topic and a language model and introduces linguistic structures (such as word
ordering, syntactic and semantic features, etc.) in topic modeling, (2) A novel
lifelong learning mechanism into neural topic modeling framework to demonstrate
continuous learning in sequential document collections and minimizing
catastrophic forgetting. Additionally, we perform a selective data augmentation
to alleviate the need for complete historical corpora during data hallucination
or replay.
