  Humans learn to recognize and manipulate new objects in lifelong settings
without forgetting the previously gained knowledge under non-stationary and
sequential conditions. In autonomous systems, the agents also need to mitigate
similar behavior to continually learn the new object categories and adapt to
new environments. In most conventional deep neural networks, this is not
possible due to the problem of catastrophic forgetting, where the newly gained
knowledge overwrites existing representations. Furthermore, most
state-of-the-art models excel either in recognizing the objects or in grasp
prediction, while both tasks use visual input. The combined architecture to
tackle both tasks is very limited. In this paper, we proposed a hybrid model
architecture consists of a dynamically growing dual-memory recurrent neural
network (GDM) and an autoencoder to tackle object recognition and grasping
simultaneously. The autoencoder network is responsible to extract a compact
representation for a given object, which serves as input for the GDM learning,
and is responsible to predict pixel-wise antipodal grasp configurations. The
GDM part is designed to recognize the object in both instances and categories
levels. We address the problem of catastrophic forgetting using the intrinsic
memory replay, where the episodic memory periodically replays the neural
activation trajectories in the absence of external sensory information. To
extensively evaluate the proposed model in a lifelong setting, we generate a
synthetic dataset due to lack of sequential 3D objects dataset. Experiment
results demonstrated that the proposed model can learn both object
representation and grasping simultaneously in continual learning scenarios.
