  Issues of safety, explainability, and efficiency are of increasing concern in
learning systems deployed with hard and soft constraints. Symbolic Constrained
Learning and Knowledge Distillation techniques have shown promising results in
this area, by embedding and extracting knowledge, as well as providing logical
constraints during neural network training. Although many frameworks exist to
date, through an integration of logic and information geometry, we provide a
construction and theoretical framework for these tasks that generalize many
approaches. We propose a loss-based method that embeds knowledge-enforces
logical constraints-into a machine learning model that outputs probability
distributions. This is done by constructing a distribution from the external
knowledge/logic formula and constructing a loss function as a linear
combination of the original loss function with the Fisher-Rao distance or
Kullback-Leibler divergence to the constraint distribution. This construction
includes logical constraints in the form of propositional formulas (Boolean
variables), formulas of a first-order language with finite variables over a
model with compact domain (categorical and continuous variables), and in
general, likely applicable to any statistical model that was pretrained with
semantic information. We evaluate our method on a variety of learning tasks,
including classification tasks with logic constraints, transferring knowledge
from logic formulas, and knowledge distillation from general distributions.
