  Data format innovations have been critical for machine learning (ML) scaling,
which in turn fuels ground-breaking ML capabilities. However, even in the
presence of low-precision formats, model weights are often stored in both
high-precision and low-precision during training. Furthermore, with emerging
directional data formats (e.g., MX9, MX6, etc.) multiple low-precision weight
copies can be required. To lower memory capacity needs of weights, we explore
just-in-time quantization (JIT-Q) where we only store high-precision weights in
memory and generate low-precision weights only when needed. To perform JIT-Q
efficiently, in this work, we evaluate emerging processing-in-memory (PIM)
technology to execute quantization. With PIM, we can offload quantization to
in-memory compute units enabling quantization to be performed without incurring
costly data movement while allowing quantization to be concurrent with
accelerator computation. Our proposed PIM-offloaded quantization keeps up with
GPU compute and delivers considerable capacity savings (up to 24\%) at marginal
throughput loss (up to 2.4\%). Said memory capacity savings can unlock several
benefits such as fitting larger model in the same system, reducing model
parallelism requirement, and improving overall ML training efficiency.
