  Human-like planning skills and dexterous manipulation have long posed
challenges in the fields of robotics and artificial intelligence (AI). The task
of reinterpreting calligraphy presents a formidable challenge, as it involves
the decomposition of strokes and dexterous utensil control. Previous efforts
have primarily focused on supervised learning of a single instrument, limiting
the performance of robots in the realm of cross-domain text replication. To
address these challenges, we propose CalliRewrite: a coarse-to-fine approach
for robot arms to discover and recover plausible writing orders from diverse
calligraphy images without requiring labeled demonstrations. Our model achieves
fine-grained control of various writing utensils. Specifically, an unsupervised
image-to-sequence model decomposes a given calligraphy glyph to obtain a coarse
stroke sequence. Using an RL algorithm, a simulated brush is fine-tuned to
generate stylized trajectories for robotic arm control. Evaluation in
simulation and physical robot scenarios reveals that our method successfully
replicates unseen fonts and styles while achieving integrity in unknown
characters.
