  Optical satellite-to-ground communication (OSGC) has the potential to improve
access to fast and affordable Internet in remote regions. Atmospheric
turbulence, however, distorts the optical beam, eroding the data rate potential
when coupling into single-mode fibers. Traditional adaptive optics (AO) systems
use a wavefront sensor to improve fiber coupling. This leads to higher system
size, cost and complexity, consumes a fraction of the incident beam and
introduces latency, making OSGC for internet service impractical. We propose
the use of reinforcement learning (RL) to reduce the latency, size and cost of
the system by up to $30-40\%$ by learning a control policy through interactions
with a low-cost quadrant photodiode rather than a wavefront phase profiling
camera. We develop and share an AO RL environment that provides a standardized
platform to develop and evaluate RL based on the Strehl ratio, which is
correlated to fiber-coupling performance. Our empirical analysis finds that
Proximal Policy Optimization (PPO) outperforms Soft-Actor-Critic and Deep
Deterministic Policy Gradient. PPO converges to within $86\%$ of the maximum
reward obtained by an idealized Shack-Hartmann sensor after training of 250
episodes, indicating the potential of RL to enable efficient wavefront
sensorless OSGC.
