In memories of his youth, Williams cheerfully recalls frolicking with white playmates, running with family dogs, and climbing fig trees. While Williams' own memories with his "playmates" are positive, it was not uncommon for enslaved children to be used as playmates or entertainment for white children until their enslavers determined their utility shift directly to labor. Moreover, even his most positive memories are troubling in their implications. For example, Williams remembers the excitement of riding horses as a child jockey, while entertaining white audiences at great risk:
My mother and her children fell to the lot of Edward Dane, brother of Thomas. This young gentleman was of a gay disposition; fond of horses and the sports of the day. . . He taught me to ride, and when I could sit my horse well "bare-back" he had a saddle made for me at the then famous "McKinzie's" saddlery, sign of the "White Horse at the corner of Church and Chalmers street. (Gentlemen had their saddles made to order in those days). I would often accompany him "up the road" on horseback to the Clubhouse, there to exhibit my youthful feats of horsemanship, for the divertissement of Mr. Dane and his friends.
Horse racing was a central part of American sporting culture in the nineteenth century. Charleston, in particular, served as a hub in the antebellum period, and the city’s annual Race Week attracted citizens from all over the country. Williams would have been familiar with the festivities, and he was convinced that Edward Dane, his enslaver, was training him to ride with the hopes of using young Williams as a jockey, only to have the plans dashed by the onset of the Civil War and a cessation to such entertainments.
Being an athlete held in bondage might have seemed an exciting opportunity for a young boy, as opposed to many other fates that could have awaited him, but all enslaved people, from those who labored in the fields to those trained as jockeys, were still subject to brutal treatment and physical risk. In his memoir, though, Williams recounts feeling a sense of satisfaction with his childish athleticism, and he takes pride in his exploits as a horseman, proclaiming that he and his horse Agile “never had a mishap.”