My first breakfast in Singapore, when I moved there many years ago, was sticky, chartreuse jam slathered over thin crisps of toasted brown bread, served with barely cooked eggs and a small cup of coffee laced with sugar and loaded with sweetened condensed milk. I moved slowly in the morning's equatorial humidity and approached this iconic cuisine, which was new to me, with apprehension. It was not exactly love at first sight, and yet a passion was born.

View Images Fragrant, earthy, and sugary, kaya jam is the star of a stack of toast.

Photograph by VIJAYNATHAN KATHANATHAN AND CHANG PICK YIN

The basis of a classic Singaporean breakfast is kaya, a custard of coconut milk, eggs, and sugar, flavored with pandan leaf, which gives the jam the perfume of freshly cut grass. In the Malay language, kaya means “rich.” But the richness doesn’t end with the jam. It’s served with barely boiled eggs, cracked into a shallow dish and seasoned with soy sauce and white pepper. Coffee, made from beans sautéed in margarine and sugar, is sweetened to the max. You can order an iced Milo, a chocolate malt drink, for additional hydration. The whole meal—order it as “kaya toast”—is a staple in kopitiams (kopi is Malay for “coffee”; tiam means “shop” in Hokkien) and will set you back about $3.

It's tempting to say that eggs are eggs—that the world's breakfasts are unremarkable meals, not worthy of emotional response (much less critique)—but I came to see that kaya toast was the perfect vehicle for exploring my unfamiliar surroundings. While the ubiquitous meal is served at almost every hawker center, I had the personally meaningful perception that the experience of eating it is as much about communities and food cultures as it is about the food itself. (Discover more top experiences in Singapore.)

Singaporeans are proud of local success stories, so the Ya Kun Kaya Toast chain was an obvious place to start. Named for an industrious Hainanese immigrant who landed here in 1926, worked in a coffee stall, and eventually founded his own, it’s now an institution known for thin-sliced toast, fragrant jam, and a warm-spirited connection to its heritage.

Old kopitiams in Singapore are becoming scarce; rarer still is the communal feeling they nourish. Tong Ah Eating House is situated in the middle of a row of shophouses on a street that was formerly a red-light district. The space feels like a bingo parlor, with stackable plastic chairs and ceiling fans. Eggs bobble in a tepid bath next to the entrance. But the offerings here are a revelation: extra-thin and crispy slices, double-toasted, scraped to remove bitter char, with homemade kaya jam less sweet—and slabs of butter more abundant—than at any other coffee shop. You can even order French toast kaya, if healthy living is of no concern to you. Regulars consider it damn shiok, lah (an extreme pleasure to eat).

Kaya toast began to influence my travels. One weekend I visited George Town, a UNESCO World Heritage city on the Malaysian island of Penang. Chinese temples, Peranakan mansions, colonial structures, and trompe l’oeil street murals are the big draw for most visitors. I came for the kaya, and it did not disappoint. My friend Antoinette Chia Yen Yen, who is from Sarawak but is always up for an adventure, joined me on the visit and guided me into the labyrinths of the old city to Toh Soon Cafe, an open-air kaya kitchen operating in an alleyway, shaded by tarps hanging overhead. This was the real deal: men squatting down to toast bread over a charcoal fire inside steel oil drums. A dozen plastic tables crammed into the alley, and the aroma of kaya hovered like a genie over the bustling scene. Here the toast was sliced into dunking strips and the runny eggs served in cups. The jam was so fresh I ate three helpings and ordered another tapow (to go).

View Images A beloved Singaporean kopitiam, Heap Seng Leong, serves a classic breakfast of coffee and kaya toast. Photograph by NICHOLAS YEO

After more than three years of obsessing over breakfast, I reached the apotheosis of my kaya quest. A search for the oldest kopitiam in Singapore led me to Heap Seng Leong, a flashback to a world of “uncles” in pajama pants, milk-can ashtrays, and old men lingering over newspapers as the day turns from balmy to incendiary. Decades of dietary fads have gone unnoticed at this kopitiam, which specializes in kopi gu you—coffee with an oil slick of butter on top. The taste is just what you’d expect: black coffee plus butter. There’s a reason you don’t do this at home. The most amazing thing I saw here was the ancient proprietor hand-slicing a loaf of bread the size of a cocker spaniel. It was not the best kaya toast, but the improbable fact that this mid-century holdover is in business at all is astonishing.

When friends visited me, the first thing I would do is whisk them off to Tong Ah. I told myself I was showing them a Singaporean secret. But I was also revealing a bit about myself, and that’s the point of obsessions.

My passion for kaya—a food item my father found so inscrutable he put it on ice cream when I brought it home to Toledo, Ohio—has nothing to do with jam and everything to do with my love of Singapore and the surrounding region. With kaya as my guide, I explored the nooks and crannies of food cultures, and along the way I discovered how to disappear into a faraway place and come away with a rich (sometimes overly rich) experience.

Best traditional breakfast spots in Singapore

Tong Ah Eating House

Local kaya-philes love the extra-crispy toast served at this iconic kopitiam located on a street lined with old shophouses. Breakfast is not the only specialty; dinner features home-style dishes.

Heap Seng Leong

Entering this kopitiam is “like stepping into a time portal,” writes Leslie Tay, the Singaporean behind a local food blog. “We need places like this so that our kids know where we came from and what it was like in the past.”

Keng Wah Sung

For deliciously messy breakfasts served on weathered tables try this old-school kopitiam, in the food-centric Geylang neighborhood, which specializes in toasted buns topped with Hainanese-style custardy kaya jam.

The editor in chief of National Geographic Traveler magazine, George Stone called Singapore home for three years. Follow his travels on Twitter @travelerstone and Instagram @georgewstone.