The Summer I Rubbed Elbows with American Royalty at Saratoga

Two afternoons, two dinners, and an unforgettable encounter with Alfred Gwynne Vanderbilt Jr., heir to one of America's greatest fortunes.

For two extraordinary afternoons during the 1990s, I found myself sitting in one of the most prestigious private boxes at Saratoga Race Course, directly on the finish line, in the company of Alfred Gwynne Vanderbilt Jr. We spent hours watching the races, sharing stories, and discussing his remarkable life. Both afternoons were followed by dinner at his favorite restaurant.

For a brief moment in my life, I had entered the world of America's old-money aristocracy, surrounded by people whose wealth, influence, and social standing were almost unimaginable to an ordinary fellow like me.

How did I manage to find myself in such extraordinary company? As it turned out, it had nothing to do with my wealth, accomplishments, or family connections. The explanation was considerably more amusing, but I'll get to that shortly.

Just Who Was Alfred Gwynne Vanderbilt Jr.?

To appreciate how extraordinary this experience was, you need to understand who Alfred Gwynne Vanderbilt Jr. was.

His great-great-grandfather was Commodore Cornelius Vanderbilt, the shipping and railroad magnate who amassed one of the greatest fortunes in American history. The Vanderbilt name became synonymous with magnificent mansions, enormous wealth, exclusive social circles, and a lifestyle beyond the imagination of ordinary Americans. Vanderbilt University in Nashville, Tennessee, bears the family name, having been founded in 1873 through a $1 million gift from the Commodore.

Alfred was born into this extraordinary world in 1912. His father died aboard the Lusitania when Alfred was only two and a half years old, leaving him an heir to millions. A 1939 TIME magazine article described him as heir to an estimated $20 million, an astonishing fortune during the Great Depression.

But Mr. Vanderbilt was much more than a wealthy heir. While still in his twenties, he held leadership positions at some of America's most prestigious racetracks, including Pimlico and Belmont Park. He helped modernize horse racing through innovations such as mechanical starting gates and photo-finish technology, owned the legendary racehorse Native Dancer, and served as a decorated PT boat commander during World War II. His adventures even included hunting in Africa with Ernest Hemingway.

He was the very model of an American racing aristocrat, a man whose family name carried enormous weight in the country's most exclusive social circles. At Saratoga, he was practically royalty.

Saratoga: Where the Social Ladder Goes Downhill

Saratoga Race Course is unlike almost any other sporting venue in America. Thoroughbred racing began there in 1863, during the Civil War, making it one of the country's oldest operating racetracks.

Even today, Saratoga feels like a journey back in time. Its wooden grandstands, Victorian architecture, and long-established traditions recall an era when the Vanderbilts, Whitneys, Astors, and other great names of American society gathered each summer to socialize and watch their horses compete.

So how did I get invited into Mr. Vanderbilt's private box? Well, at the time, I was dating a particularly attractive woman who happened to be his dentist.

Mr. Vanderbilt, as everyone called him—nobody called him Alfred—was in his 80s, but age had apparently done little to diminish his appreciation for beautiful women. He was still quite flirtatious, and my girlfriend certainly caught his attention. He took a liking to her and invited us to spend an afternoon with him at Saratoga. Fortunately, the invitation included me, and we ultimately enjoyed two afternoons in his company.

Walking down the steps toward Mr. Vanderbilt's box was something like climbing the social ladder, except at Saratoga, the social ladder went downhill. The closer you got to the finish line, the greater the wealth, influence, and social standing of the people occupying the boxes. There were wealthy bankers, famous sports figures, powerful businesspeople, and members of families whose fortunes stretched back generations. And at the very front, occupying perhaps the most prestigious box of all, was Mr. Vanderbilt.

With every step I descended, I could almost feel my own social standing rising. I noticed people in the other boxes looking at us, and I couldn't help imagining what they might be thinking. "Who is this guy with this stunning woman sitting in Alfred Vanderbilt's box?"

For a moment in my life, I was the envy of these people—the wealthy bankers, famous sports figures, powerful businesspeople, and prominent families who occupied Saratoga's most exclusive boxes. My suddenly elevated social standing was entirely a matter of whose company I was keeping, and I was thoroughly enjoying the experience.

I had no famous family name or enormous fortune. I just happened to be dating the right dentist, and for two wonderful afternoons, that was apparently all the social standing I needed.

The Jacket Rule

There was a strict dress code in this exclusive section of Saratoga. Gentlemen were required to wear jackets and ties, even on a hot summer afternoon. But Mr. Vanderbilt explained that there was one exception: a gentleman could remove his jacket, but only after Mr. Vanderbilt removed his.

Think about that for a moment. Here was a man whose standing in Saratoga society was such that other gentlemen waited for him to take off his jacket before removing theirs. I had encountered plenty of dress codes in my life, but never one where the temperature was apparently determined by a Vanderbilt.

"Oh, I Never Bet"

During one of my first conversations with Mr. Vanderbilt, I asked what seemed like a perfectly natural question: "Mr. Vanderbilt, which horse did you bet on?"

"Oh, I never bet," he replied.

I was astonished. Here I was, sitting with one of the most influential men in American horse racing, a man who had owned some of the greatest thoroughbreds in history, and he was telling me he never bet on the horses.

For Mr. Vanderbilt, Saratoga wasn't about gambling. It was about the horses, the traditions, the people, and a sport he had loved for most of his life.

His most famous horse was Native Dancer, the legendary gray thoroughbred nicknamed the "Grey Ghost." Native Dancer won 21 of his 22 career starts, losing only the 1953 Kentucky Derby to Dark Star by a head. He subsequently won both the Preakness Stakes and Belmont Stakes and became one of the most celebrated horses in American racing history.

I found it rather amusing that a man who had spent much of his life owning racehorses and running racetracks never felt the need to place a wager. Apparently, when you own horses like Native Dancer, you don't need a two-dollar ticket to enjoy yourself.

Conversations with an American Legend

During both visits, we spent much of the afternoon with Mr. Vanderbilt, from the first race through the last. We had hours to talk, and our conversations were casual, unhurried, and often surprisingly personal.

He was impeccably dressed, gracious, friendly, and remarkably easy to talk with. He also suffered from severe glaucoma, which had robbed him of most of his eyesight. I would estimate that he had lost perhaps 90 percent of his vision.

What struck me most was how ordinary our conversations were. There was no pretension, no great ceremony, and no sense that we needed to impress him. We simply talked, and during those two afternoons he shared stories that connected me to some extraordinary chapters of American history.

"Oh, I Used to Live There"

At one point, I mentioned that I had visited the magnificent Vanderbilt Mansion overlooking the Hudson River in Hyde Park, New York. Today, it's a National Historic Site, preserved as a reminder of the extraordinary wealth and lifestyle of America's Gilded Age.

I had toured its magnificent rooms and grounds, marveling at the grandeur and imagining what life must have been like for the people who lived there. I thought Mr. Vanderbilt might be interested to hear about my visit.

"Oh, I used to live there," he replied casually.

That was it. No elaborate explanation, no attempt to impress me, just a simple observation. I had visited the mansion as a tourist, admiring a piece of American history, and here I was sitting next to a man who told me he had once called it home. To me, it was a national historic treasure. To Mr. Vanderbilt, it was simply another place he remembered living.

The Richest Little Boy in America

Another conversation turned to his father, Alfred Gwynne Vanderbilt Sr., who had been among the wealthiest men in America. His father lost his life aboard the RMS Lusitania when it was torpedoed by a German submarine in May 1915. The sinking killed nearly 1,200 people and became one of the most infamous maritime disasters in history.

Mr. Vanderbilt was only about two and a half years old when his father died. He told me that he inherited approximately $4 million and, as a result, became the richest little boy in America. Whether that distinction was technically correct, I couldn't say, but that was how he told the story.

Imagine sitting at the finish line of Saratoga Race Course, casually listening to a man describe how his father had died in one of the most famous maritime disasters of the twentieth century. For him, this wasn't a story from a history book. It was his family history.

Playing with Gloria Vanderbilt

Mr. Vanderbilt also told me about playing as a youngster with his cousin, Gloria Vanderbilt. Yes, that Gloria Vanderbilt—the famous heiress, artist, fashion designer, and entrepreneur who eventually made her own fortune selling designer jeans.

To most Americans, Gloria Vanderbilt was a celebrity whose name appeared in magazines and newspapers. To Mr. Vanderbilt, she was simply a cousin he remembered playing with when they were children.

I found these conversations fascinating. People I knew only through newspapers, television, and history books were people he had known personally, sometimes since childhood. Yet he spoke about them as casually as anyone might talk about relatives or old friends.

Mr. Vanderbilt's Secret Hideaway

During one of our conversations, we were discussing restaurants worth visiting around Saratoga when Mr. Vanderbilt shared what seemed like a closely guarded secret. He told us about a little out-of-the-way restaurant north of the racetrack where he used to rendezvous with Robyn Smith, a beautiful young female jockey who had attracted considerable attention in the racing world.

Robyn Smith was no ordinary jockey. In the early 1970s, when thoroughbred racing was almost exclusively a man's profession, she became one of America's pioneering female riders. She was attractive, talented, and fiercely competitive, eventually appearing on the cover of Sports Illustrated. She rode for Mr. Vanderbilt's racing stable and made history by winning a stakes race aboard one of his horses.

Mr. Vanderbilt and Robyn had also been romantically involved. He explained that he liked this particular restaurant because it was far enough removed from Saratoga's fashionable social circles that nobody recognized them. He told us the story almost conspiratorially, as though he were letting us in on one of his little secrets. I rather enjoyed the feeling of being taken into his confidence.

There was, however, an interesting twist to the Robyn Smith story that Mr. Vanderbilt didn't mention. He had introduced Robyn to his friend, the legendary dancer and actor Fred Astaire, at Santa Anita Race Track in California in 1973. Robyn and Astaire eventually fell in love and married in 1980, when she was 35 and he was 81.

I couldn't help wondering whether Robyn Smith might have been "the one that got away" for Mr. Vanderbilt. There was something about the way he shared his memories of their secret rendezvous that left me with the impression he still had a soft spot for her. Perhaps he even regretted having introduced her to the man she eventually married. Of course, he never said anything of the sort, and I could have been completely wrong. But that was the feeling I came away with.

Naturally, my girlfriend and I couldn't resist visiting Mr. Vanderbilt's secret restaurant ourselves. I've long since forgotten its name, but I remember exactly why we went. Perhaps we were curious, or perhaps we simply wanted to indulge our own little Vanderbilt fantasy after spending those extraordinary afternoons in his private box.

Of course, unlike Mr. Vanderbilt and Robyn Smith, we had no reason to worry about being recognized. Anonymity came quite naturally to us.

Our Walks Around Saratoga

There was a ritual during both afternoons with Mr. Vanderbilt that I particularly enjoyed. At some point during the day, we would leave his exclusive box at the finish line and take a leisurely walk around the back of the racetrack, where vendors had set up booths selling food, artwork, and other merchandise.

Because Mr. Vanderbilt's eyesight was so severely impaired, we helped escort him on these walks. It was a small but meaningful role, guiding a man who had spent his life at Saratoga through a place that was so familiar to him, yet increasingly difficult for him to see.

What struck me was the reaction of the vendors when they saw Mr. Vanderbilt approaching. Everyone seemed delighted to see him, greeting him warmly as though he were an old friend, but always respectfully addressing him as "Mr. Vanderbilt." He seemed just as happy to see them.

He would stop at the booths, chat with the vendors, and take an interest in whatever they were selling. There was nothing hurried or perfunctory about these visits. He appeared genuinely interested in the people and their merchandise.

One particular encounter has stayed with me all these years. A vendor had a watercolor painting on display, and Mr. Vanderbilt was absolutely taken with it. Despite his severely impaired eyesight, he spent time admiring the painting, showing an enthusiasm and appreciation that seemed completely genuine.

I remember watching him and thinking how remarkable it was. Here was a man who had grown up surrounded by magnificent estates, priceless artwork, and some of the finest possessions money could buy. Yet he was thoroughly enjoying a watercolor painting displayed at a little booth behind the Saratoga racetrack.

What impressed me just as much was the obvious affection the vendors had for him. They didn't seem to greet him simply because he was wealthy or famous. They appeared genuinely fond of him.

Those walks told me something about Mr. Vanderbilt that I could never have learned simply by sitting in his private box. He wasn't merely respected because of his name. He was liked, and I began to understand why.

The One Thing Money Couldn't Buy

During our two afternoons together, our conversations wandered far beyond horse racing and the Vanderbilt family fortune.

At one point, I innocently asked Mr. Vanderbilt about his children. I knew he had been married several times and had a large family. It seemed like a perfectly natural question, and I certainly didn't intend to make him uncomfortable.

His response surprised me. He confided that some of his children visited him only rarely. And when they did come to Saratoga, they sometimes stayed at the Saratoga Inn rather than at his house.

I remember feeling sad for him. Here was a man who had inherited one of America's great fortunes. He had magnificent homes, extraordinary social standing, and a place of honor at one of the country's most prestigious racetracks. Yet when it came to something as simple as having his children visit and stay with him, all that wealth and privilege seemed to mean very little.

During our time together, I also had the impression that much of his regular companionship came from his personal driver and the driver's wife, who seemed to be frequently at his side. I certainly couldn't know the full extent of his personal relationships, but it struck me that a man who knew so many important people might have had a surprisingly small circle of people who were part of his everyday life.

I didn't press him for details about his family. It was clearly a personal matter, and I felt a little awkward for having brought it up. But the conversation stayed with me.

For all his money and social standing, Mr. Vanderbilt was still a father who seemed to wish he saw more of his children. It was a reminder that even American royalty experiences the same human disappointments as the rest of us. Sometimes, all the money in the world can't change that.

Two Dinners at Chez Sophie

Our experiences with Mr. Vanderbilt didn't end at the racetrack. On both occasions, he invited us to dinner at Chez Sophie, his favorite restaurant in nearby Malta, New York.

Chez Sophie was a sophisticated French restaurant disguised as an old-fashioned silver diner. From the outside, you might reasonably expect hamburgers, milkshakes, and a waitress calling everyone "hon." Step inside, however, and you discovered fine French cuisine, carefully prepared dishes, and a restaurant that had earned an excellent reputation among serious food lovers. The contrast was so striking that The New York Times once featured Chez Sophie in an article titled "Three and a Half Hours to France, by Car."

I liked the fact that Mr. Vanderbilt chose this restaurant. He could have dined almost anywhere, yet he seemed perfectly happy enjoying an excellent French meal in a place that looked like it belonged along a highway in the 1950s.

One of his favorite dishes was sweetbreads. For those unfamiliar with them, sweetbreads aren't bread at all. They're a traditional delicacy made from the thymus gland or pancreas of a calf or lamb. It's not exactly the sort of dish most ordinary Americans would think of ordering for dinner. But somehow, it seemed entirely appropriate that a gentleman with Alfred Vanderbilt's pedigree would appreciate something so traditionally associated with fine dining.

Mr. Vanderbilt introduced me to sweetbreads, and I still remember a wonderfully funny exchange with our waitress. We asked whether she knew where sweetbreads came from. We meant what part of an animal they came from, but she apparently understood the question differently.

"Utica," she answered without hesitation.

Apparently, she thought we were asking where the restaurant obtained them, perhaps from a supplier in Utica, New York. We couldn't help laughing, and I've never forgotten her answer.

Another memorable moment came when it was time to pay the bill. Mr. Vanderbilt's eyesight had deteriorated so badly that he handed me his American Express card to take care of the payment.

I still remember looking at the name imprinted on the card: ALFRED G. VANDERBILT II.

There I was, holding the American Express card of a man whose family name had been associated with extraordinary wealth for generations. For someone who had spent the afternoon enjoying a borrowed place among America's aristocracy, it seemed a fitting way to end the evening.

The Day I Learned Mr. Vanderbilt Had Died

I remember a day in November 1999 when I was reading the New York Post. There was an article announcing that Alfred Gwynne Vanderbilt Jr. had died. He was 87 years old, and he had died at his Long Island home on November 12, while getting dressed to go to Belmont Park. It seemed a fitting, if terribly sad, ending for a man who had devoted so much of his life to thoroughbred racing. Even on his final day, he was preparing to return to the racetrack he loved.

I remember feeling terribly sad. Here was a man who had been born into one of America's greatest fortunes, who had known some of the most influential people in the world, and who had spent his life surrounded by extraordinary wealth, privilege, and tradition. Yet as I sat there reading that newspaper, I wasn't thinking about his money.

I remembered how graciously he had welcomed us into his box at Saratoga. I remembered how casually he talked about his extraordinary family history, and how genuinely interested he seemed in the vendors and their little booths behind the racetrack. I also remembered the sadness in our conversation about his children.

I had been fortunate enough to spend two afternoons and two dinners with this remarkable man. Reading the announcement of his death brought back a flood of memories: our conversations at Saratoga, his stories about his extraordinary life, the walks we took together around the racetrack, and those wonderful dinners at Chez Sophie. I felt a genuine sadness at his passing, but mostly I felt fortunate to have known him, even for such a brief time.

Back to Saratoga, Where the Memories Remain

I still visit Saratoga Race Course from time to time. Much of what makes Saratoga so special has survived the passage of time. The old wooden grandstands are still there, the horses still race, and the traditions continue.

These days, you'll probably find me toward the back of the racetrack grounds, sitting in a $15 lawn chair on the sparse lawn, perhaps with a cooler of beer on ice beside me. No jacket and tie, no exclusive box at the finish line, and no wealthy bankers or famous sports figures wondering who I am.

I don't particularly miss sitting at the finish line. The beautiful girlfriend is gone, Mr. Vanderbilt has passed away, and my brief membership in Saratoga's uppermost social circle expired long ago.

But whenever I look toward those exclusive boxes, I remember walking down the steps, feeling my imaginary social standing rise with every step. I remember the gentleman who had lived in magnificent mansions, known legendary figures, and owned some of the greatest racehorses in history, yet who could become thoroughly absorbed in a watercolor painting at a little vendor's booth.

For a brief moment in my life, I was the envy of some of Saratoga's wealthiest and most influential people. But what I remember most fondly isn't that fleeting taste of social status. It's the gracious gentleman who welcomed me into his world and treated me as a trusted friend.

Those two afternoons and two dinners were experiences I will always treasure. And whenever I think back on that extraordinary summer at Saratoga, I can't help but smile.

About the Author

Stephen C. Solosky, known as The Traveling Professor, is a retired college professor, travel writer, and author who has spent decades exploring the world and collecting memorable experiences along the way. He is the founder of Small Group Tours by The Traveling Professor and writes about travel, history, and the fascinating people he has encountered.