John Fraser (1769-1813) was born in Inverness, Scotland, and became a successful merchant and a wealthy slave trader. Fraser was married to an African woman named Phenda, with whom he had five children: James, Margaret, Mary Ann, Eleanor, and Elizabeth. He and his family lived primarily in Bangalan near the Rio Pongas River in West Africa, where Fraser owned a "factory" that held slaves before sending them to the America slave markets on ships he owned. Fraser owned property in Charleston and brought his son James there to learn the family business. With the British and American ban on the slave trade in 1809, Fraser left South Carolina and bought extensive tracts of land in Spanish Eastern Florida, where he could continue importing and auctioning slaves. There he established a large rice and cotton plantation. At the height of its value, his property was worth hundreds of thousands of dollars. In 1813, however, his plantations were largely destroyed in an insurrection known as the Patriot Rebellion. Around the same time, Fraser drowned in a ship wreck on his way between Africa and America.
He divided his estate among his five children, and appointed his friends Kingsley and Philip R. Yonge as executors. The estate was disputed for the next several decades, as Fraser's sister, Ann Fraser Robinson, and her husband William Robertson, protested the will on the grounds that the "heirs" were in fact slaves, and therefore not entitled to the property, and that a proper inventory of the Fraser estate had never been made after the Patriot War. The proceedings, involving Fraser’s children Mary Ann and Elizabeth, who actively pursued their father’s claims from their homes in Rio Pongo and Sierra Leone, were exceedingly protracted, with litigation continuing to at least 1873.