Jacob Jennings Brown, one of the few American army officers to gain distinction in the War of 1812, was born into a Quaker family in Bucks County, Pennsylvania. His parents were Samuel Brown and Abi White Brown. He began his career as a surveyor and teacher; he opened a Friends School in New York City in 1798, while also studying law. By 1800, he had purchased land on the borders of the St. Lawrence River and Lake Ontario; here he founded a settlement on the Black River, which became the town of Brownville. In 1802 he married Pamela Williams (b. 1785), sister of his friend and associate, Nathaniel Williams, of Williamstown, Massachusetts. Brown and his wife had nine children.
Brown was active in local politics, becoming a judge of Jefferson County and a brigadier general in the state militia. At the outbreak of hostilities in 1812, Brown was given command of the frontier from Oswego to Lake St. Francis. In this post he successfully repulsed British forces at Ogdensburg in 1812, and the following year at Sackett's Harbor. For his efforts, Brown was immediately promoted to brigadier general in the U.S. Army, and the following year to major general. He commanded the northern division of the army in the successful Niagara campaign of 1814, where he received two severe wounds. At the close of the war, he continued in command of the northern division, becoming commanding general of the army in 1821. He moved to Washington in this year and had a stroke soon after. He died in Washington in 1828.
Edmund Kirby (1794-1849) was an aid to General Brown after the war. Kirby entered the army in 1812 as an ensign and steadily climbed the ranks to 3rd, 2nd, and 1st lieutenant, captain, major and eventually colonel in the Mexican American War. He died in 1849. Kirby married Brown's second daughter Eliza in 1825, and was deeply involved with the Brown family. Brown and Kirby shared an investment in a Mill in Dexter, New York, and in 1825, with Brown deeply in debt, Kirby purchased the Brown mansion. Members of both the Kirby and Brown families lived in the house until 1865.