Woodbury Hardy was born in Hopkinton, New Hampshire, in March 1833, the son of Ozias Hardy and his wife, Lavinia Barden. He had three brothers: Samuel (b. 1829; m. Abby A. Putney, 1858), Charles, and Sanford. Sanford moved to Illinois before 1859. Woodbury worked in South Danvers, Massachusetts, in 1860, and lived in Illinois during the Civil War. He served as a private in Company E of the 95th Illinois Infantry Regiment. After the war, he returned to Hopkinton, where he worked as a farmer for much of the rest of his life. On March 31, 1861, Woodbury Hardy married Ellen Matilda Price of South Danvers, Massachusetts. She was the daughter of William Price and Lydia Felton, and had 4 siblings: Lydia, Sarah Jane (m. Samuel F. Prey), William, and Eliza. Woodbury and Ellen Hardy had two children: Arthur and Clara.