James H. Miller was born in Kentucky in 1827, the son of Elizabeth Miller (b. 1787). He married Mary Eliza Waggener (1832-1916), the daughter of James M. Waggener, on February 24, 1848. They moved to Elm Grove and Gilead, Missouri, in the early 1850s and later lived in Long Island, Kansas. The Millers had ten children, including David J., Gilead Ann, Mary E., Leanora, Alice, Lyda, and Joseph. During the Civil War, James H. Miller served in Missouri and Arkansas with the 3rd Missouri Cavalry Regiment. He died in 1913.