James Horatio Seymour was the brother of the original builder of Seymour Point, later Penwood, Helen Seymour Sylvester, wife of William Bedell Sylvester, sister of Henry Seymour and daughter of William H. Seymour.
James moved from Brockport, New York to Sault Ste Marie, Michigan, where his brother Henry had a very successful reaper factory and later lumber and farming businesses.
When William H. Seymour died, he left the family home in Brockport to his daughter, Helen Seymour Sylvester. When Mrs. Sylvester died, the house was left to her brother James, who by that time lived in California most of the time. Mrs. Sylvester's Adirondack estate, Seymour Point, was also left to James Horatio Seymour, with a life estate reserved to Mrs. Sylvester's husband, William Bedell Sylvester.
In 1930, upon his death, the Brockport house was given to the village of Brockport along with a substantial endowment to run a library in the house. In 1936, books from the local community center were moved to Seymour House and the Seymour Library was opened to the public.
The Seymour Library survives to this day, but is now run in a larger and more modern facility. The Seymour House is used as the village offices in the village of Brockport.