Sackets Harbor was formally established in 1816 as a U.S. Army installation and named “Madison Barracks” for President James Madison.
Construction of Stone Row was started in 1816 and completed in 1819. President James Monroe visited to view the construction.
President Martin Van Buren visited Madison Barracks in 1838. A volley of muskets fired in his honor nearly resulted in tragedy when a ramrod, left in the musket barrel by one of the soldiers, flew through the air like an arrow and pierced the ground near the president.
Ulysses S. Grant, a future Civil War general and president, served two tours of duty at Madison Barracks as a junior Army officer, first from 1848 to 1849 with his wife, Julia Dent Grant, and then again in 1851, accompanied by his wife and son.
In 1906, Col. Philip Reade, commander of Madison Barracks, worked with north country community leaders to locate a new training ground that would eventually become Fort Drum.
Vacant for years, the Madison Barracks apartment buildings were restored almost 40 years ago. The site was listed on the National Register of Historic Places in 1974.