After the General's death in 1815, his sons (colorfully named White, Red, and Green) sold his 2,052½ acres to James Speir. In 1840, Speir gave the Central of Georgia Railroad right of way through his land for the rails connecting Atlanta and Savannah. Stop Number 11 was dubbed Speir's Turnout because it was one of the few places where one train could turn out to a siding to unload, allowing other trains to pass. In 1859, Speir sold his holdings to Marcus A. Evans and Russell W. Johnson. They subdivided the property, sold lots, gave land for a school, built a cotton gin, and opened a large mercantile business.

In 1860, the town's name was changed to Bartow to honor Savannah native Francis Bartow, who died during The War Between the States at the First Battle of Manassas. The town was a constant source of supplies for the Confederate soldiers until the arrival of General William Tecumseh Sherman's troops, who swept through the area on their mission to destroy crops, houses, and rail lines between Atlanta and Savannah. While Sherman turned south just prior to reaching Bartow, in order to blow up a bridge below Shake Rag (now Wadley), his men entered town to burn all the cotton and supplies they could not take with them. During their stay, the union troops camped in a pine thicket that separated the house at 1038 Speir Street from the railroad tracks.

Following the War Between the States, the rails and bridges were repaired, and Bartow truly began to grow. For a time, the growth centered on the south side of Williamson Swamp Creek. Stores, offices for a dentist and a doctor, and the first churches were built there. But, the importance of the railroad stop served as a magnet to draw people to the north side of the Creek. By 1920, Bartow's prime, the town boasted a population of nearly 700. It owned its own electric power plant and had a fine school with its own bus system (a rarity at the time), an opera house for concerts and silent movies, a hotel, an indoor roller skating rink, and three department stores (one of which sold Paris originals). Business was so good that the depot at Stop Number 11 was second only to Macon as a freight stop between Atlanta and Savannah.

 The 1930's droughts, teamed with the arrivals of the boll weevil and the Great Depression, started the town on a slow but steady decline until 1992, when the Bartow Community Club was founded. This organization became the catalyst for a chain of events. Collectively, they have revived pride in Bartow's history and helped make it a highly desirable place to live.