The Athens Factory, formerly known as the Athens Cotton and Wool Factory and more recently as the Old Mill, is located on Tax Parcel No. 17-1-D2-G-002A on the west bank of the North Oconee River at the intersection of Baldwin and Williams streets.

The historic industrial complex consists of two brick buildings joined together to form an L-shape. The smaller "wool" building was built on the riverbank parallel to the river. The larger "cotton" building extends westward at a right angle from the south end of the wool building. Rectangular in plan, the three-story wool building still has its original stone-walled millrace, which passes through the walls of the irregular-cut stone foundation that supports the ground floor. Load-bearing brick walls and internal timber framing support the partitionless upper floors and the flat roof. On the northeast elevation, a stepped parapet obscures the roof and its rafters, and flat "jack" arches of brick appear over the windows. Similar in almost every respect, the four-story cotton building displays a more slender and elongated form. The western end of the north elevation features the principal entrance, a round-headed, double doorway located beneath a segmental arch and housed in a square projecting tower. A brick-enclosed elevator shaft occupies the inner angle where the two buildings join; astride the millrace at the outer angle stands the structure that first housed the turbine and later the generator. Changes in brickwork and fenestration indicate subsequent enlargements to both buildings. At one time the original five acres contained a complex of outbuildings including a picker house, a stone warping house, dye houses, boiler rooms, and warehouses. The amended property boundary omits 2.3 acres where modern construction had severely disturbed the archeological potential.

Often confused with the Athens Factory because of its early use of the same name and the overlap of its investors, the Georgia Factory was founded in 1829 by John Johnson, William Dearing, John Nisbet, Augustin S. Clayton, and Abraham Walker, who began its construction at shoals several miles downstream known today as Whitehall. Three years later these same men, except for Johnson, purchased from William A. Carr approximately 55 acres on the North Oconee River, which included the five-acre tract where they built the Athens Cotton and Wool Factory. Both mills operated under the name Athens Manufacturing Company until the investors sold the downstream mill to John R. White around 1835. White's mill was known thereafter as the Georgia Factory, with its nearby owner's mansion and mill village of Whitehall. The in-town mill became known as the Athens Factory. Initially of wood-frame construction, the Athens Factory's first buildings suffered from fires that destroyed the complex in 1834 and again in 1857, which led in 1858 to construction of the brick buildings extant today. This waterpowered factory complex manufactured cotton yarns, woolen yarns, and cloth. The company sold off a sixteen-acre tract some hundred yards upstream for the construction of the Cook and Brothers Confederate Armory in 1862, repurchased the parcel in 1870, and converted the armory building into a "weaving" mill. The Athens Factory continued operations through the first quarter of the twentieth century, finally closing in 1926 three years short of its hundredth birthday. The company dissolved in 1947, the Southern Mill Supply Company acquired the Athens Factory buildings in 1957, and the next year Chicopee Mills bought the weaving mill in the old armory building. Then in 1977, William L. Laird purchased four acres of the Athens Factory's original five-acre tract with the existing brick buildings and rehabilitated them into the Old Mill Center, a complex that included a tavern/restaurant and shops. O'Malley's Tavern, which opened there in 1978, currently operates a fitness center, bar, and restaurant within the Athens Factory buildings.

The Athens Factory is listed on the National Register of Historic Places (July 31, 1980), and the Georgia Historic Marker Program has given marker recognition to the Cook and Brothers Confederate Armory (029-02).