The old Brown County Jail, now housing the Brown County Museum of History, was not the first jail to be built in Brownwood.  The first Brown County jail was built in 1876 at the corner of North Fisk and Water Streets.  That building and the nearby courthouse burned in March 1880 and all county records were destroyed.  Records of the Commissioner Court for 1880 showed an item "for tearing down the old jail and clearing the site for the new jail." A contract was let on June 7, 1880 for Martin, Byrue and Johnson to build a county jail.  The jail was accepted on January 15, 1881.   The cost of the jail was $5,595. By 1901, the jail was recognized as being insufficient for the county's needs.  In December 1901, voters elected to issue $30,000 in  bonds for a new jail.  County commissioners accepted a low bid of $24,925.60 from the firm of Martin, Moodie & Co. in partnership with Youngbood Brothers.  William Hood was appointed local architect and contractor.

Work began soon after the contract was signed and on June 29, 1903, the Commissioners Court accepted the new jail building.  Local stonemason Al Morton completed his work of filling the jail yard and building a stone wall around the perimeter of the building, while L. S. Leversedge & Son of Dallas erected an iron fence with post and gates on top of the coping.

The building is built of native sandstone, cement, sand and steel.  Lumber was used on the first floor, none upstairs (to prevent an inmate setting it afire.)  The walls on the inside are 18 inches thick, made of stone, sand and cement.  The thick walls are the pillars that support the building and they also make the building much cooler.  In 1902 there were no cars, trucks nor motor powered heavy equipment to build a building.  All equipment was horse or mule drawn and much of the work was done by manpower.  Some of heavy wagons were pulled by eight and ten mules teams.

The stonemasons carved each rock 18 inches wide and 18 inches thick, and the length varied to fit the place.  Some of the rocks were about six feet long, which can be seen on the front of the building.  The sandstone was dug from the rock hills surrounding Brownwood.  One quarry was located on Round Mountain, where the State School is today.  Another was on Willis Creek and another in the northern part of Brownwood towards Lake Brownwood.  Men chiseled the rock from the hillside with hand tools and it was excavated from the quarry with cables, chains, ropes, pulleys, and hoists.  The mules furnished the power to move the rock onto wagons and then into position at the jail site.

Men who worked on the building were paid one dollar a day, working six days per week and ten hours per day.  men who owned wagons and teams of horses or mules were paid extra for the use of their animals and equipment.

The sheriff and his family lived in the Sheriff's apartment on the first flood.  The back yard at the jail was the children's play ground.  There was another apartment downstairs for the jailer and his wife to live.  They were the ones who prepared the meals for the prisoners and took care of the prisoners.

On the second floor there was a large room with steel posts and where the wives could visit the inmates and for them to be able to spend time out of their cells.  It looked like a cage and was known as the "bull pen."  The cage was out in the middle with a walkway around the walls.  On Sunday families could visit by standing along the walls and they could bring tobacco and a small amount of snacks to the inmate.

In the 1930's pinto beans were served every day.  The beans were cooked in a large pressure cooker with plenty of salt port.  A serving was about a cup and a half.  Corn or potatoes were served with the beans.  Each inmate was given 7 thick slices of bread at noon that was to last until the next noon day meal.  On Sunday meat was added to the plate.

All the coffee they wanted was served at breakfast time and the only other drink was water from the bath room faucet during the day.  Bacon was served at breakfast on Sunday and if they had a slice of bread left it made a pretty good meal.  The inmates were served traditional Thanksgiving and Christmas meals.

Back in the 1930's children of the Sheriff could go to the store and purchase cigarettes or other kinds of tobacco products for members of the family.  One of the children soon learned that he could make some money by  going to the store and purchasing tobacco for the prisoners and for his efforts he was given a few coins.  If he was lucky he would earn enough money to go to a Saturday afternoon movie and buy a hamburger

Mose H. Denman was the first sheriff to live in the sheriff's apartment.  Denman was elected sheriff on two different occasions and served for 10 years.  In 1900 the term of office was two years.  He became sheriff in 1900 and served six years.  Frank Emison was sheriff for four years and Denman returned to the jail as sheriff for an additional four years.

In 1963, while still serving as the County jail, the building received a Texas historical marker.  The distinctive sandstone structure was in need of repair and renovation, but the building was determined to be structurally sound.  Today, the old Brown County jail building is one of the most impressive structures in West-Central Texas, conveying fortress like strength and romantic medieval military tradition.

When the new county jail was built and finished on June 29,1981, the Brown County Historical Commission began to take steps to preserve the building as a site for a museum of local history.  A board of directors was appointed by the Commission, meeting initially in July 1982.  It took another year for the board to get the jail cleaned up and ready to open as a museum.

The old jail has been placed on the National Registry of Historic Places and has been designated as a Historic Landmark by the State of Texas.  A new Texas Historical marker was added to the building in 2003 on the 100th anniversary of the jail..