Written records about the fortress have not been preserved and therefore reasons why the fortress ceased to exist remain unknown.
The Mrač property was transferred to the Benešovic family in at the end of 14th century. The owners had their seat, a new fortress, which they built further away from the original seat of Zdislav of Mrač, at the lowest end of a hill above the river of Benešovský potok. This fortress has been preserved to date.
Amongst some of the very first 14th century owners the names of Beneš of Lopřetice na Mrači at the end of 14th century, Oldřich of Mrač in 1410, and in 1438 Beneš of Dubá and of Mrač, ancestor of the Mračský Family from Dubá were recorded. His descendants lived at the fort until the first half of the 16th century. In 1540 the fortress was purchased by Jaroslav of Šelmberk and Kost, Senior, the owner of the not too distant Hrádek nad Sázavou (at present known as Komorní Hrádek), who later passed on the extended Mrač estate to his wife Anna, nee Krajířová of Krajek. Her heir was her son Jan (+1597), from whose inheritance the estate was returned to Komorní Hrádek to be owned by Adam of Valdštejn, Junior. His son, Jan Viktorin of Valdštejn, had to forfeit Mrač to his creditor Václav of Vrbno and Bruntál in 1648. At this time, after the Thirty Years War, the fortress was described as being deserted. In 1663 the Mrač estate was purchased by Bernard Ignác of Martinice (+1685) whose daughter transferred the inheritance to her husband Jan František of Vrbno and Bruntál, the son of the previous Valdštejn creditor in 1690. Stone from the deserted fortress was used to build both the chateau and the Vrbnov seat at Mrač. Eventually, the fort was rebuilt as a granary at the end of 17th century. It was preserved in this form until the second half of 19th century when it was reconstructed into flats for workers of the Mrač Granite Quarry. It is still used as housing today. It is a one storey building with an oblong ground plan, built in stone from the quarry, without plastering and with a hip roof. A stone wall to the east joins the house and a square courtyard.
A chateau which the Vrbno family built above the deserted fortress after they obtained the Mrač Estate, was used as a dwelling - seat of knights. According to a description from 1720 it was “a knightly masonry building”, on the ground floor there was an office, a room for the District Commissioner, a flat for a copy writer, two kitchens and two pantries. On the first floor there were rooms for the noblemen: three rooms, dining room and a chapel. The building which was not very large was to be replace by a bigger chateau, the construction work, however, was not realised. In 1725 Mrač was purchased by Jan Josef of Vrtba and joined it with Konopiště. The new seat gradually deteriorated and soon only massive ruins of external walls with window openings were all what was left. These ruins remain to be seen even today.