Mention of the castle goes back to medieval times and the von Herisberg family (later, Härsberg and Hersberg). In 1621, the Benedictines of the nearby Ochsenhausen monastery bought the castle from Härsberg descendants. After secularization, the property fell to Prince Franz Georg Karl von Metternich. It was bought by the House of Württemberg in 1806, and then by Count Konstantin Fürst von Salm-Reifferscheidt in 1838. In 1929, his descendants sold the estate to the South German Pallottine Province. In 1994 it was reopened as the Pallottine spiritual house.