The palace was commissioned by Duke of Berwick. He was also Duke of Liria. In the early 19th century, it passed to the House of Alba. Eugénie de Montijo, last empress consort of the French, died here in exile in 1920.
All but the facades were destroyed during the Spanish Civil War. It was rebuilt by Jacobo Fitz-James Stuart, 17th Duke of Alba and his daughter Cayetana. The British architect Edwin Lutyens provided some designs for the interior. The reconstruction used his plans, even though he died before then.