In 1406, the same year that James I was captured by English pirates, and Robert III died, the castle was badly damaged by an English force that had sailed into Brodick bay. Further destruction was inflicted by John of Islay in 1455. At some point after 1470 the castle was granted by James III to his brother-in-law, James Hamilton, 1st Lord Hamilton. His son, James Hamilton, 2nd Lord Hamilton was created Earl of Arran in 1503. At this point Hamilton added the Lymphad of the Isles to his Armorial bearings

The castle had been rebuilt by the Earl by 1510 in the form of a Tower house, but suffered at the hands of the Campbells and the MacLeans. During the "Rough Wooing" of Mary Queen of Scots, Brodick castle was attacked by an English force led by the Earl of Lennox on behalf of Henry VIII, in revenge for the actions of the 2nd Earl. Lord Arran was the Regent of Scotland whilst Mary was in her infancy, and was second in line to the Scots throne. In 1543, he had been heavily involved in arranging the marriage of Mary to the Dauphin of France, prior to this she had been promised to Edward, Prince of Wales. Arran was rewarded for his efforts, however, and was created Duke of Châtellerault in the Peerage of France. During Regent Arran's tenure at Brodick he continued to enlarge and expand the castle.

Brodick Castle did not escape the religious paroxysms that affected seventeenth century life (see the Wars of the Three Kingdoms). In 1639 Scotland was divided between the Presbyterianism of the Lords of the Congregation, and the Episcopalianism favoured by King Charles I. James Hamilton, 3rd marquess of Hamilton, the King's advisor on all things Scottish, was sent north to enforce the King's will, he had previously dissolved the General Assembly of the Church of Scotland when they had abolished the Episcopacy. Archibald Campbell, 1st Marquess of Argyll, was the de facto ruler of Scotland and leader of the presbyterian faction. Argyll seized Hamilton's castle of Brodick. Hamilton was made a Duke in 1643 and recovered his castle the following year at the outbreak of the Scottish Civil War. It was lost again to the Campbells in 1646, as the Royalists fortunes foundered. The Duke was captured after the disastrous Battle of Preston, and faced the block in March 1649. He was succeeded by his brother William, Earl of Lanark, but the second Duke died of wounds received at the Battle of Worcester in 1651.The Duchy of Hamilton and Earldom of Arran passed to the first Duke's only surviving child, Anne. She had been unwittingly sent to Brodick for safety. In 1650, Oliver Cromwell's Roundheads had taken control of the castle and had extended it by building an Artillery battery to defend the Firth at this strategic position.

Duchess Anne returned to her estates in Lanarkshire and West Lothian and in 1656 married William Douglas, 1st Earl of Selkirk. Anne did not return to Brodick, however her husband the newly created Duke of Hamilton for life, used the castle as a base for hunting excursions.

In the following years Brodick was used mainly as an occasional sporting estate. In the nineteenth century, it became residence for the eldest son of the 10th Duke, styled the Marquess of Douglas and Clydesdale.

William, 11th Duke of Hamilton married in 1843, the Princess Marie, daughter of the Grand Duke of Baden and Stéphanie de Beauharnais, adopted daughter of Napoleon Bonaparte. In 1844, massive building work was undertaken at the castle, almost tripling the size of the building, under the architect James Gillespie Graham. The Twelfth Duke, William had no male heirs, so although his titles passed to his distant cousin Alfred Douglas-Hamilton upon his death, he entailed the castle upon his only daughter the Lady Mary Louise Douglas-Hamilton. She married James Graham, 6th Duke of Montrose in 1906, and so after more than five hundred years Brodick castle passed out of the Hamilton family.

The Castle and gardens were acquired by the National Trust for Scotland from the Lady Jean Fforde in 1958, in lieu of death duties upon the death of her mother, the Dowager Marchioness of Montrose. The castle is open to the public during the summer, with Brodick Country Park open all year round.