Lord Carysfort died in October 1772, aged 51, and was succeeded in the barony by his son John, who was created Earl of Carysfort in 1789. Lady Carysfort died in March 1783, aged 60.
HMS Carysfort (1766) was the first ship named in his honour following his service as Lord of the Admiralty. In 1941, HMS Carysfort (R25) was named in his honour as the fifth Royal Navy warship to carry the name Carysfort.
Proby married the Hon. Elizabeth Allen, daughter of Joshua Allen, 2nd Viscount Allen, in 1750: they had a son and a daughter.[2] The daughter Elizabeth (1752–1808) married Thomas John Storer (died 1792).[3]