He was the third son of Francis Luttrell (1628–1666) of Dunster Castle.[1] His mother Lucy was a daughter of Thomas Symonds of Whittlesford, Cambridgeshire.[1] Colonel Francis Luttrell (1659–1690) was his older brother.[1]
He inherited the heavily indebted Dunster Castle after the death of both his elder brothers and then his brother Francis's son in 1703.