Whithaugh became a farm estate with the remnants of the now dispersed Armstrong clan living on the land until the eighteenth century where it was taken over by the Elliots who were reckoned to be the principal sheep farmers in Liddesdale in their day.

The house has been remodelled and extended over the centuries, the last being in the nineteenth century by the Oliver family who built the Victorian facade which now stands. Since 1973 Whithaugh is currently owned by the welters family.