The serfs revolt!

The taxes on those lands are collected without much of a trouble. Even if it is only a friction of the original that could be collected in better years... After some time, the serfs of those lands revolt against your oppressive rule! The matter is resolved as you have no army to oppose them and the neighbouring lords gather their armies to crush the revolt, in return for dividing those lands up between themselves. Now you only have the original amount of land you have held around your manor...