How do you demolish a building in your castle?

Inside your castle walls you cannot they're designed to be there. Outside your walls you can destroy those buildings like farms, lumber yards, medical tents etc.. just click on the building and hit the info tab and there you have the option to destroy a building.