Why are there sections of my town I can't build on?

Why are there sections of my town I can't build on? I took a screenshot but don't know how to add it. There are two areas in my town that are regular grass, no special mine or anything that I can't build anything on. No roads, walls or buildings. One is small, like 2x4 plot, but the other is really big... Nearly 4x6 or so and part of it is where the wall should go.

I can't figure it out. I've tried digging a river and refilling it... I'm at a loss.