How to harvest those mineral outside the town?

The areas that you see in the castle view (not the world map) that are outside your walls are there for the Explore option. You can explore with your construction vehicles by tapping on the hammer icon at the top left and select the explore option. (this unlocks when you reach level 12).

You can attack other players in the world map (that are not in your alliance). You do not conquer them, you raid resources from them if you are stronger and wound or kill their troops. If you attack enough to reduce their durability to 0 then their city will be randomly relocated to a different location.

You can send idle workers to explore, from the hammer.