You will have to re-conquer it in order to be able to summon any unlocked characters and win crystals from it. You must always defend an outpost as soon as possible (at least at the start of the game) otherwise the difficulty of defending or taking that outpost will ramp up quickly and you might get stuck in the game as you won't be able to conquer anything. If you’ve lost a Territory, concentrate on other territories for now and remember to defend them as soon as possible. Once you'll have a solid army and a good character you'll be able to reclaim any lost Territories.