So why is it when I do an upgrade on a building it's taking my flowers instead of giving me flowers?

When you upgrade your building (or buy decoration or renovate your landmark), you are adding Flower value to your island. The display next to the flower icon, on the other hand, shows how many more Flowers you will need to reach the next level. So by adding more Flowers to your island, that number will go down. When it reaches zero, you will level up, and the display will change to how many flowers you need for the next level. Any additional Flowers will go toward the next level, so no Flowers are lost.