If I have 4 active workers, it is worth investing in Farmer's Guild?

The farmers guild building is your Farmer class specific building and is definitely worth upgrading. If you raise it to level 15 you will get 20% buff to your rss production and the bountiful harvest reward (every 12 hours) will be 2.4 hours worth of resources.

You also get 15% boost to gathering speed and 30% extra load capacity. All good things for a farmer.