Is it impossible to reopen the Boulder a second time?

Is it impossible to reopen the Boulder a second time? The first time my villagers opened it, I had just started playing, and didn't send any villager through the portal that hid behind the Boulder, thinking I could always try it on a later stage in the game.

But even though some villagers are curious about the Boulder, and tries to move it, and even though I've tried adding a second villager just like I think I did the first time they opened it, the Boulder remains shut.

I thought I would at least be able to reopen it again once I had finished all the Chapter Challenges, but that doesn't seem to happen. I should maybe add that I've tried using villagers which are Masters in Farming, Building, and Researching. Was it a one time only opportunity in the game that I missed, by not trying it the first time it opened?