Is it worth participating in PvP even if I have a weak party?

Definitely! There are rewards for each ranking you'll reach, like SR and SSR hero shards, diamonds, and Arena Badge. Even if you don't progress that much, you won't stay weak for long. Eventually, you'll climb up the ladder.

The good news is that when the season ends, the rewards will be refreshed and you'll be able to collect them again as you climb up the ladder.