There aren't many people, just Kong Er, Qu Da, plus Qin Fang and Song Qingshan, making a total of four. As for the few rich young men from Hong Kong Island who came with Kong Er, they found somewhere else to go.
First, they felt their relationship with Qin Fang wasn't that close, and second, their status was a bit lower compared to Kong Er and Qu Da, so with them, they couldn't really let loose.
Besides, Qin Fang and his friends were originally just meeting up to catch up, and if they joined in, they would definitely be unwelcome, which they were self-aware of.
As for Chu Yunxuan and Helen, they wanted to join in too, but considering it was already late, and not knowing how late the four guys would stay out, they decided not to join since lack of sleep is a woman's enemy. So, they didn't tag along.