Whether it's the Black Yan Hall or Yun Kaige and his people, none have truly left this island. Instead, they are concealed nearby.
Lin Chen observed all this with his Abyssal Eye.
Within these ruins, everyone's divine soul perception was suppressed. Yun Kaige and his group believed that concealing their aura was enough to evade Lin Chen's detection. However, they didn't realize that their minor activities near the island, although Lin Chen's Abyssal Eye was also somewhat suppressed by the ruins, were not as severely affected as their soul power. Hence, Yun Kaige's actions were all seen by Lin Chen.
Even in these ruins, the Abyssal Eye can perceive everything within a five-thousand-mile radius.
Yun Kaige and his group naturally didn't know about Lin Chen's possession of the powerful Abyssal Eye, so they weren't far when concealing their forms to hide around the surroundings.