Because Ye Feng had absolutely no evidence against this guy for cheating, you could say there was no evidence of his card switching.
In this day and age, everything requires evidence. Without evidence, Ye Feng would become the one falsely accusing others and would likely be disqualified.
So, for Ye Feng to advance in the competition, he must defeat this guy. But even though Ye Feng could see through his abilities, defeating him was extremely difficult.
Because although Ye Feng could switch cards, the opponent could too. Unless Ye Feng could be faster than him, but the opponent switched cards in a split second, so being faster seemed impossible.
Therefore, the ability of the guy in the hat gave Ye Feng a headache. To defeat him, it would take more effort than just skill.