So he still has to use his ability to fight because it's easier to use his plant abilities and be more helpful to him that way; fighting using his abilities would also need more refined control than just when they needed to grow plants for consumption.
In fact, most of those people who only grow plants for consumption have more difficulty in improving their abilities because they don't refine it that much after reaching a certain stage.
It's still very helpful for plant abilities in the early stages, but if they want to continue improving, then they have to use their plant abilities for combat.
Of course, different people also have different aptitudes; that's why some people are improving fast and others are a bit behind, but at least they could all continue improving if they used their abilities properly.
The only difference would, of course, be the speed of their growth and their specialty when using their abilities.
