Clare must have remembered when Fred said that it was his fault that the two soldiers who tried to help him ended up as zombies. At first, Clare also felt guilty about what happened to his comrades at the South Fortress, but somehow he felt annoyed when someone else kept blaming himself for the disaster.
After seeing Fred's expression when they were still on the truck that took them back to District 1, he somehow realized that there was no point in feeling guilty. Everyone who wanted to go to the fortress was prepared to die; that was a fact. No one was brave enough to die, but those people must have known the risks of going to a place like the fortress.
