We would need to prove it was in the communities beneficial interests in order to obtain the CPO required. Losing X amount of houses but gaining Y benefits thus adding value to the whole area.
If it’s for the wider greater good of the overall majority of residents, then harsh as it is yes it is ok and a few shouldn’t get in the way of regeneration.