Except we don't, we vote for a person - the representative for our constituency. That is, in FPTP elections.
Now he/she most likely campaigns under a party banner and, therefore, they are in that case following a particular manifesto but they are free to make whatever alliances they so wish.
They are not required to vote with the party of which they are a member, the party of which they are a member is not obliged to keep them in that party through thick and thin and they can always cross the floor.
I know it's a bit pedantic and I understand that 'voting for a party' is the practical outcome of any electoral system which is done on a party basis but that doesn't alter the intricacies of it.
That is undoubtedly true but I would have to suggest it is not the mindset of the majority of the electorate.
In practical terms the majority of voters are voting on election day for who they would like to lead the country and tick the box accordingly.
I would go so far as to say a significant number of your average voters only find out the name of their local candidates when they read through them on the ballot sheet.
For this reason the change of prime minister outside of an election is a significant event for the voting public and a second change will not be well recieved.