Because it's not true.
https://fullfact.org/economy/labours-record-on-unemployment/
Plus here's a graph I just whipped up from the ONS data (Unfortunately they only have 1971 onwards). Blair and Callaghan both left office with lower unemployment rates, Brown's was initially going down before the GFC, and Wilson went up. It should be noted that only May had a consistent downward trend out of the Tory PMs, though that seems to just be a continuation of Cameron eventually seeing a consistent decline. Major and Heath ended up similar to Callaghan in that they squeaked under the line in their final months to finish with a slightly lower rate, and the two biggest spikes came under Thatcher and Major.
But then again, of course most PMs will see a rise in unemployment, as that tends to be one of the death knells of a government, which the next one has to come in to try to wrestle back to an acceptable level.