If EU bears the costs of hosting these consumers (social/financial etc), then it only makes sense that if you wish to access their market then you will have to pay. This payment can be via a direct capital transfer or some portion of it can be via an in kind contribution, e.g., free movement of people.
Are the leavers willing to pay the taxes required to fund keeping these willing workers out, while getting access to the EU market?