I'm sorry you're in this position and I can completely understand the stress you're feeling over it. It's important to remember though that you are the one in charge of your body, not anyone else, and if you want to wait a few more weeks to have the colposcopy, they can't make you do it now and you don't 'have' to have it done just because some testy nurse has said so on the phone. It's up to you.
I would agree that it's a good idea to go to the appointment anyway, but keep in mind that whether or not you have the colposcopy is your choice, not anyone else's. Talk to the consultant about your anxieties, explain that you are a high miscarriage risk and that you think the stress you feel over having it now would not help with that risk, let alone the procedure itself. You can say that you're very happy to have the colposcopy done, just not right now. Also ask the consultant exactly what the smear result was. All a smear result can tell you is that there is some sort of change, it can't tell you the nature of that change, but it can often detect the severity of that change. So, for example, if your smear result was 'borderline change' or 'mild dyskaryosis', I suspect your consultant will be quite happy to leave the colsposcopy until after your 12 week scan, and maybe even until after 20 weeks or after you've given birth. If the changes are mild, then there's no panic as these things tend to take a long time to progress in the cervix. If the smear result is 'severe dyskaryosis' then I suspect the consultant will be keener to have a look sooner rather than later, but only because he/she will want to look after you and do what's best for you.
So, try not to worry too much - it's your decision, not anyone else's - and just see what you think the best thing is to do after having a chat with your consultant.
Let us know how you get on.