"
a village called Swaffham"
Blimey mate, did you move from the city by any chance?
I'm afraid you'll find that Swaffham is what passes for a town by Norfolk standards
Joking aside though, I like Swaffham (I work out that way some times) and it's not a bad place to be for trail riding oportunities.
There aren't many Byways around your part of Norfolk but there's quite a few Unsealed Unclassified Roads that are worth checking out, especially if you head north toward the coast.
If your not already a member I'd suggest joining up and getting along to the Norwich Group meeting at Trowse (2nd Wednesday of the month). They'll be able to show you plenty of great lanes in your area.
Also the Cambs Group ride in the area fairly regularly and once your a member you'll be welcome to join in with group rides etc.
Cheers
Richard