A British government vetting agency raised significant concerns about Peter Mandelson’s associations with senior foreign figures before he was appointed as ambassador to the United States last year, according to newly reported information.