The hermit communist country retracted an invitation to US diplomat and its human rights envoy to North Korea, Robert King, at the last minute.
It is the second time North Korea has withdrawn an invitation to King and will come as a blow to Bae's family, who are campaigning for his release after 15 months of imprisonment.
The Korean-American missionary ran tour group for Americans and Canadians as a way of supporting the North Korean economy but was arrested in November 2012 for "hostile acts" against the state and sentenced to 15 years of hard labour.
There are concerns for Bae's health and it was reported last week that Bae had been transferred from hospital to a hard labour camp.
The Times reports that North Korean authorities withdrew the invitation to King over military exercises taking place between Washington and Seoul, and claims from Pyongyang that American nuclear-capable B52 bombers were flying near the Korean Peninsula.
The US State Department has reportedly said it is "deeply disappointed" over the retraction and has accused North Korea of trying to use Bae as a "bargaining chip".
"We remind the DPRK [North Korea] that the US-ROK [South Korea] military exercises are transparent, regularly scheduled, and defence-oriented. These exercises are in no way linked to Mr Bae's case," a US official said.
"We again call on the DPRK to grant Bae special amnesty and immediate release as a humanitarian gesture so he may reunite with his family and seek medical care."
It further reported that the State Department has expressed its support for the efforts of Bae's family and civil rights campaigner the Reverend Jesse Jackson, who has offered to go to Pyongyang to discuss his release.