Bae, a Christian missionary who ran a legal tour company in North Korea, was sentenced to hard labour for 15 years in April 2013 following accusations that he was committing hostile acts against the state and encouraging North Korean citizens to work against the government.
Despite repeated appeals for his release, he remains imprisoned and struggling with various illnesses including diabetes, high blood pressure and fatty liver.
Rodman was visiting the closed state this week and is said to have a friendly relationship with Dictator Kim Jong-un.
Bae's family were hoping that Rodman might be able to help bring about the prisoner's release, given his closeness with the all-powerful leader and after he previously tweeted Kim Jong-un asking him to "do me a solid and cut Kenneth Bae loose". However, his most recent comments suggest a change in attitude.
In a drunken outburst in Pyongyang captured by CNN on Tuesday, Rodman asked a reporter who enquired as to whether he would be speaking to the North Korean leader about Kenneth Bae's plight, "Do you understand what he did in this country?"
"No, no, no, you tell me, you tell me. Why is he held captive here in this country, why?
"I would love to speak on this," he said, before swiftly changing the subject, leading to accusations that he was insinuating that Mr Bae deserves his punishment.
Rodman has been widely criticised for his impromptu speech, and has since apologised for the outburst, saying he has "embarrassed a lot of people".
"I'm very sorry. At this point I should know better than to make political statements. I'm truly sorry."
He has attributed his remarks to having too much to drink after finding his trip intensely stressful and pressurised.
"I had been drinking. It's not an excuse but by the time the interview happened, I was upset. I was overwhelmed," he said.
In an exclusive interview with a CNN news reporter, Bae's sister Terri Chung said that her family are upset that Rodman did not use his friendship with Kim to help her brother.
"He was in a position to do some good and to help advocate for Kenneth," she said.
"He refused to do so. But then instead has chosen to hurl these outrageous accusations against Kenneth. He clearly doesn't know anything about Kenneth, about his case. And so we were appalled by that.
"This isn't some game. This is about a person's life," she finished.
However, the Bae family have since released a statement saying that they accept Rodman's apology, but remain shocked by his outrageous comments.
"As Rodman has stated, being drunk and stressed is not an excuse for what he said, but we acknowledge he is human and we all do make mistakes," the statement reads.
"Our greatest concern remains the health and freedom of my brother Kenneth. We hope and pray that Rodman's comments and ongoing antics have not further endangered my brother. Kenneth's health and freedom are precarious."