Australian Christian Lobby Managing Director Lyle Shelton said the issue of same-sex marriage achieved an incredibly high profile during the election campaign but proved not to be a vote winner for Mr Rudd and Labor.
"Any analysis of Saturday's election must take into account that this issue was front and centre of Labor's campaign but failed to attract votes amongst the broader Australian community," Mr Shelton said.
"Mr Rudd's bullying of Christian Pastor Matt Prater on Q&A last week has caused many people to worry about freedom of speech and freedom of belief should the law on marriage be changed.
"There are many Australians who for reasons of conscience will always believe that marriage is between a man and a woman and will want to teach their children this," Mr Shelton said.
"Mr Rudd's tone suggested that there was no place in Australia for these beliefs."
Mr Shelton said it should not be up to politicians to change a law that would impact so heavily on freedom of speech, belief and religion.
"The far-reaching consequences of changing the definition of marriage mean a conscience vote of the Australian people, not politicians, is what is needed."
Moves by the ACT and NSW parliaments to introduce same-sex marriage meant Australia risked having a hodgepodge of marriage laws when in reality the marriage power resided with the Commonwealth.
ACL's election panel webcast to 300 churches last week heard former Attorney-General Robert McClelland and former deputy prime minister John Anderson both call for a constitutional referendum to settle this matter.
"I think the High Court will ultimately, if there is a change in the law, be required to determine what the constitutional founders meant when they included the marriage power in the constitution," Mr McClelland said.
"I don't think that's a comfortable place for the High Court to be. I think if there is going to be any reform in this area, and I don't think there should be, but if there is it should be through the public debating it through the course of a referendum."
Mr McClelland reminded the audience that his 2009 reforms to 85 laws meant there was no discrimination against same-sex couples.
Mr Anderson said he was "deeply troubled by the lack of civility" in the debate.
"I think this debate puts politicians under unbelievable pressure because the community is so polarised.
"I actually think this now needs to go to the people."
Mr Anderson said a referendum should come with a "publicly funded, properly articulated yes and no case".
He said there should also be an honest acknowledgement of what changing the law on marriage would mean for freedom of belief and the right to express that belief.
He also said the rights of children, wherever possible, to be raised by both their biological parents should be part of the debate.