Mr Andrews, who was set to deliver the opening speech, was under pressure by the Greens to boycott the event. The MP, however, revealed his decision was made after learning that the event was to be hosted by conservation Christian ministry, Catch the Fire.
"Tolerance is a critical value in a western liberal democracy like Australia. It was for this reason that I intended to address the World Congress of Families meeting," Mr Andrews said in the statement on Friday.
"The calls for me not to attend demonstrate the intolerance of the Greens and the left - instead of arguing their case in the public arena they seek to shut down debate."
"Equally, I cannot support intolerance from other quarters. As I have been informed today that the event is now to be hosted by Catch the Fire, I have decided not to attend."
A recent report by the Human Rights Commission labelled the World Congress of Families a 'hate group' promoting 'anti-LGBT bigotry'.
The American-based organisation is also reportedly affiliated with groups behind the violence against LGBT people in Uganda, Russia and Nigeria.
Victorian Attorney-General Robert Clark and Liberal MPs Bernie Finn and Jan Kronberg have since joined Mr Andrews in withdrawing their attendance of the event.
Several groups including The Progressive Atheists and Radical Women plan to protest at the event venue. According to a spokeswoman from the socialist feminist organisation, thousands are expected to rally, the Sydney Morning Herald reported.
Kevin Andrews with Prime Minister Tony Abbott (Kevin Andrews via Facebook)