Col Gaddafi said Islam should "become the religion of all Europe" upon his arrival in Rome on Sunday.
The secretary of the Vatican's Congregation for Evangelisation, Archbishop Robert Sarah, has dismissed the comments as a "non-solicited provocation lacking seriousness", according to Agence France-Presse.
"To speak of the European continent converting to Islam makes no sense, because it is the people alone who decide consciously to be Christian, Muslim or to follow other religions," he said.
Archbishop Sarah said the "true danger" for Europeans was relativism, a "lack of attention to faith", "the weakness of religion", and an "indifference" to the sacred.
These are, he said, "true enemies for our faith which could create a fertile ground for the eventual future penetration of Islam in all of Europe".
He voiced concern over restrictions to religious freedom in "Muslim areas" and a lack of "reciprocity between Muslim countries and the West", according to the news agency.
Col Gaddafi was in Rome on an official state visit to mark the second anniversary of a friendship treaty signed with Italy, its one time coloniser.
He stirred further consternation when he ended his two-day visit with a farewell speech in which he demanded that the EU pay him £4.1 billion (Approximately $7 billion AUD) a year to stop the illegal immigration of Africans.
He said: "Europe runs the risk of turning black from illegal immigration, it could turn into Africa," he said.
"We need support from the European Union to stop this army trying to get across from Libya, which is their entry point.
"At the moment there is a dangerous level of immigration from Africa into Europe and we don't know what will happen.
"What will be the reaction of the white Christian Europeans to this mass of hungry, uneducated Africans?
"We don't know if Europe will remain an advanced and cohesive continent or if it will be destroyed by this barbarian invasion. We have to imagine this could happen but before it does we need to work together."