The 20-year-old pop singer donned a nude pvc bikini for her routine, in which she danced suggestively with singer Robin Thicke and made crude sexual gestures.
The Parents Television Council said "heads should roll at MTV" for giving the performance the greenlight, despite a large chunk of the show's global audience including young teenagers.
The complaint also questioned the decision to air a condom advert during the show.
"This much is absolutely clear: MTV marketed adults-only material to children while falsely manipulating the content rating to make parents think the content was safe for their children," Dan Isett, the PTC's director of public policy said.
"MTV continues to sexually exploit young women by promoting acts that incorporate 'twerking' [a gyrating dance move] in a nude-coloured bikini.
"How is this image of former child star Miley Cyrus appropriate for 14-year-olds?"
Cyrus's performance stole the limelight from other acts at the show as media pundits and celebrities waded in to the discussion over whether the Hannah Montana star took things too far.
Robin Thicke's mother Gloria Loring told Yahoo!: "I just keep thinking of her mother and father watching this. Oh, Lord, have mercy.
"I was not expecting her to be putting her butt that close to my son. The problem is now I can never 'unsee' it.
"I don't understand what Miley Cyrus is trying to do. I just don't understand. I think she's misbegotten in this attempt of hers.
"And I think it was not beneficial."
Brooke Shields, who played Miley's mom in the Hannah Montana show, said the performance was "desperate".
She told the Today show: "I just want to know who's advising her, and why [twerking] is necessary ... [children] can't watch that.
"I feel like it's a bit desperate."
She said she felt Cyrus was "trying so hard" to go against her previously squeaky clean Disney image.
"She can sing beautifully and I feel like if she lets that lead, rather than let her bottom lead ... She's trying so, so hard," said Shields.
Entertainment news site E! Online tweeted after the performance: "Raise your hand if you feel personally victimized by Miley Cyrus' #VMAs performance."
Dawson's Creek star James Van Der Beek tweeted: "Things I learned watching the #VMAs2013: There's nothing you can do with a foam finger that you can't air on MTV."
Her dad, country music star Billy Ray Cyrus, however didn't seem so perturbed by his daughter's performance, tweeting on Monday: "Thanking God for so many blessings tonight. Continue to pray for world peace. More love ...less hate."