In a 23-12 vote, the council backed a Western-sponsored resolution which said it was time for a "comprehensive investigation into alleged serious violations and abuses of human rights and related crimes by both parties in Sri Lanka".
Sri Lanka's ambassador to the UN in Geneva, Ravinatha Aryasinha, slammed the resolution as a "serious breach of international law", telling the council it eroded the sovereignty of his country and would not help the reconciliation process.
The resolution, pushed notably by the United States and European nations, noted that UN rights chief Navi Pillay had herself demanded an "international inquiry mechanism in the absence of a credible national process with tangible results".
Pillay's office will now be tasked with investigating events between 2002 and 2009, the year Sri Lanka crushed a decades-old insurgency by separatist Tamil Tiger rebels.
UN monitors say that thousands of ethnic Tamils died in the army's final offensive that ended the Tigers' 37-year fight for an independent state.
Pillay had told the council on Wednesday that it was crucial to recall the "magnitude and gravity" of the violations allegedly committed by both the government and the rebels, who were known for their trademark suicide bombings.
The 1972-2009 conflict claimed 100,000 lives, according to a UN estimate.
Sri Lankan President Mahinda Rajapakse, who has tightened his grip on power after crushing the Tigers, has argued that he is being unfairly targeted by Western nations.