He fell out with President Hassan Sheikh Mohamud last month after he tried to sack some of the president's allies from the cabinet, BBC reported.
Mohamud took office in 2012 in a UN-backed move to end years of conflict.
Islamist militant group al-Shabab still controls many areas of the country.
The development has brought to an end a dispute between Somalia's two political leaders, which has paralysed government activities for several weeks, the report added.
Following two days of deliberations and debate on a non-confidence motion vote in parliament, 250 out of the 275 lawmakers attended Monday's session.
As many as "184 voted in favour of the motion while 66 voted against. The motion of no-confidence therefore is adopted," reported Xinhua citing Parliament Speaker Mohmaed Osman Jawari announced after the vote.
The president now has 30 days to appoint a new prime minister.
--IANS (Posted on 02-12-2013)