The continent is home to half of the world's usable uncultivated land, yet has the highest poverty rate.
But the Bank said farmers' inability to prove ownership, legal disputes and land grabs had held back cultivation.
Land governance needs to be improved if Africa is to fully exploit its resources and create jobs, it said.
Writing in the report, Securing Africa's Land for Shared Prosperity, the Bank's vice-president for the continent, Makhtar Diop, said: "Despite abundant land and mineral wealth, Africa remains poor.
"Improving land governance is vital for achieving rapid economic growth and translating it into significantly less poverty and more opportunity for Africans, including women who make up 70% of Africa's farmers, yet are locked out of land ownership due to customary laws.
Ghana News Headlines
For latest news in Ghana, visit Graphic Online news headlines page Ghana news page
"The status quo is unacceptable and must change so that all Africans can benefit from their land," Mr Diop said.
The report recommends that governments secure tenure rights for communities and individuals, possibly using new and relatively cheap satellite technology to conduct land surveys.
The BBC's international development correspondent, Mark Doyle, says legal wrangles over land ownership are common in Africa.
"It's often the case that plots in urban areas have been divided, then sub-divided through generations of families, creating confusion over who owns what," our correspondent says.
"This makes long-term investment in things like farm machinery or irrigation risky."
The Bank said now was a good time to rationalise land ownership, so Africans could benefit more from surging commodity prices and strong levels of foreign investment.
Improved land management could also prevent land grabs, it said. In recent years, investors from richer countries, outside Africa, have bought millions of hectares of land and are able to claim that it is unoccupied.
But in some cases, the Bank said, established communities had been pushed off their farms.