With food prices reaching record highs again this year, what goes on inside a 650ft Chicago skyscraper topped by a statue of the goddess Ceres is coming under intense scrutiny.

It is here that the world’s oldest futures and options exchange, the Chicago Board of Trade (CBOT), was established in 1848 to serve the great grain belt that had opened up in the American midwest. And it is here that the international price of agricultural commodities is set to this day.

“There’s a lot of weather in the market, the northern growing season has been traumatic, with drought in Europe and China and tornadoes and floods in the US. No one is panicked yet, but any additional crop loss, say in Russia, will quickly bring new worry to the market and that could quickly turn to panic. We may be one more event away from panic,” Dan Basse, president of AgResource, one of Chicago’s most respected commodity analyst companies, warned as we watched the opening of a day’s trading last month.

G20 agriculture ministers will meet in Paris on 22 June to discuss food security and prices. Speculative activity and how to contain it is high on their agenda.

Debate has been raging since 2008, when price rises provoked riots around the world, about whether or not the new money that has flooded into the commodities markets since 2003 is the cause of the problem – and if so, how to regulate it.

In Chicago, before the financial day begins, teams of traders pump themselves up outside on chain-smoked cigarettes and outsize McDonald’s coffees. The coloured blazers they use to make themselves easily identifiable on the trading floor have been reduced to bright jackets with string-vest backs to counter the heat generated by a day’s speculation. They keep on their toes in training shoes.

Inside, when the bell announces the start, there is a frenzy of noise. Traders yell at one another and wave their arms in violent gesticulation, palms out to signal sell, palms in to signal buy. There are “scalpers” who buy and sell within seconds, “floor brokers” hedging for corporate accounts, and hundreds of runners rushing orders to the recorders.

At the end of May, the price of corn was up again – most traders and analysts expected it to continue rising along with other commodities.

Basse is one of those who thinks underlying fundamentals – a serious mismatch between supply and rapidly growing global demand – are behind this year’s price rises.

“Speculation is the easy thing to point the finger at and it’s easy to fix. Back in 2008, when prices were up and there was lots of money pouring in, that may have pushed prices up, but today we don’t see that as having a significant effect,” Basse said.

“Look at growth in world livestock demand and in biofuels demand, and you can see what’s been driving the agricultural bull market.”

He painted a troubling picture of what is likely to come. He estimates the world needs to bring around 10.3m hectares of new land a year into food production “just to keep stocks steady”, but he says that will be increasingly hard to do as the land that remains available is reduced to what is environmentally fragile.

A “weekend” farmer of GM crops himself, Basse admits the promise that biotech seeds would deliver big increases in yields has turned out to be illusory. He also fears that “superweeds are coming on so fast with GM that US farmers are going to have to go back to more traditional cultivation methods [as opposed to the practice with GM seeds of not tilling the soil and simply spraying to control pests] – but they don’t have the capacity to do that.”

Europe, Basse said, will soon have no choice but to lift its ban on imports of GM crops for animal feed. With its own crops suffering drought, it will have to turn to Brazil, the only major supplier of non-GM imports. However, the Chinese have already bought up large chunks of the Brazilian crop. The policies in the US and the EU of promoting biofuels will be unsustainable.

The company that owns CBOT, the Chicago Mercantile Exchange group (CME), also rejects the notion that the enormous rise in speculation in agricultural commodities in recent years has caused food price rises.

Farmers and processors of physical goods have long used commodities exchanges such as Chicago’s to hedge against risks such as bad harvests. Speculators willing to take the risk perform a useful role in providing liquidity. But much of the recent growth in speculation has been through new “structured” products invented by banks and sold to investors.

After intense lobbying, banks won deregulation of commodities markets in the US in 2000, allowing them to develop these new products. Goldman Sachs pioneered commodity index funds, which offer investors a chance to track changes in a spread of commodity prices including key agricultural commodities.

Between 2003 and 2008, investment in commodity index funds rose from $13bn to $317bn (£193bn). But the CME’s head of product development, Fred Seamon, said: “There is no credible evidence that suggests index funds or any group of traders are a cause for high prices or increased volatility. There may be a correlation, but that’s a completely different thing.”

CME argues that the volume of speculation is not a problem, because the overall composition of the agricultural commodities market has not changed; the increase in activity by index funds has been matched by an increase in trading by those who are commercial participants, that is those who have a direct interest in the physical goods.

“That’s an indefensible position,” Chicago–based hedge fund manager Mark Newell of Quiddity retorted. He and another hedge fund manager, Mike Masters, prepared testimony to the US Senate when it was looking into the effect of speculation on food prices in 2008.

“When billions of dollars of capital is put to work in small markets like agricultural commodities, it inevitably increases volatility and amplifies prices – and if financial flows amplify prices of food stuffs and energy, it’s not like real estate and stocks. When food prices double, people starve ,” Masters said.

The UN rapporteur on the Right to Food, Olivier de Schutter, added his weight to Masters’ side of the debate at the end of last year when he concluded a speculative bubble was responsible for a significant part of the food price rises.

An OECD study, however, did not find a link. Aid agencies such as Oxfam and Christian Aid are calling for reregulation.

In the US, the regulator – the Commodities Futures Trading Commission – has until July to produce a new framework for the commodities markets for Congress. It has been looking at imposing limits on the size of positions that traders can take, and at regulating the commodity index fund trades that are currently unregulated because they take place “over the counter”; that is, between investors and banks. But the financial industry has proved resistant to reforms. G20 ministers will have to decide their own position soon, too.

Newell, meanwhile, remains convinced that without action prices will continue to go up, partly because of underlying fundamentals, but also because, just like in 2008, “the game’s afoot again”.