An energy price shock is serving as a reminder of the world’s continued dependency on fossil fuels—even amid efforts to shift to renewable sources of energy.

Demand for oil, coal and natural gas has skyrocketed world-wide in recent weeks as unusual weather conditions and resurgent economies emerging from the pandemic combine to create energy shortages from China to Brazil to the U.K.

The...

An energy price shock is serving as a reminder of the world’s continued dependency on fossil fuels—even amid efforts to shift to renewable sources of energy.

Demand for oil, coal and natural gas has skyrocketed world-wide in recent weeks as unusual weather conditions and resurgent economies emerging from the pandemic combine to create energy shortages from China to Brazil to the U.K.

The situation has laid bare the fragility of global supplies as countries drive to pivot from fossil fuels to cleaner sources of energy, a shift many investors and governments are trying to accelerate amid concerns about climate change.

The transition figures to be challenging for years to come, energy executives and analysts say, due to a stark reality: While fossil fuel investment is falling, fossil fuels account for most energy—and green energy spending isn’t growing fast enough to fill the gap.

Demand for power remains robust even as supply chains begin to strain. In some cases, supplies of renewable resources such as wind and hydroelectric power have fallen short of forecasts, further boosting demand for fossil fuels.

The International Energy Agency, a group that advises countries on energy policies, this month projected global oil demand will reach about 99.6 million barrels a day next year, near pre-pandemic levels. It forecasts that coal demand is set to exceed 2019 levels this year and rise somewhat until 2025, though how quickly it falls from there will depend on government actions to phase out the fuel.

“A lot less product is available to meet this now rapid growth we’re seeing,” Exxon Mobil Corp. Chief Executive Darren Woods said in virtual remarks at a conference in Russia Wednesday. “If we don’t balance the demand equation and only address the supply, it will lead to additional volatility.”

The world’s oil production is still rising, but struggling to catch up with a surge in consumption from countries recovering from the pandemic, according to the U.S. Energy Information Administration.

Oil investments dry up

Global oil and gas exploration spending, excluding shale, averaged about $100 billion a year from 2010 to 2015, but dropped to an average of around $50 billion in the years that followed after a crash in crude prices, according to Rystad Energy.

Total global oil and gas investment this year will be down about 26% from pre-pandemic levels to $356 billion, the IEA said Wednesday. That is about where it would need to remain for the next decade, before declining further, to meet the goals of the Paris agreement, according to the IEA. The international pact seeks to limit global temperature increases to no more than two degrees Celsius from preindustrial levels, and preferably 1.5 degrees.

To meet global energy demand, as well as climate aspirations, investments in clean energy would need to grow from around $1.1 trillion this year to $3.4 trillion a year until 2030, the Paris-based agency found. Investment would advance technology, transmission and storage, among other things.

“The world isn’t investing enough to meet its future energy needs, and uncertainties over policies and demand trajectories create a strong risk of a volatile period ahead for energy markets,” the IEA report said. It added that ramping up renewables would require greatly enhanced spending in other sectors, such as mining, to produce and refine the raw materials needed for wind turbines, solar arrays and utility-scale battery storage.

Romney Marsh Solar Farm in the U.K., which provides enough power for around 6,000 households.

Photo: Gareth Fuller/Zuma Press

The development of wind and solar farms and other renewable power sources has accelerated within the past two decades as the technologies have dropped in costs due to economies of scale, becoming more competitive with fossil-fuel based electricity generation. Global renewable energy capacity, excluding hydropower and pumped storage, topped 1.5 million megawatts last year, according to the International Renewable Energy Agency, up from less than 55,000 megawatts in 2000.

Greener sources have gained market share in the U.S. and Europe, aided by government subsidies and other policies aimed at reducing the use of coal, the dirtiest fossil fuel. In 2019, before the onset of the pandemic, the U.S. consumed more renewable energy than coal for the first time since 1885.

That growth is expected to continue. The world added 280,000 megawatts of renewable electricity last year, up 45% from the prior year, according to the IEA. The agency called that growth rate “the new normal” and expects similar amounts to be added this year and next year.

Still, fossil fuels make up the majority of power generation globally. Renewables accounted for 26% of global electricity generation in 2019, according to IRENA.

Heading to Glasgow

World leaders set to gather for a major climate change conference in Glasgow in two weeks are aiming to accelerate the transition to cleaner energy to reduce greenhouse gas emissions. But they are still grappling with core questions that have complicated such negotiations for decades, including whether richer countries will pay to help poorer countries make the shift.

Supply-chain issues also constrain how quickly the world can increase wind and solar power. Most solar arrays are currently produced with energy from coal-fired power plants in China, which supplies more than three fourths of the world’s polysilicon. Some Western governments and companies are attempting to shift solar manufacturing away from coal, but that threatens to drive up solar costs.

In addition to greening the power grid, many countries are advancing policies to speed a shift to electric vehicles. That is poised to reduce the amount of oil used in transportation, which currently makes up around 60% of oil demand according to the IEA. But while nearly all major auto makers including General Motors Co. and Volkswagen AG are betting big on EV production, and sales are gaining traction, adoption is expected to be gradual.

A Volkswagen assembly line producing the Volkswagen ID.3 electric car.

Photo: ronny hartmann/AFP/Getty Images

In Europe, which experienced a decline in power generation partly due to an unusual slowdown in offshore wind speeds, natural-gas prices have almost tripled in three months, leading some fertilizer makers to halt production because they can no longer produce it economically. In China, electricity shortfalls caused by high coal prices led local officials to curtail hours at some factories, affecting production of semiconductors and other key exports.

The U.S. has been less affected than other countries, but it too has seen higher prices, and concerns about further increases in winter are mounting. On Wednesday, the U.S. Energy Information Administration warned that the nearly half of American households that primarily warm their homes with natural gas will spend an average of 30% more on their bills compared with last year.

Prices of Brent crude, the global benchmark, topped $85 a barrel Friday, their highest level in three years. Traders are betting prices will continue to rise, stoking a roaring options market.

One factor weighing on crude is that gas and coal shortages are pressuring some power plant operators and manufacturers to burn oil instead.

Saudi Arabian Oil Co., known as Aramco, said this month that it plans to increase its oil production capacity from 12 million to 13 million barrels a day by 2027. Rival Abu Dhabi National Oil, the main oil producer in the United Arab Emirates, said it would be spending $122 billion in part to boost its oil production capacity to five million barrels a day by the end of the decade, from about four million a day today.

Overall, OPEC estimates the world is projected to require $11.8 trillion in oil-and-gas investment through 2045 to meet growing demand. In a report released last month, it predicted its members’ oil will constitute 39% of global crude consumption in 2045, up from about 33% now.

“We are witnessing strains and conflicts related to energy affordability, energy security and reducing emissions,” said Mohammed Barkindo, OPEC’s secretary-general, in an interview last month.

California’s rocky transition

Governments trying to fast-track a transition to cleaner sources of energy are finding that it requires vast amounts of investment, and can meet unexpected obstacles. In the U.S., California is in the midst of retiring numerous fossil-fuel power plants to help decarbonize its power grid by 2045, as a state law requires.

The California Public Utilities Commission has ordered utilities to buy an unprecedented amount of renewable energy, battery storage and other carbon-free resources to fill the gap and keep up with growth in coming years: more than 14,000 megawatts, or roughly a third of the state’s forecast for peak summer demand.

While the companies are on track thus far, the California Energy Commission and the state’s grid operator recently expressed concern that the purchases may not be enough to prevent electricity shortages in the coming summers. The state is also planning to retire its last nuclear power plant, Diablo Canyon, which generates nearly 10% of the electricity in the state, by 2025.

California has narrowly avoided rolling blackouts this year, amid wildfires that have disrupted power transmission and a severe drought that has reduced hydroelectric production throughout the West, including from the Hoover Dam.

The Hoover Dam Lookout above the Colorado River, during low water levels.

Photo: Roger Kisby/Bloomberg News

The state’s power grid operator has called on residents to conserve power several times this summer and took emergency measures to buy additional supplies to reduce the risk of blackouts. The state also recently added four temporary natural-gas generators at power plants to help alleviate the shortage.

‘Long, long goodbye’

After years of losing money on America’s shale oil boom, which produced ample supplies but few profits, investors and Wall Street financiers alike are clamoring for companies to limit investment in future projects and return cash to them instead.

That push has stunted growth in all but one of the oil fields that fueled the shale boom. Companies like Continental Resources Inc. in North Dakota’s Bakken field and EOG Resources Inc. in South Texas’ Eagle Ford shale kicked off the boom back when oil often fetched more than $100 a barrel. But producers in those regions have drilled up some of the most prolific land and are running into limitations juicing as much oil from new wells in maturing fields. This year, oil output for almost all of the 20 largest producers in both the Eagle Ford and Bakken has stayed below pre-pandemic levels, according to data from ShaleProfile, an industry analytics platform.

“We’re starting the long, long goodbye,” Bob Fryklund, a strategist at IHS Markit, said of the Bakken and Eagle Ford plays.

Oilfield pump jacks in North Dakota draw crude oil from a well in the Bakken Formation.

Photo: Bayne Stanley/Zuma Press

The only place in the contiguous U.S. that shale companies are growing is the Permian Basin in West Texas and New Mexico. But even there, production has yet to fully recover, and the companies dispatching more drilling rigs are typically smaller, private operators that don’t have the muscle to lift output significantly. Larger, publicly traded producers have restrained Permian activity, and only eight of the 20 top producers had output above March 2020 levels as of July, the latest ShaleProfile data show.

“We’ll be lucky to grow 5% a year over the next several years in the Lower 48 [U.S. states],” Scott Sheffield, chief executive of Pioneer Natural Resources Co., the largest producer in the Permian, told investors in August.

Alaskan oil production is also in sharp retreat. Many of the largest Western oil companies have pulled back from Alaska, including BP PLC, which sold its North Slope properties to closely held Hilcorp Energy in 2020 for $5.6 billion.

Last year, production in Alaska fell to an average of 448,000 barrels a day, a 20-year-low, according to the EIA. Though vast reserves of untapped oil and gas remain in the state, a combination of forces are constraining investment. One of the most significant is a lack of financing. Under pressure from environmental groups, the six largest U.S. banks have pledged in recent years not to finance additional Arctic drilling.

Drill rigs and oil field operations near Deadhorse Airport in Prudhoe Bay, Alaska.

Photo: Daniel Acker/Bloomberg News

Wildcatter Bill Armstrong, founder of Armstrong Oil & Gas Inc., which made one the largest oil finds in U.S. history in the North Slope in 2013, argues that as long as crude demand remains strong, investors’ pullback from Alaska will only result in development in countries with less stringent regulations.

Mr. Armstrong sold his interest in the find, known as the Pikka unit, to Oil Search Ltd. for $850 million. The company, which agreed to merge with Australia’s Santos Ltd. in August, has said it had trouble lining up bank financing and is years away from production.

“It’s just been sitting there,” Mr. Armstrong said of the discovery. “It’s like its own OPEC country once it gets online. But we kind of need it now.”

—Katherine Blunt contributed to this article.

Write to Christopher M. Matthews at christopher.matthews@wsj.com, Collin Eaton at collin.eaton@wsj.com and Benoit Faucon at benoit.faucon@wsj.com