Imagine there’s no recession. It’s easy if you try.

But imagining an easy path ahead for the stock market is a lot harder.

People have been worrying the U.S. economy is about to enter a downturn for a while now, but what is notable is that it hasn’t happened yet. The economy is still growing, the job market is strong, and at least so far the travails of big tech companies and the fallout from the failures of Silicon Valley Bank, Signature Bank and, lately, First Republic haven’t changed that.

“The case of avoiding a recession is, in my view, more likely than that of having a recession,” Federal Reserve Chairman Jerome Powell said during his press conference following the central bank’s decision to raise rates Wednesday.

What might that look like? First and most obviously, the troubles confronting regional and community banks need to be contained, and the looming debt-ceiling fight between House Republicans and the White House resolved. More broadly, job growth needs to moderate without unemployment spiking, with inflation getting into the Fed’s comfort zone and the economy still growing.

So far, so good. The Labor Department on Friday reported that the economy added 253,000 jobs last month—the latest indication that, while the labor market has slowed, employment is still growing solidly. Inflation, while still high, has begun to cool. Household balance sheets look strong, with the savings people accumulated in the early stages of the pandemic providing them with a buffer against hard times. Businesses that benefited from the pandemic, such as many big tech companies and those in the business of making, transporting and selling goods, are experiencing softening demand, but many in the service sector appear to be doing well, as people continue to re-engage with prepandemic activities.

In the no-recession scenario, services might keep taking up the slack from the goods sector, bringing the two sectors’ share of the consumer spending back to something like 2019 levels. If that were to happen all at once, spending on services would be 3.8% higher than in the first quarter, and spending on goods would be 7.6% lower.

It won’t happen all at once, but it seems reasonable to expect that demand for goods will grow much more slowly than the overall economy. That wouldn’t be good news for the stock market as a whole, because the companies in the stock market are much more geared toward goods than the economy. For example, manufacturers and retailers accounted for about half of S&P 500 sales last year, according to FactSet estimates. But those two industries only accounted for less than one-quarter of overall U.S. private-sector sales, as measured by gross output, according to the Commerce Department.

Moreover, in this scenario, demand for workers from services businesses would keep overall wage growth firmer than in a recession, so goods-oriented companies would have a harder time cutting labor costs, with profit margins getting pressured as a result.

If there isn’t a recession, the Fed also wouldn’t have reason to sharply cut rates. In that case, long-term interest rates likely wouldn’t fall from where they are now, and arguably might head higher. With its rate increase on Wednesday, the central bank brought its target on overnight rates to a range of 5% to 5.25%. In contrast, the yield on the 10-year Treasury, which should reflect investors’ assessment of overnight rates’ average level over the next decade, is about 3.5%. Higher long-term rates would, of course, likely entice many investors to put more of their money into bonds, and less into stocks.

Everybody should be hoping Mr. Powell is right, and the U.S. avoids a recession. Even if it does, though, stocks could still be in for a struggle.