LOS ANGELES — Nearly four years after Washington bailed out Wall Street, small banks have yet to repay $11 billion of taxpayer money.

Uncle Sam wants out and is threatening to unload its stakes in the banks at big discounts to new investors. Many of the 324 institutions, mostly tiny community banks and niche players, wonder whether they’ll be able to stay in business.

Some stragglers would become financially unstable if they repaid their part of the $245 billion doled out during the financial crisis by the Treasury Department’s Troubled Asset Relief Program.

Critics said the government, having moved so fast to bail out the big banks, should be nurturing the smaller ones.

“Many of these banks feel forced into a situation they can’t control - a potential fire sale with new and perhaps unfriendly shareholders,” said Jerry Comizio, a banking lawyer at Paul Hastings in Washington. “For some banks, though, given their small size and financial condition, they may not have any other viable option.”

The biggest U.S. banks, four of which have more than $1 trillion in assets each, were able to repay TARP in short order. The fact that so many small banks - most with less than $1 billion in assets each - cannot do so illustrates how the struggling economy has crushed business strategies that often relied on housing-related loans and small-business mortgages.

The Treasury Department bought preferred stock in the banks on the condition that they compensate the government by paying a 5 percent dividend for five years, which then jumps to 9 percent late next year or in 2014. Counting the dividend payments, the government has turned a $19 billion profit overall on its bank bailouts.

Nonetheless, nearly half the 707 banks that received TARP funding have yet to repay their debt, including 164 that at last count had missed one or more dividend payments because regulators have restricted their operations.

The Treasury Department is encouraging the banks to buy back their own stock, which some are planning to do. But most are required to clean up soured loan portfolios, strengthen management and raise new capital privately before regulators will allow them to buy their own stock or even to pay dividends.

Bankers in the state said they were struggling to find cash from friendly new investors at prices that wouldn’t wipe out the existing shareholders, often people with deep ties to their communities. The government has signaled that it hopes to dump all its investments within 18 months.

“There’s no capital out there for little banks, no capital for inner-city banks and no capital for banks with a lot of troubled assets,” said Wayne Kent-Bradshaw, an executive with Broadway Financial Corp. in Los Angeles.

The three-branch bank, founded in 1947 by African-Americans to help blacks who had been shut out by mainstream banks, “is all of those,” Bradshaw said.

In a bid to keep the company’s Broadway Federal Bank in business, the Treasury Department has agreed to trade its dividend-paying stock for non-dividend shares at a 50 percent discount. But the deal is on only if Broadway raises $5 million in new capital and persuades other preferred stockholders to take a haircut.

Bradshaw said investors have pledged $3.5 million, and sale of the bank’s headquarters yielded $1.5 million after taxes - a total of $5 million. But he said the Treasury Department may still insist that all new funding come from investors.

“We’re negotiating with everyone, including Treasury, to get this worked out,” he said.

Some small banks have kept the funds by choice, biding time until the dividend rises. Others have agreed to takeovers by larger rivals in deals that repay the government.

Timothy Massad, the assistant Treasury secretary for financial stability, said deals such as the one with Broadway are struck only if they represent the best outcome for taxpayers.

“Some of those institutions were very close to failing,” Massad said, which would wipe out the government’s entire investment.

More common are the discounted sales. In the past three months, the Treasury Department has sold its stake in 20 institutions, with recoveries ranging from 74 percent to 98 percent of its original investments. The next such auction will be held in late July.

The department also plans to unload its stock in about 200 smaller banks this fall by combining shares of various institutions in pools for sale. Bidders are likely to include hedge funds and other specialists in disposal of troubled assets.

Massad described the process as efficient for the government and beneficial for the banks.

“It helps them attract new, private capital to replace temporary government support, while ensuring that taxpayers get the best price,” he said. “The government is able to exit its stake, while the bank can continue to keep the capital on its books, which helps them support lending to local families and businesses.”

The banks can forestall the sales if they buy back their own stock for what the Treasury Department’s consultants deem to be a reasonable minimum price.

But most of the banks are still so weak that regulators won’t allow them to deplete their capital by buying back their shares, said Gary Findley, a small-bank consultant in Anaheim, Calif. He expects a consolidation trend to continue, with the department’s actions encouraging more takeovers.

Some key observers fret over the prospect of investors aiming to break up or resell the banks at a quick profit.

“A key part of our recovery is getting community banks back on their feet,” said Christy Romero, the government’s special inspector general for TARP. She suggested that the Treasury Department consider extending its support to at least some small banks, perhaps by not raising the dividend after five years.

“These guys don’t have Warren Buffett handing them tons of money,” Romero said, a reference to the $5 billion that the Omaha billionaire invested in Bank of America Corp. in August.

Exiting TARP has been profitable for some, such as Wilshire Bancorp in Los Angeles. The bank was the winning bidder in March when the Treasury Department auctioned its $60 million in Wilshire shares, buying them back for $56.6 million, about 94 cents on the dollar.

“It was a very good opportunity for us,” said Alex Ko, Wilshire’s chief financial officer.

The deal raised eyebrows at nearby BBCN Bancorp, the largest bank in the Korean American niche, which repaid its $122 million debt to the government in full June 27.

“We got no discount,” said Angie Yang, BBCN’s senior vice president for investor relations. “There’s just no rhyme or reason how they’re picking these banks to be auctioned.”