Dorsey Waxter, the Upper East Side dealer who is president of the Art Dealers Association of America, was approached a few months ago by a collector with a 1950s Sam Francis painting to sell. Waxter has specialized in the artist for decades and proposed a price of just over $2 million. The collector told her an auction house pitched a price of about $3 million. She didn’t take the consignment.

“I couldn’t support the number the auction house was giving,” Waxter said in an interview. “They could go out on a limb. If they can’t sell it, they can take it to auction.” (Waxter declined to name the collector or auction house.)

Private sales are soaring at increasingly versatile auction houses—boosting profits, blurring market distinctions, and stressing dealers. At Sotheby’s, privately negotiated sales increased 30 percent last year, to $1.2 billion including commissions, according to a filing with the Securities and Exchange Commission. Private sales grew at twice the rate of its auctions, generating commissions of $88.2 million. They accounted for 10 percent of revenue, up from 6 percent in 2010.

But a close look at Sotheby’s suggests a possible chink in this fast-expanding territory. Private sale commission margins have contracted, even more than auction margins. Over the past three years, commissions for every $100 of private sales have dropped, from $9.82 in 2010 to $8.13 last year. They’re down 17 percent, versus a 13 percent decline in auction margins. (Sotheby’s doesn’t break out private sales commission margins but artnet calculated them based on publicly disclosed sales and commissions.)

Guests at a March 13 reception for “Shuimo/Water Ink: Enchanted Landscapes,” a “selling exhibition” of contemporary Chinese ink paintings that runs through March 28th at Sotheby’s New York. Rather than being auctioned, the works are being sold privately.

If margins continue to narrow, they could slow the auction house’s profit growth.

A Sotheby’s spokesman declined to comment or make an executive available for this story. Referring to auctions and private sales in a conference call last month, William Ruprecht, the company’s chairman, chief executive and president, said, “Competition is affecting our revenue margins especially for very high-value works.”

Todd Levin, a New York adviser with Levin Art Group, likens privately negotiated sales to the Wild West. “Since the entire transaction takes place away from prying public eyes, all manner of complex deal-making can be structured that might otherwise require a modicum of public awareness and/or greater oversight,” he said.

As at auctions, savvy private sale consignors try to negotiate commissions down, said Thomas C. Danziger, a New York lawyer who specializes in art. “As more people use this as a means of selling, they’re getting more sophisticated,” he said in an interview. “They’re not going to sign onto whatever the printed rate card is.”

Also likely contributing to shrinking margins: the average price per Sotheby’s private sale rose in 2013, according to an SEC filing. Private sale commissions often decline on a percentage basis for pricier art, Danziger said. For example, the New York Times reported that Sotheby’s privately brokered the sale of a newly discovered Leonardo da Vinci painting in May 2013 for more than $75 million. (Sotheby’s declined to comment.)

To broker private sales of all varieties, auction houses deploy their knowledge of what clients covet and own. In the eyes of many dealers, the houses are invading their turf. “It could become an existential threat,” Waxter said. “I wouldn’t discount any possibility.” She said that most—but not all—private sales are for work that’s been sold before. For example, right now Sotheby’s has a “selling exhibition” of Chinese ink landscapes, most of them created in 2013 and 2014, at its New York galleries through March 28. “That’s not the same thing as representing artists,” she said of auction houses selling new work. “They’re cherry picking.”

Waxter said auction houses by their nature can’t replicate the experience and expertise of veteran dealers. Ideally, from an auctioneer’s point of view, an auction house is a dealer with an especially strong global reach; yet at auction there’s always the risk that an artwork will publicly flop and be stigmatized, or “burnt.”

While Ruprecht has called private sales more “episodic” than auctions and thus more difficult to forecast, the influx of new customers is consistent. Marc Porter, chairman of Christie’s Americas and head of international private sales, said a quarter to a third of auction bidders are neophytes. “We are constantly learning about new people in the market,” he said in an interview. (Christie’s is privately held and doesn’t disclose its profits.)

Christie’s and Sotheby’s private sales were about even in 2013, according to company figures. Sotheby’s private sales last year were no longer “squarely behind its main rival,” as investor Dan Loeb wrote in an October critique filed with the SEC. Sotheby’s cited its private sales growth five times in a proxy statement a year ago when disclosing Ruprecht’s 2012 compensation of $6.3 million. Ruprecht took a 10 percent pay cut as profits plunged 37 percent. In 2013, profits rose 20 percent, but a raise isn’t a foregone conclusion, given Loeb’s proxy fight to add three new directors and his attack on Ruprecht’s pay and performance. (Loeb’s Third Point LLC filed a preliminary proxy statement yesterday, March 17.) Sotheby’s typically discloses executive compensation at the end of March.