Some investors had hoped to sell the much-hyped shares on the issue day to take advantage of pent-up buyer demand.
But technical problems kept many investors from buying shares in the morning or selling them later in the day.
The payout is meant to reimburse those who lost money because of the fault.
The opening of trading was delayed by half an hour and some investors were unable to tell whether their orders were processed, while others said they were left holding shares they did not want.
Facebook's shares went on sale at $38 a share on 18 May, but rose sharply in early trading, something many buyers had hoped to capitalise on by selling during the day.
They ended the day barely above the starting level though, and have floundered since, falling to around $25 a share.
Nasdaq says it will reimburse those who tried to sell into the first-day bounce at $42 or less, but either couldn't sell or sold at a lower price than they intended.
The $40m is more than 10 times the $3m previously paid by Nasdaq for technical errors, but already there have been complaints from some investors it will not be enough to cover losses.
Nasdaq's chief rival, the New York Stock Exchange, has accused Nasdaq of giving itself an unfair advantage, saying the move gives investors an incentive to move more of their trading to Nasdaq.
In a statement it said: "This is tantamount to forcing the industry to subsidise Nasdaq's mis-step and would establish a harmful precedent that could have far-reaching implications for the markets, investors and the public interest."
The exchange added it would "strongly press our views" against the proposal, which has to be approved by the watchdog the Securities and Exchange Commission.