Boris Johnson is facing calls from senior Tories to resign after he admitted attending a drinks party during lockdown.
The prime minister apologised for the way he handled the event in the Downing Street garden in May 2020 and said he understood the public's "rage" over it.
Cabinet members including deputy PM Dominic Raab rallied round Mr Johnson.
But Scottish Tory leader Douglas Ross and MPs William Wragg, Caroline Nokes and Roger urged him to go.
Mr Ross, an MP and a Member of the Scottish Parliament, said he had had a "difficult conversation" with Mr Johnson after he apologised on Wednesday in the House of Commons.
He said he would write to the 1922 Committee, which organises Conservative leadership contests, to register his lack of confidence in the prime minister.
"He is the prime minister. It is his government that put these rules in place, and he has to be held to account for his actions," Mr Ross said.
Commons leader Jacob Rees-Mogg said those calling for Mr Johnson were "people who are always unhappy" and dismissed Scottish Conservative leader Douglas Ross as "quite a lightweight figure".
Conservative MP Andrew Percy criticised Mr Rees-Mogg, saying: "As someone who apparently loves the Union, his personal attack on Douglas... is a gift to the petty nationalists in the SNP who want to break this country up."
A minimum of 54 Conservative MPs must send letters to the committee in order to trigger a leadership challenge.
The drinks gathering, held on 20 May 2020 and described in the invitation as "socially distanced", was attended by around 30 people, who were invited to bring their own alcohol. Food, including sausage rolls and crisps, was reportedly laid out on trestle tables,
Mr Johnson admitted at Prime Minister's Questions on Wednesday that he had joined colleagues at the event for around 25 minutes to "thank groups of staff" for their hard work during the pandemic, but had "believed implicitly that this was a work event".
He added: "With hindsight I should have sent everyone back inside. I should have found some other way to thank them, and I should have recognised that - even if it could have been said technically to fall within the guidance - there would be millions and millions of people who simply would not see it that way."