After the House of Representatives took the unprecedented step on Oct. 3, 2023, of removing its own speaker, Kevin McCarthy of California, with eight Republicans joining all 208 voting Democrats to “vacate the chair,” what followed was weeks of uncertainty. Until conservative Louisiana Rep. Mike Johnson was elected speaker of the House on Oct. 25, no candidate had been able to secure the necessary number of Republicans to win a vote on the House floor. And without an elected speaker, the chamber was effectively stuck.

The two proximate reasons for the GOP's struggle to pick and keep a speaker are that it is internally divided and its majority in the House is small. But as a scholar of American politics, I believe the party's problems also stem from long-term pressures that have made it hard for both parties in Congress to exercise the kind of authority they need to govern.

An effective legislative party exercises four kinds of authority. The first is the ability to choose the chamber's top leader and write the chamber's rules — which both form part of what political scientists call organizational authority.

The second kind is procedural authority, or the ability to set the legislative agenda and decide which bills come to the floor for a vote.

The third type of authority is over policy — being able to pass the party's desired bills and amendments on the floor.

Finally, parties have electoral authority, meaning they help their members get reelected and help challengers beat incumbents from the other party.

All four kinds of authority are important, but the first two are especially so. Without them, the minority party in the House could choose the speaker, write the chamber's rules in its favor and bring only the bills it prefers to the chamber floor.

From the 1970s through the mid-1990s, as the House Democratic Party — the majority party at the time — became more liberal, its members worked to expand the party's authority in all four dimensions, hoping to marginalize Republicans and conservative Democrats.

As a result, minority party Republicans had fewer and fewer chances to bring bills and amendments to the floor, and when they did, they usually lost to more liberal proposals. Democrats' election fundraising also grew rapidly, and they gave more attention to recruiting strong candidates to run for office.

Republicans adopted these tactics after they became the majority party in the House in 1995 and elected Newt Gingrich as speaker. Under Gingrich's leadership, the GOP pushed through a package of major conservative bills in the first 100 days of the new Congress, and Democrats were mostly powerless to stop them.

Though Republicans appeared to have a monopoly on power in the House in 1995, their rise to the majority also coincided with significant changes in the political environment that would threaten the authority of both parties.

For one thing, congressional elections became more competitive, giving the minority little incentive to help the majority govern. Smaller margins in the House between the two parties gave the majority party less leeway to allow for defections on floor votes. And new legislators were elected to Congress who were skeptical of leaders and the tradition of party loyalty.

The most obvious sign of fraying party authority was a new willingness of lawmakers, starting in the mid-1990s, to vote against their own party's nominee for speaker. In 2019, Democrat Nancy Pelosi had to engage in intense backroom negotiations to get elected speaker; in January 2023, the House went through 15 rounds of balloting before selecting McCarthy as speaker.

In recent years, other forces have further weakened the authority of the House Republican Party in particular. These include the rise of Fox News as the dominant voice of conservatism and the emergence of outside groups that lobby GOP lawmakers to avoid compromise and act aggressively on behalf of conservative policies.

The election of Donald Trump in 2016 accelerated this trend by turning the Republican Party's base toward angry populism and encouraging some incumbents to openly defy their party in the hopes of tapping into that base.

It also encouraged other conservative mavericks to run for Congress, such as Rep. Lauren Boebert of Colorado, who opposed McCarthy's initial selection as speaker or who otherwise made it hard for him to lead the party.

The current situation has created a serious collective action problem for the GOP, meaning that too many lawmakers think only about their own political needs, even at the risk of hurting the party as a whole. Polls show that the stalemate over choosing a speaker was damaging to the Republican Party's reputation. But so many party members believe they will win the support of conservative voters in their districts or gain more coverage on right-wing media by being intransigent that, until Johnson's election, a speaker candidate had yet to unite the GOP over three weeks of effort.

It's not just conservative populists who are undermining the party's authority. Even though Jim Jordan, an Ohio Republican, had been nominated by the party to be McCarthy's replacement, he gave up after losing three rounds of balloting on the floor. That was thanks to the opposition of institutionally minded Republicans and Republicans from swing districts.

The new speaker, despite having been able to pull together his GOP colleagues, will face the same challenges that McCarthy did. The party's majority will still be small, it will still be divided, and the same forces that have weakened their party's authority will remain in place.

It may thus take changes in the broader political environment — along with an election that either costs the GOP its majority or gives it a larger, more cohesive majority — for the House Republicans to be able to rebuild their collective authority and act as a cohesive and effective legislative party.

The Conversation is an independent and nonprofit source of news, analysis and commentary from academic experts.

Matthew Green is a professor of politics at Catholic University.