Your query is invalid. In the main query you group by group_id only, but select a player_id and totalScores. Which player_id? Which totalScores? There can be many per group_id. MySQL should raise an error here, and if it doesn't this indicaes that you are working in MySQL's notorious cheat mode that silently applies the ANY_VALUE function on such columns. Don't do that. You get arbitrarily chosen values that can even stem from different rows. In MySQL, always

SET sql_mode = 'ONLY_FULL_GROUP_BY';

in order not to be allowed to write invalid queries.
Apart from that, you have an ORDER BY clause in a subquery. A subquery cannot be ordered, and in fact, some DBMS would raise an error here. Subquery results are unordered sets of data by definition.
If run in MySQL cheat mode, your query selects one row per group_id, each which an arbitrarily chosen player_id and and arbitrarily chosen total score. This is not what you want.
Here is a query that solves the task properly. It joins the tables as in your query, but then ranks the players by group_id with ROW_NUMBER . At last it shows the highest ranked player per group.

with scores as
(
  select
    p.group_id,
    p.player_id,
    coalesce(sum(case when player_id = first_player then first_score else second_score end), 0) as score
  from players p
  left join matches m on p.player_id in (m.first_player, m.second_player)
  group by p.group_id, p.player_id
)
, ranked as
(
  select group_id, player_id,
    row_number() 
      over(partition by group_id order by score desc, player_id) as rn
  from scores
)
select group_id, player_id
from ranked
where rn = 1
order by group_id;

(You could rank right away, but I thought it more readable to build the sums first and then rank in a separate step.)
As you say you don't want to join (which seems a very weird request) and you only need the players' total scores from the matches table, you can replace the outer join by a subquery in the players' select clause:

with scores as
(
  select
    group_id,
    player_id,
    (
      select coalesce(sum(case when player_id = first_player then first_score else second_score end), 0)
      from matches m
      where p.player_id in (m.first_player, m.second_player)
    ) as score
  from players p
)

Demo: https://dbfiddle.uk/PxP_gMXI