Generate all three pairs of values and then rank them according to their differences. Tiebreaker is to sort by the value of the larger number:

with data as (
    select *,
        row_number() over (
            partition by RegdNo
            order by
                abs(v1 - v2),
                case when v1 > v2
                     then v1 else v2 end desc
        ) as rn
    from FinalMarks cross apply (values 
        ("1val", "2val"),
        ("1val", "3val"),
        ("2val", "3val")
    ) as v(v1, v2)
) 
select *, (v1 + v2) * 1.0 / 2
from data where rn = 1;

With exactly 3 values you can "sort" them using some algebra on aggregate results. You can then easily determine whether the middle value is left or right of the midpoint of the interval. This is mostly for fun although it doesn't require analytic functions, which might make it useful on older systems:

with sorted as (
    select RegdNo, min(Subject_code) as Subject_code, 
        min(v) as v1,
        cast(round(avg(v) * 3, 0) as int) - min(v) - max(v) as v2,
        max(v) as v3
    from FinalMarks cross apply (values 
        ("1val"), ("2val"), ("3val")
    ) as v(v)
    group by RegdNo
)
select *,
    case when sign(v1 + v3 - v2 * 2) = 1
        then v1 + v2 else v2 + v3 end * 1.0 / 2
from sorted;

https://dbfiddle.uk/fo5z5hfr

BONUS

Here's a fairly concise expression that also works. It's based on finding the two values that are on the same side of the trio's average. The symmetry in the expression is easier to follow than a long series of inequalities:

select *,
    case  
        when s1 = s3 then v1 + v3 when s2 = s3 then v2 + v3 when s3 = s1 then v3 + v1
        -- else there must have been a tie
        when s1 = 1  then v2 + v3 when s2 = 1  then v1 + v3 when s3 = 1  then v1 + v2
    end * 1.0 / 2
from FinalMarks
  cross apply (values("1VAL"+"2VAL"+"3VAL", "1VAL", "2VAL", "3VAL")) e1(vt, v1, v2, v3)
  cross apply (values(sign(vt - v1*3), sign(vt - v2*3), sign(vt - v3*3))) e2(s1, s2, s3);

It's a lot of typing, but doing the comparisons for every possible variation (and hoping no-one will ask this to be done for more than 3 numbers 😉):

select 
   RegdNo,
   Subject_code,
   [1VAL],
   [2VAL2],
   [3VAL3],
   Result
from (
   select
      RegdNo, Subject_code, [1VAL], [2VAL2], [3VAL3], A, B, A1, B1, C, ABS((A+B)/2) Result,
      ROW_NUMBER() over (PARTITION BY RegdNo ORDER BY C , GREATEST(A,B) DESC) as R
   from (
      SELECT
         RegdNo, Subject_code, [1VAL], [2VAL2], [3VAL3], 
         [1VAL] as A, [2VAL2] as B, 1 as A1, 2 as B1,
         GREATEST([1VAL],[2VAL2])-([1VAL]+[2VAL2])/2  as C
      FROM
         FinalMarks

      UNION ALL

      SELECT
         RegdNo, Subject_code, [1VAL], [2VAL2], [3VAL3],
         [1VAL] as A, [3VAL3] as B, 1 as A1, 3 as B1,
         GREATEST([1VAL],[3VAL3])-([1VAL]+[3VAL3])/2 
      FROM
         FinalMarks

      UNION ALL

      SELECT
         RegdNo, Subject_code, [1VAL], [2VAL2], [3VAL3],
         [2VAL2] as A, [3VAL3] as B, 2 as A1, 3 as B1,
         GREATEST([2VAL2],[3VAL3])-([2VAL2]+[3VAL3])/2  
      FROM
         FinalMarks
   ) x
) y
WHERE y.R=1

Results:

You can use a combination of LEAST() and GRETEST() functions. Here is a snippet.

Calculate avg of first two vals

SELECT (1VAL + 2VAL2) / 2 as Avg12 FROM table_name;

Calculate avg of second and third vals

SELECT (2VAL2 + 3VAL3) / 2 as Avg23 FROM table_name;

Calculate absolute diff between the first and two averages.

SELECT ABS(1VAL - Avg12) as Diff12, ABS(1VAL - Avg23) as Diff23 FROM (
SELECT 1VAL, (1VAL + 2VAL2) / 2 as Avg12, (2VAL2 + 3VAL3) / 2 as Avg23 FROM table_name

) as subquery;

and finally compare results

SELECT
    RegdNo,
    Subject_code,
    1VAL,
    2VAL2,
    3VAL3,
    CASE
        WHEN Diff12 <= Diff23 THEN Avg12
        ELSE Avg23
    END as Nearest_Average
FROM (
    SELECT
        RegdNo,
        Subject_code,
        1VAL,
        2VAL2,
        3VAL3,
        (1VAL + 2VAL2) / 2 as Avg12,
        (2VAL2 + 3VAL3) / 2 as Avg23,
        ABS(1VAL - ((1VAL + 2VAL2) / 2)) as Diff12,
        ABS(1VAL - ((2VAL2 + 3VAL3) / 2)) as Diff23
    FROM table_name
) as subquery;

I hope this helps.

Taking the latest available information from the question it seesm you want to average all 3 numbers, then choose the 2 largest values closest to that average, then average the 2 that are chosen. I believe this meets the test cases provided. Note the syntax below works in SQL Server.

CREATE TABLE FinalMarks (
    RegdNo INT, Subject_code VARCHAR(10), VAL1 INT, VAL2 INT, VAL3 INT
    )
  ;

INSERT INTO FinalMarks (RegdNo, Subject_code, VAL1, VAL2, VAL3)
VALUES
      (1, 'MATH 25', 10, 20, 30) -- expect 25
    , (2, 'MATH 29', 28, 44, 30) -- expect 29
    , (3, 'MATH 60', 55, 45, 65) -- expect 60
    , (4, 'MATH 42', 55, 39, 44) -- expect 42
    , (5, 'MATH 51', 58, 44, 30) -- expect 51
    ;
/*
(20+30)/2 = 25
(28+30)/2 = 29
(55+65)/2 = 60
(39+44)/2 = 42 (41.5)
*/

4 rows affected

WITH CTE AS (
    SELECT
        RegdNo,
        Subject_code,
        VAL1,
        VAL2,
        VAL3,
        (VAL1 + VAL2 + VAL3) / 3.0 AS AvgAll
    FROM FinalMarks
)
SELECT
    RegdNo,
    Subject_code,
    ROUND((SELECT AVG(VAL*1.0)
           FROM (
               SELECT TOP 2 VAL
               FROM (VALUES (VAL1), (VAL2), (VAL3)) AS T(VAL)
               ORDER BY ABS(VAL - AvgAll)
           ) AS T), 1) AS AvgClosest
FROM CTE;

fiddle

Here is a different syntax for the same results (for SQL Server):

SELECT
    RegdNo,
    Subject_code,
    ROUND(AvgClosest, 1) AS AvgClosest
FROM FinalMarks
CROSS APPLY (
    SELECT AVG(VAL*1.0) AS AvgClosest
    FROM (
        SELECT TOP 2 VAL
        FROM (VALUES (VAL1), (VAL2), (VAL3)) AS T(VAL)
        ORDER BY ABS(VAL - (VAL1 + VAL2 + VAL3) / 3.0)
    ) AS T
) AS T;

and that can be used as a lateral join instead of cross apply , e.g. in Postgres:

SELECT
    RegdNo,
    Subject_code,
    ROUND(AvgClosest, 1) AS AvgClosest
FROM FinalMarks,
LATERAL (
    SELECT AVG(VAL*1.0) AS AvgClosest
    FROM (
        SELECT VAL
        FROM UNNEST(ARRAY[VAL1, VAL2, VAL3]) AS T(VAL)
        ORDER BY ABS(VAL - (VAL1 + VAL2 + VAL3) / 3.0)
        LIMIT 2
    ) AS T
) AS T;