I have the following table:

CREATE TABLE Test(
ID int,
Category NVARCHAR(MAX),
Value1 float,
Value2 float
)
GO

INSERT INTO Test
VALUES(1,'Category 1',1,4)
,(1,'Category 1',2,3)
,(1,'Category 2',3,5)
,(1,'Category 3',4,2)
,(2,'Category 2',5,1)
,(2,'Category 2',6,3)
,(2,'Category 2',7,6)
,(2,'Category 3',8,3)
,(2,'Category 1',1,2)
,(3,'Category 1',2,4)
,(3,'Category 1',3,5)
,(3,'Category 1',4,7)
,(3,'Category 2',5,8)
,(3,'Category 3',6,9)

I would like to sum Value1 and Value2 by ID and Category and have a different calculation based on the value of the category, while retaining the category value.

For reference the desired result is as follows:

I have tried the following:

SELECT
  ID,
  Category,
  CASE
    WHEN Category = 'Category 1' THEN (SUM(CASE WHEN Category = 'Category 1' THEN Value1 ELSE 0 END) + SUM(CASE WHEN Category = 'Category 2' THEN Value1 ELSE 0 END)) - SUM(CASE WHEN Category = 'Category 3' THEN Value1 ELSE 0 END)
    WHEN Category = 'Category 2' THEN (SUM(CASE WHEN Category = 'Category 1' THEN Value1 ELSE 0 END) - SUM(CASE WHEN Category = 'Category 2' THEN Value1 ELSE 0 END)) * SUM(CASE WHEN Category = 'Category 3' THEN Value1 ELSE 0 END)
    WHEN Category = 'Category 3' THEN SUM(CASE WHEN Category = 'Category 3' THEN Value1 ELSE 0 END)
    ELSE 0
  END AS Value1,
    CASE
    WHEN Category = 'Category 1' THEN (SUM(CASE WHEN Category = 'Category 1' THEN Value2 ELSE 0 END) + SUM(CASE WHEN Category = 'Category 2' THEN Value2 ELSE 0 END)) - SUM(CASE WHEN Category = 'Category 3' THEN Value2 ELSE 0 END)
    WHEN Category = 'Category 2' THEN (SUM(CASE WHEN Category = 'Category 1' THEN Value2 ELSE 0 END) - SUM(CASE WHEN Category = 'Category 2' THEN Value2 ELSE 0 END)) * SUM(CASE WHEN Category = 'Category 3' THEN Value2 ELSE 0 END)
    WHEN Category = 'Category 3' THEN SUM(CASE WHEN Category = 'Category 3' THEN Value2 ELSE 0 END)
    ELSE 0
  END AS Value2
FROM dbo.Test
GROUP BY ID, Category
ORDER BY ID, Category

But I believe the calculation does not work because of the GROUP BY Category which means that it can't find e.g. Category 2 values when the case when is Category 1. Currently the code works using a UNION ALL and selecting the table each time a new calculation is needed, but the hope is that there will be a faster way to do this as the main table is 500k+ rows long and currently the query takes a long time to run as there are many other calculations to perform. Any help would be much appreciated.