The simplest method, in this case, would be to use a different delimit identifier to brackets ( [] ) for your column's name:

EXEC sys.sp_rename N'dbo.tbl_m_comp_sysboot_time."[avail_memory_mb]"', N'avail_memory', N'COLUMN';

If you "must" use brackets, then you need to escape the right bracket ( ] ):

EXEC sys.sp_rename N'dbo.tbl_m_comp_sysboot_time.[[avail_memory_mb]]]', N'avail_memory', N'COLUMN';

db<>fiddle

Note that I also removed the database name in both examples. This is because sys.sp_rename doesn't support 4 part naming. Connect to the database first, and then use the system procedure.

If you ever need to know what an object's literal should be, you can use QUOTENAME to find it out as well. For example:

SELECT QUOTENAME(N'[avail_memory_mb]');

Which brings back the nvarchar(258) value [[avail_memory_mb]]] . So, if you really wanted, you could do something like:

DECLARE @ObjectPath nvarchar(1000) = CONCAT(QUOTENAME(N'dbo'),N'.',QUOTENAME(N'tbl_m_comp_sysboot_time'),'.',QUOTENAME(N'[avail_memory_mb]'));
EXEC sys.sp_rename @ObjectPath, N'avail_memory', N'COLUMN';