SELECT 
  JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]'), 
  (JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') = CAST('null' AS JSON))
FROM ate.readings_columns_new;

or

SELECT 
  JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]'), 
  (JSON_TYPE(JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]')) = 'NULL')
FROM ate.readings_columns_new;

See the docs for JSON_TYPE.

A bit of a belated answer but I just hit this problem and couldn't find anything reasonably documented. The solution I ended ended up using was the json_type function as 'abl' pointed out above.
The trick was to compare with the string 'NULL' not null or NULL.
As a test throw the following into a mysql prompt and play around with the values
(if using phpMyAdmin don't forget to check 'show this query here again' and 'retain query box' - the universe is frustrating enough without losing edits..)

set @a='{"a":3,"b":null}';

select if(json_type(json_extract(@a,'$.b')) = 'NULL',1,0);

I ended up with the following.

mysql> set @a='{"a":3,"b":null}';
Query OK, 0 rows affected (0.00 sec)

mysql> select if(json_type(json_extract(@a,'$.b')) = 'NULL',1,0);
+----------------------------------------------------+
| if(json_type(json_extract(@a,'$.b')) = 'NULL',1,0) |
+----------------------------------------------------+
|                                                  1 |
+----------------------------------------------------+
1 row in set (0.00 sec)

mysql> set @a='{"a":3,"b":1}';
Query OK, 0 rows affected (0.00 sec)

mysql> select if(json_type(json_extract(@a,'$.b')) = 'NULL',1,0);
+----------------------------------------------------+
| if(json_type(json_extract(@a,'$.b')) = 'NULL',1,0) |
+----------------------------------------------------+
|                                                  0 |
+----------------------------------------------------+
1 row in set (0.00 sec)

As the bare bones of a stored procedure - which is what I needed it for - using the 'if' statements rather than the if() function.

drop procedure if exists test;
delimiter $$

create procedure test(in x json)
begin

if json_type(json_extract(x,'$.b')) = 'NULL' then
  select 1;
else 
  select 0;
end if;

end$$

delimiter;

mysql> call test('{"a":3,"b":1}');
+---+
| 0 |
+---+
| 0 |
+---+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> call test('{"a":3,"b":null}');
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Well I had a suspicion but I found a workaround that confirms that a JSON null value is not the same as a MySQL null value.
I tried various methods to get a similar null value but the only one that works is to extract a null JSON value from an array like the value I'm attempting to check against:

SELECT JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') , (JSON_EXTRACT(`COLUMNS_HEADERS`, '$[1]') = JSON_EXTRACT('[null]', '$[0]'))
FROM ate.readings_columns_new;

This seems like bad form, but was the only way I could get a value that evaluated as equal to the null values in my array.

Another trick is MySQL's NULLIF function

SELECT COLUMNS_HEADERS->>"$[1]", NULLIF(COLUMNS_HEADERS->>"$[1]",'null') IS NULL)

(I'm also using ->> which is an alias for JSON_UNQUOTE(JSON_EXTRACT())
That way querying a column containing {"id":1} , {"id":2} , {"id":null} & {"name":4} for the JSON path $.id will return 1,2,NULL,NULL instead of 1,2,null,NULL