您需要将左手的列括在括号中：

WHERE (c.item_a_id, c.item_b_id) 
      NOT IN (SELECT a.item_id,b.item_id 
              FROM table_a a 
               JOIN table_b b on a.item_id = b.item_id)

但通常NOT EXISTS条件比NOT IN条件更快

WHERE NOT EXISTS (SELECT *
                  FROM table_a a 
                    JOIN table_b b on a.item_id = b.item_id
                  WHERE a.item_id = c.item_a_id 
                    AND b.item_id = c.item_b_id)

如果我理解正确的话，您应该能够对A表和B表执行两个连续的左连接，那么有效的匹配就是两个表都没有任何连接匹配的匹配。

SELECT *
FROM table_c c
LEFT JOIN table_a a
    ON c.item_a_id = a.item_id
LEFT JOIN table_b b
    ON c.item_b_id = b.item_id
WHERE
    a.item_id IS NULL AND
    b.item_id IS NULL;

顺便说一下，上面的查询专门称为左反连接。