We have our secrets stored in AWS Secrets manager. Hence I figured, let use Connection() to create connection to MsSQL and then use the Hook to execute SQL query.
c = Connection(
conn_id="test_mssql_1",
conn_type="mssql",
description="connection description",
host="**************",
login="**************",
password="*****************",
schema="***********"
#extra=json.dumps(dict(this_param="some val", that_param="other val*")),
)
mssql_hook = c.get_hook()
data_records = mssql_hook.get_records("Select top 10 * from SomeTable")However when executing this query, I get an error
airflow.exceptions.AirflowNotFoundException: The conn_id `test_mssql_1` isn't definedAny thoughts on what is incorrect or what should be the right approach ? This is the first foray inot COnnection() call. We dont want to use Connections from UI since our credentials come from Secrets Manager. Thanks
1条答案
按热度按时间5rgfhyps1#
You never want to have secret information such as passwords in code for security reasons. When you call
get_hook(), Airflow will look up the given connection id in Airflow's database. The error you're seeing says "I tried looking up connectiontest_mssql_1in the database but couldn't find it".A better way to do this is to save the information as a "Connection" in the Airflow UI under Admin -> Connections, and then you can fetch the credentials at runtime by referencing the Connection id as:
Since you mention you're using AWS Secrets Manager, you might also want to look at "Secrets Backends" in Airflow, which enable you to automatically fetch credentials from other systems than Airflow's database: