Created
June 13, 2018 16:39
-
-
Save cupdike/935cec4d82db45715876e3d4b8b2f758 to your computer and use it in GitHub Desktop.
Airflow Beeline Connection Using Kerberos via CLI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### There aren't many good examples of how to do this when also using kerberos | |
(venv) [airflow@cray01 dags]$ airflow connections --add \ | |
--conn_id beeline_hive \ | |
--conn_type 'beeline' \ | |
--conn_host 'myserver.mydomain.com' \ | |
--conn_port 10000 \ | |
--conn_extra '{"use_beeline": true, "auth":"kerberos;principal=mysvcname/[email protected];"}' | |
### Then, a sample DAG to use it | |
import time | |
import airflow | |
from airflow.operators.hive_operator import HiveOperator | |
from airflow.models import DAG | |
args = { | |
'owner': 'airflow', | |
'start_date': airflow.utils.dates.days_ago(2) | |
} | |
dag = DAG( | |
dag_id='HelloHiveDag', | |
default_args=args, # supplied to all operators, accessible directly as properties on the task instance | |
schedule_interval=None) # only run via cli | |
t1 = HiveOperator( | |
hive_cli_conn_id='beeline_hive_bd', | |
task_id='HelloHiveTask', | |
hql= """ | |
USE DEFAULT; | |
CREATE TABLE IF NOT EXISTS employee (eid int, name String, salary String, destination String) | |
ROW FORMAT DELIMITED | |
FIELDS TERMINATED BY '\t' | |
LINES TERMINATED BY '\n' | |
STORED AS TEXTFILE; | |
""", | |
dag=dag) |
My recollection is that you add the connection via the CLI and then it can be accessed via the UI.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Just wanted to ask you, which plugin you downloaded to get the --conn_type 'beeline' appear in the drop down list, I am facing the error while using the almost similar code with Beeline connection