Lidor Ettinger naturalett
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| workers: | |
| enabled: true | |
| replicas: 3 | |
| nodeSelector: | |
| kops.k8s.io/instancegroup: airflow-workers |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| persistence: | |
| enabled: true | |
| storageClass: efs-airflow | |
| accessMode: ReadWriteMany | |
| size: 1Gi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Parameter | Explanation | |
|---|---|---|
| core.parallelism | Maximum number of tasks running across an entire Airflow installation | |
| core.dag_concurrency | The number of task instances allowed to run concurrently by the scheduler | |
| core.non_pooled_task_slot_count | Number of task slots allocated to tasks not running in a pool | |
| core.max_active_runs_per_dag | Maximum number of active DAG runs - per DAG | |
| scheduler.max_threads | How many threads the scheduler process should use to use to schedule DAGs | |
| celery.worker_concurrency | Number of task instances that a worker will take if using CeleryExecutor |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| pipeline { | |
| agent { | |
| kubernetes { | |
| label podLabel | |
| yaml pod | |
| } | |
| } | |
| stages { | |
| stage('Checkout') { | |
| steps { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # !! AUTO-GENERATED !! | |
| # application_id: {{{ ni_application_id }}} | |
| # version: {{{ version }}} | |
| import airflow | |
| from airflow import DAG | |
| from airflow.contrib.operators.emr_add_steps_operator \ | |
| import EmrAddStepsOperator | |
| from operators.emr_cluster_name_to_id import EmrClusterNameToIdOperator | |
| from airflow.contrib.sensors.emr_step_sensor import EmrStepSensor |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| --- | |
| ni_application_id: ${project.name} | |
| version: ${project.version} | |
| main_class: com.naturalint.SparkExample | |
| schedule: None | |
| timeout_minutes: 45 | |
| jar_location: s3://<bucket-name>//${gitRepositoryName}/${s3ArtifactsDir}/${project.version}${gitCommitIdStr}/${project.build.finalName}.jar | |
| emr_cluster_name: ${gitRepositoryName}-ci | |
| properties: | |
| - property: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| APP_ID = <APP_ID> | |
| def getJsonWebToken(privateKey) { | |
| try { | |
| privateCrtKey = getRSAPrivateKey(privateKey) | |
| time = accessTime() | |
| def jsonWebToken = Jwts.builder() | |
| .setSubject("RS256") | |
| .signWith(RS256, privateCrtKey) | |
| .setExpiration(time['expirationTime']) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def validateAuth(jsonWebToken) { | |
| try { | |
| def httpConn = new URL("https://api.github.com/app").openConnection(); | |
| httpConn.setRequestProperty( 'Authorization', "Bearer ${jsonWebToken}" ) | |
| httpConn.setRequestProperty( 'Accept', 'application/vnd.github.machine-man-preview+json' ) | |
| return httpConn.getResponseCode(); | |
| } catch(Exception e){ | |
| echo "Exception: ${e}" | |
| error "Authentication request failed" | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| INSTALLATION_ID = <INSTALLATION_ID> | |
| def getToken(jsonWebToken) { | |
| try { | |
| def httpConn = new URL("https://api.github.com/app/installations/${INSTALLATION_ID}/access_tokens").openConnection(); | |
| httpConn.setRequestProperty( 'Authorization', "Bearer ${jsonWebToken}" ) | |
| httpConn.setRequestProperty( 'Accept', 'application/vnd.github.machine-man-preview+json' ) | |
| httpConn.setRequestMethod("POST"); | |
| def responseText = httpConn.getInputStream().getText() | |
| def slurper = new JsonSlurper() |
naturalett
/ getPreviousCheckNameRunID.groovy
Created
February 23, 2020 22:30
getPreviousCheckNameRunID
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ORGANIZATION_NAME = <ORGANIZATION_NAME> | |
| def getPreviousCheckNameRunID(repository, commitID, token, checkName) { | |
| try { | |
| def httpConn = new URL("https://api.github.com/repos/${ORGANIZATION_NAME}/${repository}/commits/${commitID}/check-runs").openConnection(); | |
| httpConn.setDoOutput(true) | |
| httpConn.setRequestProperty( 'Authorization', "token ${token}" ) | |
| httpConn.setRequestProperty( 'Accept', 'application/vnd.github.antiope-preview+json' ) | |
| checkRuns = httpConn.getInputStream().getText(); | |
| def slurperCheckRun = new JsonSlurper() |
OlderNewer