Skip to content

Instantly share code, notes, and snippets.

View MrCreosote's full-sized avatar

MrCreosote

9 followers · 0 following
  • Lawrence Berkeley National Laboratory
View GitHub Profile
@MrCreosote
MrCreosote / gist:a1fd88aaeb70fd0b6ea16ad18e89b91c
Created July 17, 2024 22:30
spark_yarn_client_mode_logs.txt
24/07/17 21:34:01 INFO SparkContext: Running Spark version 3.5.1
24/07/17 21:34:01 INFO SparkContext: OS info Linux, 4.15.0-213-generic, amd64
24/07/17 21:34:01 INFO SparkContext: Java version 17.0.11
24/07/17 21:34:01 INFO ResourceUtils: ==============================================================
24/07/17 21:34:01 INFO ResourceUtils: No custom resources configured for spark.driver.
24/07/17 21:34:01 INFO ResourceUtils: ==============================================================
24/07/17 21:34:01 INFO SparkContext: Submitted application: Spark Pi
24/07/17 21:34:01 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
24/07/17 21:34:01 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
24/07/17 21:34:01 INFO ResourceProfileManager: Added Res
@MrCreosote
MrCreosote / nersc_api_test.py
Created July 29, 2024 23:22
Testing NERSC superfacility API
import requests
import sys
from authlib.integrations.requests_client import OAuth2Session
from authlib.oauth2.rfc7523 import PrivateKeyJWT
TOKEN_URL = "https://oidc.nersc.gov/c2id/token"