Skip to content

Instantly share code, notes, and snippets.

View jjasont's full-sized avatar
🔍
on the pursuit of clean data

Jason Tan jjasont

🔍
on the pursuit of clean data
10 followers · 10 following
  • Singapore
View GitHub Profile
@jjasont
jjasont / README.md
Last active January 17, 2025 08:47
ISO Week-Year Calculation for AWS Quicksight (as of 30 July 2021)

ISO Week-Year Calculation for AWS Quicksight (as of 30 July 2021)

This gists is an attempt to 'hack' or a workaround for making the starting week as Monday (ISO 8601), retrieving an ISO week number and the corresponding year of ISO week number As there's no such function per 30 July 2021 to perform any extraction of week number in Quicksight, the following are supposedly resolve such cases.

Start week is Monday (Ref #2)

Define a column with the following formula, for this usage it will be named week_start_monday

There's an edge case handling for when the date of interest is Sunday. When the day is Sunday, it should belong to the previous week instead of the next one (since the week runs from Monday-Sunday, instead of Sunday-Saturday)




  


  





    


          
    

      

        
        

          

              @jjasont
          

          

            
                jjasont
                / parent-and-child-runs.py
            
            

              Created
              August 15, 2023 17:41
                — forked from smurching/parent-and-child-runs.py
            

              
                creating-child-runs-in-mlflow
              
          

        

      

        


      

    
    


        



  


  

        

          
          import mlflow
        

        

          
          

        

        

          
          # There are two ways to create parent/child runs in MLflow.
        

        

          
          

        

        

          
          # (1) The most common way is to use the fluent
        

        

          
          # mlflow.start_run API, passing nested=True:
        

        

          
          with mlflow.start_run():
        

        

          
            num_trials = 10
        

        

          
            mlflow.log_param("num_trials", num_trials)
        

        

          
            best_loss = 1e100
        

  





    


  





    


          
    

      

        
        

          

              @jjasont
          

          

            
                jjasont
                / spark_write_config.md
            
            

              Last active
              December 22, 2023 16:22
            

              
                Spark Write Out Config
              
          

        

      

        


      

      

    
The latest and faster Spark write out can be done with S3A/magic committer


Ensure the usage of s3a prefix when reading and writing from/to S3 for better performance



spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
spark.hadoop.fs.s3a.committer.magic.enabled: "true"
spark.hadoop.fs.s3a.committer.name: "magic"
spark.sql.sources.commitProtocolClass: org.apache.spark.internal.io.cloud.PathOutputCommitProtocol