jimathyp

Sonarlint

Turning off rules

https://community.sonarsource.com/t/how-to-temporarily-turn-off-a-rule/4135/3

You can add a #NOSONAR comment in your code or, if you use SonarLint with SonarQube or SonarCloud,

jimathyp

Spark - using Pyspark functions

usually prefer using Spark SQL

but sometimes makes sense to use pyspark functions

select where like

jimathyp

"Black" - A Python Formatter

Black will format your code. It can do it automatically on save, or when required.

Black is opinionated - this means it has basically one and only one way that the Python code will be formatted. There is almost no configuraiton.

One major opinion it enforces is the use of double quotes in place of single quotes.

This can be a major change to an existing project.

jimathyp

Installing pip packages globally

https://github.com/pypa/pipx

Certain tools you may wish to use throughout your virtual environments.

One way to achieve this is to install the package 'globally', ie. not in a virtual env.

However more tools - more risk of dependency issues.

jimathyp

Python decorators

In Python a function is an object. It can be assigned to a variable, and calling a function is a separate action.

def some_func():
  print("hi")
  
a = some_func

jimathyp

Spark SQL

• Constants
• Series generation
• Reading JSON
• Schemas
• Day of week

jimathyp

VS Code Shortcuts

Show shortcuts - Ctrl K, Ctrl S

Duplicate workspace - Ctrl Shift P

Open file in new window - Ctrl K, release, O

jimathyp

Spark and Delta

• Delta Basics
• Errors
  • Cannot create spark delta table from JSON input
• Partitioning with delta

Delta Basics

jimathyp

Output postgres query to CSV

https://www.postgresql.org/docs/13/app-psql.html

\timing on
\pset footer on


-- change field separator if query may contain commas

jimathyp

General Excel tips

Enter a new line within a cell = Alt + Enter