Python is one of the tools in my toolbox, so I never used it exclusively. This means i need to get an update, occasionally.
Rant mode on
They want to have an expressive language, so they keep adding features upon features to it, instead of caring too much about performance (which means that no JIT - just-in-time compiler - has been added to cpython).
I would have thought that typescript is gaining over python on github, because node.js is JIT based, unlike cpython, but that guess was wrong: github article. The github article suggests a different reason:
""" Typed languages reduce hallucination surface area. They also give models more structure to reason about during generation. That’s not a theoretical benefit. It’s now a behavioral signal in the data:
AI models tend to perform better on languages which expose information about correctness, like a type system Developers using AI tools are more likely to adopt typed languages for new projects The more teams rely on AI assistance, the more language choice becomes an AI-compatibility decision, not merely a personal preference """
So they say that 'vibe coders' prefer typescript, as it is making it easier to do the 'vibe coding'...
(however python has something to compensate for that: type annotations. https://typing.python.org/en/latest/spec/annotations.html however typing is optional, not enforced during runtime by cpython. Still you have static type checkers, so not entirely clear why typescript is all the rage now.)
(unrelated: amazing - how they push copilot on every occasion, here at github...)
bash command to show, what stuff is being imported (excluding ./.venv - it it is the virtual directory)
find . -path './.venv' -prune -o -name '*.py' | xargs grep -h -E '^import|^from' | tr -d '\r' | sort | uniq
find . -path './.venv' -prune -o -name '*.py' | xargs grep -h -E '^import|^from' | awk '{ print $2 }' | tr -d '\r' | sort | uniq
or look at local requirements.txt or pyproject.toml , that's easier ;-) .. but this one also gives you imports between internal packages ..
Why do they use enumerate(list_obj) for iterating over index,value tuples of a list value vs d_obj.items() for iterating over keys,value tuples of a dict value?
a=['a','b','c']
# show the whole list of (index, value) tuples
print(list(enumerate(a)))
# enumerate(a) returns an iterator, it returns a tuple of (index, value) for each iteration
for idx,val in enumerate(a):
print(idx,val)
# this iterates over the values of an array
for val in a:
print(val)
a={'foo':1, 'bar':2}
# show the whole list of (key, value) tuples
print(list(a.items()))
# a.items() returns an iterator, it returns a tuple of (key, value) for each iteration
for k,v in a.items():
print(k,v)
# a.keys() returns n iterator over the keys of a dict
for k in a.keys():
print(k)
# this also iterates over the keys of a dict!
# (when iterating this way over array you get the values of the array)
for k in a:
print(k)
Another thing
dict_value.items() and enumerate(list_value) are returning iterator objects. You can't use an iterator in the same way as a list object. If a list is needed, then you need to create the list explicitly from an iterator. This step performs the whole iterations implicitly - in O(n).
Iterators were added in python3 - in python2 dict_value.items() was returning a full lists, so iteration was more awkward with python2 (it takes another iteration over the list to create a full list. So iterator in python3 were a great improvement, no extra iteration is needed in python3, just for creating the iterator!
since python 3.7 - when iterating over keys of a dictionary you get the order of insertion of the keys.
>>> a={}
>>> a['z']=1
>>> a['a']=1
>>> a['k']=1
>>>
>>> for k,v in a.items():
... print(k, v)
...
z 1
a 1
k 1
import re
a="tok1 tok2 tok3"
print("split with space separator between tokens")
for tok in a.split():
print(tok)
a="tok1#tok2#tok3"
print("split with # separator between tokens")
for tok in a.split('#'):
print(tok)
a="tok1, tok2; tok3;, "
print("re.findall - split with regex that specifies token form (that is still sane)")
for tok in re.findall(r'\w+', a):
print(f"tok: >{tok}<")
print("** DON'T USE re.split ITS EXTREMELY CONFUSING **")
print("re.split - split with regex that specifies separator token. Confusing part: empty token returned at end!")
for tok in re.split(r'\W+', a):
print(f"tok: >{tok}<")
print("re.split - split with regex that specifies separator token. Confusing part: empty token returned at end!")
for tok in re.split(r'[\s,;]+', a):
print(f"tok: >{tok}<")
print("re.split - also: capturing groups () are returned extra. This returns tokens, separators and empty token at end!")
for tok in re.split(r'([\s,;]+)', a):
print(f"tok: >{tok}<")
never really understood the feature. Now this example explained it for me:
how to comment a multiline string as sql comment
sql_str = """SELECT name
FROM employee_table"
# lines is a list
lines = [ f"-- {line}" for line in sql_str.split('\n') ]
print("\n".join(lines))
# venv create
python3 -m venv .venv
# venv activat
source .venv/bin/activate
# create requirements / freeze current requirments
pip3 freeze > requirements.txt
# setup from requirements
pip3 install -r requirements.txt
rant mode on:
requirements.txt are nice and cosy while they work - on a given platform at a given moment.
Often better to do pip install xyz on a new virtual environment. This gets you the latest version of the package, it also has a better chance of finding a prebuilt package cached by the pip host. If there is no pre-build binary available, then things get rough - you need to build all the native components, this can be a rough ride.
However this doesn't work, if a breaking change occurred in a dependent library. In this case you will either need to change your own code, or you will break a dependency inside a third-party libraries (which is worse)
This is indirectly connected to the fact that cpython doesn't have a just in time compiler (JIT). They keep saying, that performance critical stuff can be pushed into a native library (that's what they do with numpy, for example) However it is not easy to build all this stuff from scratch. Therefore these libraries rely on pre-build binary packages, because otherwise you would risk serious damage with reproducing the full build process. Doing pip install numpy for the latest version is a rescue, as the binary package is most likely available for the latest version.
Actually you end up with reproducing the full build cycle of binary packages in a different setting: when using python package on a docker image. Now that's a really hard exercise, trust me...
(Now I am beginning to sound like 'Bernd das Brot' ...)
venv is 'alte sachen' - old news, all the cool kids use poetry or uv. These are written in rust, so they say it is very fast.
https://python-poetry.org/docs/basic-usage/
With poetry, must use pyenv - to use the right python environment when setting up the virtual environment.
Installing poetry
curl -sSL https://install.python-poetry.org | python3 -
# puts it in /home/user/.local/bin/poetry
# create pyproject.toml in current directory, interactively
poetry init
# create .venv environment in current dir. This creates file poetry.lock from pyproject.toml
poetry install
# for repeated usage, after editing pyproject.toml (so no need to do pip freeze - it's already here)
poetry install --no-root
# activate env
source .venv/bin/activate
# add package
poetry add 'thepackagename'
tooled towards ai/machine learning. More focus on storing binary packages, so you get less into trouble with building a package from scratch. Problem: companies with 200+ employees require a paid license, so some place don't like it at all...
advised, it's a good thing.
python suffers a bit from 'it works on my machine' - they kee adding a lot of stuff in the standard library, the language keeps changing. So it's handy to be able to switch the environment version, to have the same thing as your customer.
(if you need an absolutely stable env, then people actually use perl - no joke. Also I used to like perl ...)
# install pyenv. puts it into ~/.pyenv need to add ~/.pyenv/bin to path
curl https://pyenv.run | bash
# list currently installed python versions, system is the version in system path
pyenv versions
# first time - install a new python version
pyenv install 3.13.3
# switch between versions for the current user
pyenv local 3.10.18
pyenv local 3.13.3
# pythons available for installation by means of penv install
# all pythons that pyenv knows about (that's quite a lot of them, didn't hear about half of them)
pyenv install -l | sed -e 's/ \([a-z0-9]*\).*/\1/p' | sort | uniq
2 - the old cpython2
3 - cpython3
nogil - alternative build of cpython (on the same code base, the build system has --disable-gil flag). Promise freedom from global interpreter lock.
micropython - re-implementation of python interpreter, for memory constrained systems.
activepython - ActiveState's variant of cpython (they offer support & some kind of security certification)
stackless - based on cpython. has cooperative multithreading, like go-routines, and other goodies. Maintains the interpreter stack on the heap, so fewer stack overflows with stackless.
--
ironpython - .net based python
graalpy - on top ov GraalVM (supposed to be more memory & cpu efficient variant of the JVM)
graalpython
jython - java based (JDK)
--
anaconda - like miniconda, but bigger (more preinstalled packages)
anaconda2
anaconda3
miniconda - the have their python that is more integrated with conda package manager. supposed to be better with data science
miniconda2
miniconda3
mambaforge - forge of conda stuff
miniforge - miniconda forge of the same kind
miniforge3
--
cinder - meta/facebook has it's cpython fork. Supposedly with it's JIT. (facebook likes adding a JIT, like with php...) They also have normal garbage collection (unlike reference counted GC with cpython). But it comes with lots of incompatibilities (like pypy)
pypy - the oldest jit based python. also has it's own GC (not reference counted) Also has lots of incompatibilities with cpython.
pypy2
pypy3
pyston - another one of those JIT based variants, with a focus on performance.
Common problem with projects with a just-in-time compiler :
- incompatible C API leads to many imported packages being not supported.
- differences in memory semantic: cpython has reference counting, and most JIT languages have generational GC (or mark&sweep or whatever)
- C language changes fast, it is a hard job to keep up.
- impossible amount of stuff that needs to be maintained.
That's quite an ecosystem, if you ask me...
Turns out cpython has a JIT since version 3.13
Need to disable it like this. The world is a changing place...
PYTHON_JIT=1 python my_script.py or like this python -X my_script.py
import pprint
data = { ... }
def save_data(output_file, data):
with open(output_file, 'w', encoding='utf-8') as f:
formatted_string = pprint.pformat(data, indent=4)
json.write(formatted_string)
# pretty print to string
formatted_string = pprint.pformat(data, indent=4)
print(formatted_string)
# print it now
pprint.pprint(data, indent=4))
You see this a lot: if the given python file is run as then run do_it() . This allows you to use the python file both as a standalone script and import it as a library!
Often people put in unit tests here, when running the file as is. Sometimes it makes sense to have a utility tht does something with the file/module...
if __name__ == "__main__":
do_it()
import json
def save_json(output_file, json_data):
with open(output_file, 'w', encoding='utf-8') as f:
json.dump(json_data, f, ensure_ascii=False, indent=4)
Don't use json.dump() for pretty printing: For example you can't dump a set() object to json - it's not 'json serializable'. This means there is no construct in json that would express a set(). Json has lists, dictionaries, scalars - but no sets.
sqlfluff - for pretty-printing sql statements. (depends on the presence of cfg file .sqlfluff configuration file for the sql dialect/aspects of the style)
import sqlfluff
sql_content = ```CREATE TABLE foo(name VARCHAR(10), etc VARCHAR(10))```
fixed_sql = sqlfluff.fix(sql_content, config_path='../.sqlfluff')
print(fixed_sql)
- you need to export to doc
- in confluence web UI: .. / Export / Export to doc
- must convert from .doc format to .docx format
- only way to do this effectively is in MS word. Open the .doc file and save as .docx
- convert .docx to markdown, so that the tables are rendered as markdown tables (and not as convoluted html_
- there is a python package for that
pip install markitdown[all] - the python package comes with markitdown command line tool
- there is a python package for that
- in python: parse the markdown effectively. There is a python package for that: misletoe. It produces a sometimes tricky AST, but it is manageable.
built-in dataclasses for plain old data
from dataclasses import dataclass
from typing import Optional
@dataclass
class Entry:
view_name: str
table_name: str
file_name: str
tab_name: str
view_sql_generated: Optional[bool]
on_conflict: str
# now that gives you a nice class for Entry: complete with ctor - __init__, __repr__, __hash__, __eq__, __ne__ - a good citizen of python!
Also can construct if from a dict read from json, and the __init__ 'gets' it as is!
Now you will get troubles when serializing to json, but
The cool kids will use pydantic library - it adds data validation on __init__ and has less problems with json serialization.
They keep changing the way how to do decorators, apparently.a That's how you do it, these days.
from functools import wraps
import traceback
import time
import pprint
def catch_exception(func):
@wraps(func)
def wrapper(*args, **kwargs):
start = time.monotonic_ns()
try:
data = func(*args, **kwargs)
duration = time.monotonic_ns() - start
return { 'status': True, 'data': data, 'error': None, 'stack_trace': None, 'duration_ns': duration }
except Exception as e:
duration = time.monotonic_ns() - start
stack_trace = traceback.format_exc()
return { 'status': False, 'data': None, 'error': e, 'stack_trace': stack_trace, 'duration_ns': duration }
return wrapper
@catch_exception
def do_it(dividend, divisor):
print(f"{dividend} / {divisor} = {dividend/divisor}")
res = do_it(4,2)
print( pprint.pformat(res) )
res = do_it(5,0)
print( pprint.pformat(res) )
# it returns a DataFrame object
import pandas as pd
def empty_excel(val):
return isinstance(val, (int, float, complex)) and math.isnan(val)
def convert_tbl(excel_file, sheet_name):
df = pd.read_excel(excel_file, sheet_name=sheet_name, header=None)
for _, row in df.iterrows():
for idx in range(row.size):
val = row.iloc[idx]
# now beware! an empty cell can be a number, which is infinite
if not empty_excel(val):
print(str(val))
Pandas has a very big problem: sometimes you read the stuff and it is NOT EXACTLY the same as in Excel !!!
xlwings works directly with an installed copy of excel, and it doesn't have that problem !!! Now xlwings talks with excel: on Mac via AppleScript, on Windows via DCOM. It is a bit slow, has it's quirks, but it works! (and doesn't get lost when data is formatted.)
Prefer to store them in csv files - pandas.read_csv
care to pass chunksize parameter to pandas - chunking is much faster with csvs...
They keep adding and adding features to optional type hints (package typing) !!! ( https://pyrefly.org/en/docs/python-features-and-peps/ ) (i knew the basic ones, but now theres ten times more!) ... and ten years later they say people are using the resulting mess, because he IDE's support it ( https://pyrefly.org/blog/why-typed-python/ ) (but pylint checker doesn't, I guess that's bad for pylint)
interesting fact: php enforces type annotations during runtime, but python doesn't do that. why?
- type checking would add runtime overhead, and python prefers to add new features, while php prefers to add a just in time compiler.
- (turns out that facebook pumped a lot of money into PHP development)
Actually that's why the author Guido van Rossum quit! https://www.packtpub.com/en-us/learning/how-to-tutorials/why-guido-van-rossum-quit
# 'walrus' operator, assign value and return it, in one step (regular assignment doesn't return a value)
# they had to add it, as = is both
>>> (a:=123)
123
>>> a
123
# but that one is a syntax error:
>>> a:=123
File "<stdin>", line 1
a:=123
^^
SyntaxError: invalid syntax
# but you can use it in an assignmen - without surrounding it with the ( ) !!!
>>> if a:=random.randint(0,1):
... print("is one")
...
is one
>>> a
1
# but you can even compose it further with an additional check - and now you need the ( ) definitely
>>> if (a:=random.randint(0,2)) > 1:
... print('bigger than one!')
.
# and all that in the name of readability:
# Python: we avoid the classical C problem by adding == vs = (in C you can have
# if (a=b) { // that's a very bad one, if you mistyped == for = ; because the syntax allows is
# but then he says: we want to be concise, so let's add yet another one (and everyone's head explodes...)
Better not to have arguments over picky code reviews, that's what experience shows...
- Use ValueError exception over base Exception. Only add your own exception class for significant categories of errors that can be categorized into subclasses, like permission exceptions.
you need to format an sql statement - and a very enticing option is to use f-strings...
but is is a bad option - you get sql injections with that one - very bad if you need to read input from somewhere...
obligatory reference: https://xkcd.com/327/
I can use either
- psycopyg2/3 native stuff:
- sqlparams
psycopyg2/3 : cursor.execute() gets template + tuple for arguments. Now tuple of one element needs to be (entry,) - without the comma, python just thinks it's a nested expression.
Accessing postgress: psycopg2 - best option. this doesn't use jdbc drivers, or stuff like that.
install with:
pip install psycopg2-binary
from dynaconf import Dynaconf
import psycopg2
from psycopg2.extras import RealDictCursor
# have to tell the session if it is read_only or not!
def db_connect(read_only = True):
dynaconf_settings = Dynaconf(
lowercase_read=True,
settings_files=[
"settings/settings.toml"
],
)
conn_str = dynaconf_settings["POSTGRESQL_CONNECTION_STRING"]
conn = psycopg2.connect(conn_str, cursor_factory=RealDictCursor)
conn.set_session(readonly=read_only, autocommit=True)
return conn
def check_table_exists(conn, view_name_with_schema):
schema_name, view_name = view_name_with_schema.split(".")
try:
with conn.cursor() as cursor:
stmt = f"""
SELECT EXISTS (
SELECT 1
FROM pg_tables
WHERE schemaname = '{schema_name}'
AND tablename = '{view_name}'
);
"""
cursor.execute(stmt)
field_names = cursor.fetchall()
value = field_names[0]['exists']
return value
except Exception as e:
print(f"failed to check if view exists {view_name}")
print(f"error {e}")
return None
def check_view_exists(conn, view_name_with_schema):
schema_name, view_name = view_name_with_schema.split(".")
try:
with conn.cursor() as cursor:
stmt = f"""
SELECT
a.attname AS column_name,
pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type
FROM
pg_catalog.pg_attribute a
INNER JOIN
pg_catalog.pg_class c ON c.oid = a.attrelid
INNER JOIN
pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE
n.nspname = '{schema_name}' -- Replace with your schema name (e.g., 'public')
AND c.relname = '{table_name}' -- Replace with your table name
AND a.attnum > 0
AND NOT a.attisdropped
ORDER BY
a.attnum;
"""
cursor.execute(stmt)
field_names = cursor.fetchall()
value = field_names[0]['exists']
return value
except Exception as e:
err(f"failed to check if view exists {view_name}")
return None
def get_field_names_and_types(conn, schema_and_tbl):
try:
# get list of fields in source view
with conn.cursor() as cursor:
stmt = f"""
SELECT
a.attname AS column_name,
pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type,
a.attnotnull AS not_null
FROM pg_catalog.pg_attribute a
WHERE a.attrelid = '{schema_and_tbl}'::regclass
AND a.attnum > 0
AND NOT a.attisdropped
ORDER BY a.attnum;
"""
cursor.execute(stmt)
rows = cursor.fetchall()
ret = ()
for r in rows:
orig = data_type_full = r['data_type']
ret.append( ( r['column_name'] ], data_type_full ) )
#print(f"type info for {schema_and_tbl} : {ret}")
return ret, True
except Exception as e:
err(f"failed to get field types for {schema_and_tbl}")
return [], False
def get_db_table_rows(conn, table_name_with_schema):
try:
with conn.cursor() as cursor:
stmt = f"""
SELECT *
FROM {table_name_with_schema};
"""
cursor.execute(stmt)
rows = cursor.fetchall()
ret_rows = []
for row in rows:
ret_row = {}
for column in row.items():
val = column[1]
if val is None or val == "":
val = ""
else:
val = str(val)
ret_row[column[0]] = val
ret_rows.append(ret_row)
return ret_rows, True
except Exception as e:
print(f"failed to check if view exists {table_name_with_schema}")
print(f"error {e}")
return None, False
def remove_sql_comment(line):
while True:
pos = line.rfind('--')
if pos == -1:
return line
line = line[:pos]
def check_select_stmt(conn, sql_stm):
stm = sql_stm.strip()
stm += "\nLIMIT 0;"
print(f"checking:\n{stm}")
status ,msg, _ = run_select(conn, stm)
return status, msg
def run_select(conn, sql_stmt):
try:
with conn.cursor() as cursor:
cursor.execute(sql_stmt)
rows = cursor.fetchall()
return True, "", rows
except Exception as e:
return False, f"sql statement failed, error {e}", None
You can use std library configparser package for INI files. But INI files are considered uncool.
You can use a the json package to parse a json config file.
You can use environment variables like os.getenv - but you should not put in passwords into environment variable - everyone can see them.
So you have other libraries or integration of all these options, like dynaconf...
from dynaconf import Dynaconf
# sometimes they use Dynaconf to merge several cfg files in various formats (including markdown)
# and merge then into one hash
dynaconf_settings = Dynaconf(
lowercase_read=True,
settings_files=[
"settings/settings.toml"
],
)