FEED Validator

for Atom and RSS and KML

Congratulations!

This is a valid Atom 1.0 feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

Your feed appears to be encoded as "utf-8", but your server is reporting "US-ASCII" [help]

line 167, column 40: Two entries with the same value for atom:updated: 2025-03-27T21:06:00+01:00 [help]

      <updated>2025-03-27T21:06:00+01:00</updated>
                                        ^

Source: http://reinout.vanrees.org/weblog/atom.xml

<?xml version="1.0" encoding="utf-8" ?>
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://reinout.vanrees.org/" xml:lang="en">
<link rel="self"
href="http://reinout.vanrees.org/weblog/atom.xml" />
<link href="http://reinout.vanrees.org/weblog/"
rel="alternate" type="text/html" />
<div xmlns="http://www.w3.org/1999/xhtml">
<a href="http://www.atomenabled.org/feedvalidator/check.cgi?url=http%3A%2F%2Freinout.vanrees.org%2Fweblog%2Fatom.xml">
<img title="Validate my Atom feed" width="88"
height="31"
src="http://www.atomenabled.org/feedvalidator/images/valid-atom.png"
alt="[Valid Atom]" border="0px" />
</a>


This is an Atom formatted XML site feed. It is intended to be viewed in
a Newsreader or syndicated to another site. Please visit

<a href="http://www.atomenabled.org/">Atom Enabled</a>

for more info.


</div>
<title type="html">Reinout van Rees' weblog</title>
<subtitle>Python, grok, books, history, faith, etc.</subtitle>
<updated>2009-04-04T21:44:00+01:00</updated>
<id>urn:syndication:a55644db8591c020bd38852775819a9a</id>
<entry>
<title>Python Leiden (NL) meetup: Rendering spatial data in 3d using zarr, jax, babylon.js and DuckDB - Jesse K.V.</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2025/03/27/3-rendering-3d.html" />
<id>http://reinout.vanrees.org/weblog/2025/03/27/3-rendering-3d.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2025-03-27T00:00:00+01:00</published>
<updated>2025-03-27T21:06:00+01:00</updated>
<category term="pun" />
<category term="python" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/2025/03/27/">my summaries</a> of
the second <a class="reference external" href="https://pythonleiden.nl/">Python Leiden (NL) meetup</a> in
Leiden, NL).
He's working with civil engineering, hydrology and weather data. And... he wanted to toy
with 3D models. His example site: <a class="reference external" href="https://topography.jessekv.com/">https://topography.jessekv.com/</a> . You can click almost
anywhere in the world and get the upstream catchment. (I checked it: yes, it seems to
work pretty well!)
He runs all requests through a single asyncio python thread. As a personal challenge he
wanted it to handle the heavy load of a hacker news post. In the end, it worked
fine. One async python thread was more than enough.
One of the tricks he used was to preprocess as much as reasonable so that most clicks
are direct lookups in a database (vector data). Depending on the size of the selected
area, he uses more detailed rasters for small areas and coarser ones for big areas.
He wanted a mostly-working prototype quickly, so he experimented with LLMs. Generating
math code was mostly bad, but the UI code was OK.
He used <a class="reference external" href="https://duckdb.org/">duckdb</a> with a spatial extension. Duckdb uses GDAL
vector routines. This is what he used to pre-process the catchment areas on his M1 mac
laptop. Afterwards, he exported it to postgres. Postgres is much more optimised for
actual production use.
Duckdb doesn't always work perfectly, but if you're able to define your workload in such
a way (parallelised) that you stay within the limits of your memory, you can get real
good performance out of it.
Duckdb's file-based approach is also handy. Just like sqlite's files. Easy for
experimenting.
<a class="reference external" href="https://zarr.dev/">zarr</a> is what he used for pre-processing the 3D landscape. Zarr is
efficient for storing large arrays of gridded data. Zarr is way nicer than
<tt class="docutils literal">netcdf</tt>. It is designed to leverage the linux page cache. And you store compressed
data in memory. Storing on S3 is also well-integrated.
<a class="reference external" href="https://docs.jax.dev/en/latest/">jax</a> is an easy way to take numpy/scipy to make it
run in parallel. <tt class="docutils literal">JAX-metal</tt> is a jax backend that runs on his M1 macbook's
GPU. Processing is aligned to chuncks for more efficient reads and writes.
For landcover, he used <tt class="docutils literal">jax.scipy.stats.mode</tt> and for elevation
<tt class="docutils literal">jax.numpy.nanmean</tt>. (<tt class="docutils literal">NaN</tt>: not-a-number: elevation models are made with radar
technology and water areas reflect radar, resulting in NaN).
A useful trick he used was to introduce a bit of wait time for some expensive
operations to make sure his service wouldn't get flooded with requests. Simply waiting a
few seconds and then popping up a dialog "you're going to download 400MB, y/n?" already
helped.
</div>
]]>
</content>
</entry>
<entry>
<title>Python Leiden (NL) meetup: how to use uv for dependency management - Michiel Beijen</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2025/03/27/2-uv.html" />
<id>http://reinout.vanrees.org/weblog/2025/03/27/2-uv.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2025-03-27T00:00:00+01:00</published>
<updated>2025-03-27T21:28:00+01:00</updated>
<category term="pun" />
<category term="python" />
<category term="django" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/2025/03/27/">my summaries</a> of
the second <a class="reference external" href="https://pythonleiden.nl/">Python Leiden (NL) meetup</a> in
Leiden, NL).
<a class="reference external" href="https://docs.astral.sh/uv/">uv</a> is the new python packaging solution that everybody
should be using. He demoed it in a standard demo django wagtail project that still
had a <tt class="docutils literal">requirements.txt</tt>. Creating a virtualenv and doing a <tt class="docutils literal">pip install</tt> worked,
but it took a bit of time.
Next, he tried the venv/pip compatibility layer of <tt class="docutils literal">uv</tt>. So <tt class="docutils literal">uv venv</tt> and <tt class="docutils literal">uv pip
install -r requirements.txt</tt>. Oh... <tt class="docutils literal">python -m venv .venv</tt> took 3 seconds, and <tt class="docutils literal">uv
venv</tt> 0.04 seconds. Wow. The <tt class="docutils literal">uv pip</tt> was also much faster. Wow.
"Faster" is nice in development, but also when running your test in a CI enviroment
(like github actions).
<tt class="docutils literal">uv</tt> can also manage your python installations. It downloads a stand-alone python for
you when needed, for instance if you need a version you don't have locally.
Lastly, he added a <tt class="docutils literal">pyproject.toml</tt> and put the dependencies from <tt class="docutils literal">requirements.txt</tt>
into the pyproject.toml instead. Calling <tt class="docutils literal">uv run manage.py</tt> automatically activates
the virtualenv, install everything and run manage.py just like you'd have called
<tt class="docutils literal">python manage.py</tt>.
Installing it in such a way creates an <tt class="docutils literal">uv.lock</tt> file with all the pinned packages
just such as uv downloaded them. The file should be checked in. The dependencies only
say "I want <tt class="docutils literal">whitenoise</tt> and don't care what version", but the <tt class="docutils literal">uv.lock</tt> tells all
your colleages (or your server..) which version you've installed, keeping everything in
sync.
Dependency scanners also support <tt class="docutils literal">uv.lock</tt>. At least, "renovate" already supports it
and "dependabot" support is underway.
Handy: <tt class="docutils literal">uv tool install your-tool</tt> for installing some tool like "pre-commit" in its
own virtualenv somewhere and make it available globally. <tt class="docutils literal">uvx your-tool</tt> installs the
tool temporarily for just one command. And with <tt class="docutils literal">uv add --script your-script.py
some-dependency</tt> you can add dependencies to single-file scripts. Running the script
with <tt class="docutils literal">uv run your-script.py</tt> installs those dependencies for the duration of the
script.
</div>
]]>
</content>
</entry>
<entry>
<title>Python Leiden (NL) meetup: serialisation in Python - John Labelle</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2025/03/27/1-serialisation.html" />
<id>http://reinout.vanrees.org/weblog/2025/03/27/1-serialisation.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2025-03-27T00:00:00+01:00</published>
<updated>2025-03-27T21:06:00+01:00</updated>
<category term="pun" />
<category term="python" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/2025/03/27/">my summaries</a> of
the second <a class="reference external" href="https://pythonleiden.nl/">Python Leiden (NL) meetup</a> in
Leiden, NL).
Nice subtitle for the talk: "python serialisation: this ain't your father's
cruesli"... :-)
He wants to show us how dangerous it is to de-serialize content provided by someone
else. His examples are at <a class="reference external" href="https://github.com/airza/deserialization_labs">https://github.com/airza/deserialization_labs</a>
Serialisation: converting a data structure living in memory into data you can store on
disk or send over. Deserialisation is converting it back into a python object. There are
interoperable formats like json and xml. Most languages have their own specific
methods: python has <tt class="docutils literal">pickle</tt>.
Serialising a dict or list is often easy. <tt class="docutils literal">json.dumps({"some": "structure"})</tt> But what
if you've got some non-standard data structure like a python object? json serialisation
won't work out of the box. And if you've got huge data structures, json (being
human-readable) is slow and huge.
Pickle stores python objects in some binary format on disk. Fun fact: pickle was added
to python in 1995, json only exists since 2006. I'll paste one of his examples to make
clear how picle works:
<pre class="literal-block">
import pickle
from tsukimi import Tsukimi
cat = Tsukimi("Fluffy", "Empty")
pickle.dump(cat, open("tsukimi.pickle", "wb"))
</pre>
Deserialising works like this:
<pre class="literal-block">
import pickle
cat = pickle.load(open('tsukimi.pickle', 'rb'))
print(cat.fur)
print(cat.brain)
</pre>
Pickle just stores the name of the class it needs to re-create plus the contents of the
attributes. So not the methods or so.
Pickle is explained here: <a class="reference external" href="https://docs.python.org/3/library/pickle.html">https://docs.python.org/3/library/pickle.html</a> . It has a nice
warning right at the top: Warning: The pickle module is not secure. Only unpickle data
you trust.
Pickle stores all attributes by default, if you don't want that you can define a
special <tt class="docutils literal">__reduce__()</tt> function that specifies just the attributes you want and the
name of the class that can restore them. But... the name of that class is just looked
up, there's no validation. So you can also pass something that's not the name of your
class, but something like <tt class="docutils literal">os.system</tt> which just calls anything on the command
line...:
<pre class="literal-block">
import os
import pickle
class EvilCat:
def __reduce__(self):
return os.system, ('export > version.txt',)
evil = EvilTsukimi()
pickle.dump(evil, open("evil.pickle", "wb"))
</pre>
If the code that loads this pickle reads the version.txt (as in the exercise that he had
us run), you suddenly see all the server's environ variables.
So: never let people give you pickles. Use json for user input. Or protobuf.
Pytorch (a pydata library) uses pickles. They recently started overwriting the
unpickler's functionality, but he showed some ways to get around its "limitations".
He recommended looking at
<a class="reference external" href="https://github.com/b4rdia/HackTricks/tree/master/generic-methodologies-and-resources/python/bypass-python-sandboxes">https://github.com/b4rdia/HackTricks/tree/master/generic-methodologies-and-resources/python/bypass-python-sandboxes</a>
</div>
]]>
</content>
</entry>
<entry>
<title>Python Leiden (NL) meetup: python in applied mathematics - Tobias Datema</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2025/01/27/2-maths-study.html" />
<id>http://reinout.vanrees.org/weblog/2025/01/27/2-maths-study.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2025-01-27T00:00:00+01:00</published>
<updated>2025-01-27T21:14:00+01:00</updated>
<category term="pun" />
<category term="python" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/tags/pun.html">my summaries</a> of
the first <a class="reference external" href="https://pythonleiden.nl/">Python Leiden (NL) meetup</a> in
Leiden, NL).
Tobias studied applied mathematics at Delft University.
One of the fields he used python for was graph theory. A graph consists of points
("vertices") connected by lines ("edges"). It is a large field with many real world
projects like social networks and logistics. He showed a demo he made with <a class="reference external" href="https://networkx.org/">networkx</a>, a python library that makes it real easy to do these kinds of
graph calculations.
Graphs need to be shown. He used <a class="reference external" href="https://pyviz.org/">pyviz</a> for that by converting
the networkx graph to the format understood by pyviz.
Another field is machine learning. He did an experiment with a simulated
self-driving car. He used a library that handles the basics like "reacting to a closed
line on the road" and "measuring the distance to the dashed line on the road". The
simulation is shown in a visual form, which makes it funny to look at.
In his study, python was also handy for statistics and numerical analysis.
</div>
]]>
</content>
</entry>
<entry>
<title>Python Leiden (NL) meetup: fawltydeps - Johan Herland</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2025/01/27/1-fawltydeps.html" />
<id>http://reinout.vanrees.org/weblog/2025/01/27/1-fawltydeps.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2025-01-27T00:00:00+01:00</published>
<updated>2025-01-27T21:15:00+01:00</updated>
<category term="pun" />
<category term="python" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/tags/pun.html">my summaries</a> of
the first <a class="reference external" href="https://pythonleiden.nl/">Python Leiden (NL) meetup</a> in
Leiden, NL).
<a class="reference external" href="https://github.com/tweag/FawltyDeps">FawltyDeps</a> is a python dependency
checker. "Finding undeclared and unused dependencies in your notebooks and projects".
Note by Reinout: since 2009 I'm one of the maintainers of <a class="reference external" href="https://pypi.org/project/z3c.dependencychecker/">z3c.dependencychecker</a>.... also a python dependency checker
:-) So this talk interested me a lot, as I didn't know yet about fawltydeps.
A big problem in science is the "replication crisis". Lots of research cannot actually
be reproduced when you try it... Data science is part of this problem. Reproducing your
jupyter notebook for instance.
Someone looked at 22k+ jupyter notebooks. Only 70% had declared their dependencies, 46%
could actually install the dependencies and only 5% actually could be
run. <tt class="docutils literal">ModuleNotFoundError</tt> and <tt class="docutils literal">ImportError</tt> were the number 1 and 3 in the list of
exceptions!
What is a dependency? For instance "numpy", if you have a <tt class="docutils literal">import numpy as np</tt> in your
file. Numpy isn't in the python standard library, you have to install it first.
You can specify dependencies in <tt class="docutils literal">setup.py</tt>, <tt class="docutils literal">pyproject.toml</tt>, <tt class="docutils literal">requirements.txt</tt>
and so. If you import something and don't specify it, it is an "undeclared
dependency". When you later on remove an import and don't adjust your
<tt class="docutils literal">requirements.txt</tt>, you have an "unused dependency". That's not immediately fatal, but
it might take up unnecessary space.
FawltyDeps was started to help with this problem: find undeclared and unused
dependencies. It reports them. You can ask for a more detailed report with line numbers
where the dependencies were found.
FawltyDeps supports most dependency declaration locations. requirements.txt, setup.py,
pyproject, conda, etc. And it works with plain python files, notebooks, most python
versions and most OSs. You can configure it on the commandline and in config
files. There's even a handy command to add an example config to your <tt class="docutils literal">pyproject.toml</tt>.
Handy: you can add it as a pre-commit hook (<a class="reference external" href="https://pre-commit.com">https://pre-commit.com</a>). And: there's a
<a class="reference external" href="https://github.com/tweag/FawltyDeps-action">ready-made github action</a> for it,
including good reporting.
Fawltydeps has to deal with several corner cases:
<ul class="simple">
<li>Package names that don't match what you import. <tt class="docutils literal">import sklearn</tt> and the dependency
<tt class="docutils literal">scikit-learn</tt>.</li>
<li>Or <tt class="docutils literal">setuptools</tt> that provides both <tt class="docutils literal">setuptools</tt> and <tt class="docutils literal">pkg_resources</tt>.</li>
<li>For this it looks at various locations for installed packages to help figure out those
mappings. It helps if you've installed FawltyDeps in your project's virtualenv.</li>
<li>You can add your own custom mappings in your configuration to help FawltyDeps.</li>
<li>You can exclude directories.</li>
<li>There's a default list of "tool" packages that FawltyDeps doesn't complain about if
you include them as dependency without importing them. Ruff, black, isort: those kinds
of tools.</li>
<li>Django projects can have dependencies that aren't actually imported. You can ignore
those in the config to prevent them to be imported.</li>
<li>At the moment, extra dependencies (like <tt class="docutils literal">[test]</tt> or <tt class="docutils literal">[dev]</tt> dependencies) are just
handled as part of the whole set of dependencies.</li>
</ul>
</div>
]]>
</content>
</entry>
<entry>
<title>New upcoming Dutch python meetup: Leiden (NL)</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2025/01/16/python-leiden.html" />
<id>http://reinout.vanrees.org/weblog/2025/01/16/python-leiden.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2025-01-16T00:00:00+01:00</published>
<updated>2025-01-16T11:05:00+01:00</updated>
<category term="python" />
<category term="django" />
<category term="pun" />
<content type="html"><![CDATA[
<div class="document">
Important things first: 27 January there's a <a class="reference external" href="https://www.meetup.com/python-leiden-user-group/events/305358713/">python meetup in Leiden (NL)</a> of the new
<a class="reference external" href="https://pythonleiden.nl/">python Leiden user group</a>.
Meetup groups come and go, often depending on one or two people or on a company that
organises it. And yes, meetup.com has basically cornered the market, at least in my
experience.
There used to be a "PUN", python usergroup Nederland, meeting that would be held in
various cities, depending on the company that hosted it in turn. (For those in NL: Den
Haag, Zoetermeer, Rotterdam, Utrecht, Veenendaal, Arnhem, Amsterdam, ...). Managed via a
mailinglist, as meetup.com didn't exist yet. Later lots of python and/or django
meetup.com-based-meetups were organised. To me, it felt a bit weird that all of them
seemed to be city-oriented. Amsterdam python meetup (3 different ones), Amsterdam
django meetup, Utrecht (2x), Eindhoven, Rotterdam. I went to many of them, mostly it is
just an hour of travel by public transport :-)
I like going to those meetups. You get a feel for what people are doing with
python. You get ideas. You learn about libaries (sometimes even from the standard
library) that you didn't know about. New python tricks. For me, it is a great method to
keep up-to-date on what's possible and on what people are enthousiastic about.
At the moment, the number of python meetups in the Netherlands seems a bit low. Perhaps
I'm missing something? (I see there's a pydata one in Amsterdam that I missed.) The last
two I attended were the nice <a class="reference external" href="https://www.meetup.com/nl-nl/pyutrecht/">PyUtrecht</a>
ones. So: I'll be attending the Leiden one :-)
(Note: I'm talking about meetups, we're blessed with two one-day python conferences in the
Netherlands. <a class="reference external" href="https://pygrunn.org/">pygrunn</a> in May and <a class="reference external" href="https://pycon-nl.org/">pycon NL</a> in October.)
</div>
]]>
</content>
</entry>
<entry>
<title>Pycon NL: The zen of polymorphism - Brett Slatkin</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2024/10/10/9_-polymorphism.html" />
<id>http://reinout.vanrees.org/weblog/2024/10/10/9_-polymorphism.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2024-10-10T00:00:00+01:00</published>
<updated>2024-10-10T14:50:00+01:00</updated>
<category term="pycon" />
<category term="python" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/2024/10/10/index.html">my summaries</a> of
the one-day <a class="reference external" href="https://pycon-nl.org/">Pycon NL</a> conference in
Utrecht, NL).
Brett is the writer of the <a class="reference external" href="https://effectivepython.com/">effective python</a> book.
As an example, he picked a calculator that can add and multiply integers. The numbers,
the operators: everything is an object in his example. You can do it with one big
function, object oriented or with dynamic dispatch.
One big function: just a big old recursive function with a lot of if/else. Just
something to get started. Once you add more operators ("power of", "square root"), the
if/else quickly gets out of hand and becomes much to long and unreadable.
And lets say you don't only want to do the calculation, but you only want to have a
nicely printed version of the calculation. Suddenly your only choice is to copy/paste
the entire big function and change the calculation to print statements. And then you
need to keep the two in sync...
Second try: object oriented programming. Polymorphism. A generic <tt class="docutils literal">Operator</tt> class
with a <tt class="docutils literal">calculate()</tt> method. Then subclasses called <tt class="docutils literal">Add</tt>, <tt class="docutils literal">Multiply</tt>, etc.
Python does a magical thing: you can just call <tt class="docutils literal">calculate()</tt> on whatever
operator. Python handles part of the if/else for you. It calls the right calculate
method. That's one of the nice things about OO programming.
Also nice: adding the printing functionality means adding a print method on the various
classes. The rest of the code can remain the same. Nice.
But... it is easy to imagine that the objects grow and grow and grow. Instead of a "god
function" you get a "god object" with tens of methods... An object does a lot of things,
so the one object probably needs multiple libraries. If all the formatting-related code
would have been in one file, only that file would have needed the formatting
library. With OO code, you tend to have a file per object, so every file needs every
library and functionality is split over multiple files.
OO advantages: behavior next to data, easy to add more methods, avoids dispatching
duplication. Drawbacks: behaviour spread over classes, scattered dependencies,
dispatching is magical.
Dynamic dispatching. Python has <tt class="docutils literal">from functools import singledispatch</tt>, since a
long time. It isn't widely known. Only one person at the conference had used it in
anger! You can use it like this:
<pre class="literal-block">
@singledispatch
def my_print(value):
print("not implemented")
@my_print.register(int)
def _(value):
print(f"Integer: {value}")
@my_print.register(str)
...
</pre>
So... you can have a function with multiple implementations for specific types. You can
have pretty dumb classes (dataclasses?) and register a <tt class="docutils literal">calculate()</tt> method for each
type. That way similar behavior can be kept together in the same file.
So... you get a lot of the benefits of OO programming without lots of the drawbacks.
Dynamic dispatch advantages: simple classes, functionality is grouped, isolated
dependencies, code organized on correct axis. Downsides: data and behavior separate,
less encapsulation, a bit more friction when adding new classes. When you add a class,
you have to remember which functions you need to add in which files.
Another drawback is that the functions tend to poke around a bit in the objects in a way
that's not really clean, but... we're using python so objects aren't as sacrosanct as in
other languages and we're dependent upon our own good behavior anyway :-)
His summary:
<ul class="simple">
<li>One big function: fine to start, but quickly deteriorates.</li>
<li>OO: good when classes share behaviurs and when larger systems are strongly
interconnected.</li>
<li>Dynamic dispatch: good when data types are shared, but larger systems are indipendent.</li>
<li>Mixing OO and dynamic dispatch: good when both cohesion in the small and system
decoupling in the large are needed.</li>
</ul>
</div>
]]>
</content>
</entry>
<entry>
<title>Pycon NL: From state machines to event-sourced systems - Lukáš Ševčík</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2024/10/10/9-fintech.html" />
<id>http://reinout.vanrees.org/weblog/2024/10/10/9-fintech.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2024-10-10T00:00:00+01:00</published>
<updated>2024-10-10T13:49:00+01:00</updated>
<category term="pycon" />
<category term="python" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/2024/10/10/index.html">my summaries</a> of
the one-day <a class="reference external" href="https://pycon-nl.org/">Pycon NL</a> conference in
Utrecht, NL).
Full title: Events in fintech: from state machines to event-sourced systems.
He works at kiwi.com, a payment processor. They recently refactored their systems to be
more event-based.
Previously, they used state machines. A payment/customer/whatever can be in several
well-defined states. Between states (but not necessarily between all states) there are
transitions.
Advantages of state machines: they're simple. Easy to understand. They're
efficient. Easy to test: just states and transitions.
Disadvantages: there's no inherent history. That's sometimes bad, as it is hard to
answer "why is my payment in this state?!?" There's also lack of context. And changing a
state machine can be hard.
Some problems they encountered: race conditions. Two states are started for the same
account. One gets a +15, the other a -20. As the states don't know about each other, the
resulting account balance can be incorrect.
Now on to events. Event sourcing / event driven architecture is what he called
it. You start with a command "withdraw money, amount = 15". This gets placed in a queue
and gets processed. The processing results in another event "15 has been removed" that
gets send to the database.
Events are immutable facts about things that have happened in the system. They are
always in the past tense and include all relevant data. And: avoid internal implemention
details if possible.
"Event sourcing": you can re-construct the current state of the system by looking at all
the events. The current state has a source: the accumulation of all events. Great for
having a complete audit trail. You can also reconstruct past states ("you have
programmatic access to the past" :-) ). You also have the possibility to actually solve
race conditions that occurred.
If you have the time, you can even go towards the <a class="reference external" href="https://martinfowler.com/bliki/CQRS.html">CQRS</a> pattern (command query responsibility
segregation).
There are challenges of event sourcing. Eventual consistency: if you encounter a
problem, you can fix it. But the fix changes history, so the system is "eventual
consistent", not "really really always consistent". Also: there are higher storage
requirements. Complexity also is higher.
The challenges have possible solutions. The "saga pattern" has start and end events. You
could also try some locking mechanism. An "optimistic concurrency control". Not always
possible, but handy: "idempotent events", events that you can apply a couple of times
after each other without doing any harm.
</div>
]]>
</content>
</entry>
<entry>
<title>Pycon NL: Using PyPI trusted publishing for ansible releases - Anwesha Das</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2024/10/10/8-pypi-trusted-publishing.html" />
<id>http://reinout.vanrees.org/weblog/2024/10/10/8-pypi-trusted-publishing.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2024-10-10T00:00:00+01:00</published>
<updated>2024-10-10T13:14:00+01:00</updated>
<category term="pycon" />
<category term="python" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/2024/10/10/index.html">my summaries</a> of
the one-day <a class="reference external" href="https://pycon-nl.org/">Pycon NL</a> conference in
Utrecht, NL).
Using software is easy, releasing software is harder.
Originally trained as a laywer, she wondered why software had to be released so
often. As a lawyer, she had to work with laws sometimes already written in 1860 :-)
Nowadays she is the release manager of ansible.
Ansible is ansible-core in combination with 95+ different python modules called
collections.
Originally, releasing to the python package index, pypi, wasn't really safe. Every
person doing the release needed some <tt class="docutils literal">__token__</tt> in their <tt class="docutils literal">~/.pypirc</tt>. This can be
compromised. And the token can be overscoped. And... can you be sure that every person
doing the release is doing it in a safe way? That the person's laptop is secure enough?
Pypi now allows you to use trusted publishing. OIDC, "openID connect", is used
behind the scenes to connect pypi to github/gitlab/etc. It is a way to create
short-lived tokens to upload to pypi from a github/gitlab job.
A specific github repository and specific github action within that repository is set up
as "trusted" by one of the maintainers of the project on pypi. The github action will,
when uploading, use the OIDC mechanism to request a short-lived access token from
pypi. It then uses the token to upload the release to pypi.
(Personal note: I'm using it myself for a couple of projects and it works like a
charm).
Ansible's own release github action is here:
<a class="reference external" href="https://github.com/ansible-community/ansible-build-data/blob/main/.github/workflows/ansible-release.yml">https://github.com/ansible-community/ansible-build-data/blob/main/.github/workflows/ansible-release.yml</a>
</div>
]]>
</content>
</entry>
<entry>
<title>Pycon NL: Strategies for avoiding package.json for fullstack python devs - Donatas Rasiukevicius</title>
<link rel="alternate" type="text/html"
href="http://reinout.vanrees.org/weblog/2024/10/10/7-fullstack-python-package-json.html" />
<id>http://reinout.vanrees.org/weblog/2024/10/10/7-fullstack-python-package-json.html</id>
<author>
<name>Reinout van Rees</name>
</author>
<published>2024-10-10T00:00:00+01:00</published>
<updated>2024-10-10T12:31:00+01:00</updated>
<category term="pycon" />
<category term="python" />
<category term="django" />
<content type="html"><![CDATA[
<div class="document">
(One of <a class="reference external" href="https://reinout.vanrees.org/weblog/2024/10/10/index.html">my summaries</a> of
the one-day <a class="reference external" href="https://pycon-nl.org/">Pycon NL</a> conference in
Utrecht, NL).
In 2024, what is a fullstack python dev? Well, python. Database
administrator. Integrator with external services. A bit of terraform or kubernetes. AWS
clicky-clicky expert. And a bit of frontend.
So the frontend is only a small part of your job. You have a <tt class="docutils literal">package.json</tt> that makes
no sense to you, so you just <tt class="docutils literal">npm install</tt> it and download the entire internet into
your local <tt class="docutils literal">node_modules/</tt> folder. But somehow it seems to work.
The frontend? You have frameworks. More frameworks. Meta-frameworks that framework your
frameworks. Frameworks don't make things simpler, they just move the complexity
somewhere else. If you use django's ORM, you don't have to deal with the complexity of
SQL. But in exchange you have to learn the django ORM.
He wants to look at three layers:
<ul class="simple">
<li>Markup / html.</li>
<li>Styling / css / design systems.</li>
<li>Interactivity / javascript.</li>
</ul>
Markup and html. On one hand you can have bindings. "Fasthtml", for instance. A
mapping from python to html. <tt class="docutils literal">return Div(H1("hurray"))</tt>. You just move the complexity.
Or "native web components". You have custom <tt class="docutils literal"><star-rating></tt> elements that get
translated into actual html. You need specific javascript code for this, so it isn't
really portable between frameworks.
Another alternative: templating. <tt class="docutils literal">Jinja2</tt> is used in most programming languages. You
can do some templating, but it quickly gets unreadable.
All these solutions are great in their own way, but also suck in their own way.
Styling/css. This is an area that actually saw a lot of improvements in the last
years! CSS now supports variables out of the box, so no need for "sass" or so anymore.
You used to use bootstrap, jquery and a few other things and try to style your divs and
spans. You don't need to do that anymore: there is more than just span and div
nowadays. Classless: you use html's new elements such as <tt class="docutils literal"><article></tt> and get
something not-too-bad out of the box. You don't use custom class statemens anymore.
CSS has its own utility frameworks now, like tailwind. He dislikes tailwind (just use a
style on your element...). For the rest, css seems in pretty good shape.
Interactivity/javascript. Javascript used to be essential for things like hovers
and tooltips. But: that's build into html/css now! No need for javascript for this.
You could look at web assembly. <a class="reference external" href="https://pyscript.net/">https://pyscript.net/</a> for running python in the
browser. Nice. But you need to know both the internal browser API and the webassembly
bindings and get python to run... He looks at this field about once every half year to
see if it is ready for normal use.
HTMX could be nice. <a class="reference external" href="https://htmx.org/">https://htmx.org/</a> . Interactivity for your html page with
auto-replacing of certain parts of the page without you needing to do javascript. It is
pretty popular, but he found lots of the functionality pretty hard to use. After two
years, he found out he used only one part of it most of the time. So he wrote some small
javascript thingy to do just one simple kind of replacement.
Interactivity: most of it sucks.
Summary: there is no one silver bullet for your project. In many cases you're going
to benefit from building something yourself. So: if there's no silver bullet, just
produce a lot of regular bullets. Small custom self-made one-offs.
</div>
]]>
</content>
</entry>
</feed>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

Download the "valid Atom 1.0" banner.
Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)
Add this HTML to your page (change the image src attribute if necessary):

<a href="http://www.feedvalidator.org/check.cgi?url=http%3A//reinout.vanrees.org/weblog/atom.xml"><img src="valid-atom.png" alt="[Valid Atom 1.0]" title="Validate my Atom 1.0 feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//reinout.vanrees.org/weblog/atom.xml

Home · About · News · Docs · Terms