![\"Operating](\"/assets/articles/ebpf-go/arch.png\")
In the previous article, we explored how slices work under the hood. Today, we'll dive into another fundamental Go data structure: maps. We'll see how Go implements maps as hash tables, how they handle collisions, and how they grow dynamically.
\n\nThis will be a two part series. This first part will cover the old
\nmapimplementation which was in use until Go 1.24. The second part will cover the newmapimplementation using swiss tables.
The mini note below covers the basics of hash tables and how they are implemented in simple terms. Feel free to skip it if you already know the basics.
\nComing back to the main topic, let's see how maps are implemented in Go.
\nIn Go, before version 1.24, a map is a hash table implementation that stores key-value pairs. Unlike slices which use integer indices, maps allow you to use any comparable1 type as a key to access values efficiently. A type is comparable if values of that type can be compared with == and != operators.
Here's how a map is defined in the Go runtime:
\n 1type hmap struct {\n 2 count int\n 3 flags uint8\n 4 B uint8\n 5 noverflow uint16\n 6 hash0 uint32\n 7 buckets unsafe.Pointer\n 8 oldbuckets unsafe.Pointer\n 9 nevacuate uintptr\n10 extra *mapextra\n11}\nWhen you create a map like m := make(map[string]int), Go allocates this hmap structure under the hood. The map you're working with is actually a pointer to this structure, which is why maps behave like reference types in Go2.
The important fields in this data structure are:
\ncount: Number of key-value pairs currently stored in the mapB: Number of bucket bits (not the bucket count itself). The actual number of buckets is 2^B (or 1 << B in Go). So if B is 3, you have 8 buckets.buckets: Pointer to an array of buckets where key-value pairs are storedhash0: A random seed used by the hash function. This seed is generated when the map is created and it makes the hash function non-deterministic.\nThe
\noldbucketsandnevacuatefields are used during map growth, which we'll explore in detail later.
Let's start with a simple map creation:
\n1userAges := map[string]int{\n2 "alice": 25,\n3 "bob": 30,\n4 "carol": 28,\n5}\nThis creates a map with string keys and integer values. But how does Go organize this data internally?
\nWhen you work with a map, the Go runtime doesn't just store your key-value pairs in a simple array. Instead, it uses a hash table with buckets. Think of buckets as containers that hold multiple key-value pairs, and the hash table is an array of these buckets.
\nThe key idea is, when you insert a key-value pair, Go hashes the key to determine which bucket it belongs to. All keys that hash to the same bucket are stored together. This organization allows for fast lookups. Go can hash your key once and immediately know which bucket to search in, rather than scanning through all the data.
\nAnd here's how our userAges map looks like internally, at a high level:
You can see the map has 8 buckets (numbered 0-7), which means B = 3 since 2^3 = 8.
Each colored cell represents a bucket that contains at least one key-value pair. The three keys we inserted, "alice", "bob", and "carol" were hashed and distributed across different buckets in the array. Notice they didn't all go to the same bucket; the hash function distributed them to help balance the load.
The gray buckets are empty, ready to receive new entries as we add more key-value pairs to the map.
\nGo maps use overflow pointers to handle collisions. When a bucket becomes full, instead of reorganizing everything, Go simply allocates a new overflow bucket and links it to the full one, creating a short chain.
\nNow let's look at what's inside each bucket. Each bucket in Go can hold up to 8 key-value pairs. You can see that this value is hardcoded in Go source code:
\n1package abi\n2\n3const (\n4\t// Maximum number of key/elem pairs a bucket can hold.\n5\tMapBucketCountBits = 3 // log2 of number of elements in a bucket.\n6\tMapBucketCount = 1 << MapBucketCountBits\n7)\nHere's the actual bucket structure:
\n1type bmap struct {\n2 tophash [abi.MapBucketCount]uint8\n3}\nThe tophash array is an interesting optimization. It stores the top 8 bits of each key's hash value. When Go searches for a key in a bucket, it first scans the tophash array. Only when it finds a matching tophash, the key comparison is done. This mean that Go can quickly skip over slots that definitely don't contain the key, without having to compare complex key types.
\nThe actual keys and values are stored after the
\ntophasharray, but they're not visible in the struct definition because their types are determined at runtime. Go knows the types of your keys and values at compile time, so it can calculate the correct memory layout dynamically.
If you search through the runtime code in Go source code (as I did), you can find the actual creation process of a bucket:
\n 1func MapBucketType(t *types.Type) *types.Type {\n 2\t// Builds a type representing a Bucket structure for\n 3\t// the given map type. This type is not visible to users -\n 4\t// we include only enough information to generate a correct GC\n 5\t// program for it.\n 6\t// Make sure this stays in sync with runtime/map.go.\n 7\t//\n 8\t//\tA "bucket" is a "struct" {\n 9\t//\t tophash [abi.MapBucketCount]uint8\n10\t//\t keys [abi.MapBucketCount]keyType\n11\t//\t elems [abi.MapBucketCount]elemType\n12\t//\t overflow *bucket\n13\t//\t }\nLet's see how our example map looks with bucket details:
\nHere, you're looking at bucket 0 of our userAges map. Let's break down what each part means:
The tophash array: [ab, cd, ef, 00, 00, 00, 00, 00]
The first three slots contain tophash values (0xAB, 0xCD, 0xEF). These are the top 8 bits from hashing "alice", "bob", and "carol" respectively. The remaining five slots show 0x00, indicating they're empty and available for new entries.
The keys array: [alice, bob, carol, _, _, _, _, _]
The first three positions hold our actual keys. The empty positions (shown with commas or underscores) are unused slots waiting for new key-value pairs.
\nThe values array: [25, 30, 28, _, _, _, _, _]
The values are stored in the same order as their corresponding keys. Alice's age (25) is at index 0, Bob's (30) at index 1, and Carol's (28) at index 2.
\nWhen you insert a key-value pair into a map, Go needs to decide which bucket to place it in. This is where the hash function comes in to play.
\nLet's say we're inserting "gokce": 25 into our map. Here's what Go does step by step.
First, Go applies a hash function3 to the key along with the map's random seed (hash0):
1hash := hashFunction("gokce", map.hash0)\n2// Let's say this produces: 0xAB7F3C2E4D1A6B8F (64-bit hash)\nGo then uses the lower B bits of the hash to select which bucket to use. If our map has B = 3 (meaning 8 buckets), Go takes the lowest 3 bits:
1bucketIndex := hash & ((1 << B) - 1) // Bitwise AND with mask\n2// If hash ends in ...110 (binary), bucketIndex = 6\nThis logic is defined in the bucketMask function in Go source code:
1// bucketShift returns 1<<b, optimized for code generation.\n 2func bucketShift(b uint8) uintptr {\n 3\t// Masking the shift amount allows overflow checks to be elided.\n 4\treturn uintptr(1) << (b & (goarch.PtrSize*8 - 1))\n 5}\n 6\n 7// bucketMask returns 1<<b - 1, optimized for code generation.\n 8func bucketMask(b uint8) uintptr {\n 9\treturn bucketShift(b) - 1\n10}\nThis is much faster than using the modulo operator (%). Since the bucket count is always a power of 2, we can use a bitwise AND instead.
After that, Go takes the top 8 bits of the hash and stores them in the bucket's tophash array and finds an empty slot in the selected bucket and stores the tophash, key, and value there.
Let's see how different keys get distributed across buckets:
\nSo what happens when multiple keys hash to the same bucket? This is called a collision, and it's inevitable in hash tables. The hash function tries to distribute keys evenly, but with a limited number of buckets some will inevitably end up with more entries than others.
\nLet's say we have bucket 0 with three entries: "alice", "bob", and "carol". Now we keep adding more keys that also hash to bucket 0 (oops!)
1userAges["dave"] = 35 // Hashes to bucket 0, slot 3\n2userAges["eve"] = 22 // Hashes to bucket 0, slot 4\n3userAges["frank"] = 32 // Hashes to bucket 0, slot 5\n4userAges["grace"] = 29 // Hashes to bucket 0, slot 6\n5userAges["henry"] = 31 // Hashes to bucket 0, slot 7\nAt this point, bucket 0 has all 8 slots filled. What happens when we try to add one more key that hashes to the same bucket?
\n1userAges["ivan"] = 26 // Also hashes to bucket 0... but it's full!\nGo doesn't reorganize the entire map at this point. Instead, it allocates an overflow bucket and links it to the original bucket. The overflow bucket is just another bmap structure -with the actual key and value arrays defined dynamically in runtime- and the original bucket's overflow pointer now points to it.
Here's how overflow buckets work:
\nThe overflow bucket is linked to the main bucket, creating a short chain. When Go looks up a key in bucket 0, it first checks the main bucket, and if the key isn't found there, it follows the overflow pointer to check the overflow bucket.
This chaining approach is simple and efficient for small numbers of overflows. However, if you have many overflow buckets, lookups become slower (you have to scan multiple buckets). That's why Go monitors overflow bucket usage and triggers map growth when there are too many of them.
\nWhen you access a value with value := userAges["alice"], it might look like a simple operation, but there's quite a bit happening under the hood. Let's take a look at what happens when a map value is accessed by a key.
"alice" using the map's hash function and seed:1hash := hashFunction("alice", hash0) // e.g., 0xAB7F3C2E4D1A6B8F\nB bits of the hash, Go calculates which bucket to search the key inside:1bucketIndex := hash & ((1 << B) - 1) // e.g., bucket 6\n2tophash := hash >> (64 - 8) // e.g., 0xAB\ntophash array looking for a match. Let's say bucket 6 looks like this:tophash: [0xAB, 0xCD, 0xEF, 0x00, 0x00, 0x00, 0x00, 0x00]\nkeys: ["alice", "bob", "carol", ...]\nGo scans the tophash array, sees that the value at index 0 matches the tophash value of the key, and now it knows exactly where to look at. After retrieving the value at the same index in the values section of the bucket and returns it.
\nIf Go scans all tophash entries (including any overflow buckets) without finding a match, it returns the zero value of the map's value type.
\n1value, ok := userAges["alice"]\n2\n3if ok {\n4 fmt.Printf("Alice is %d years old\\n", value)\n5} else {\n6 // value is 0 (zero value for int)\n7 fmt.Printf("Alice not found, got zero value: %d\\n", value)\n8}\nWhen accessing a non-existent key, Go returns the zero value of the map's value type (0 for numbers, "" for strings, nil for pointers, etc.). The two-value form (value, ok) lets you distinguish between a key that exists with a zero value and a key that doesn't exist at all.
\nA return of
\n("", true)and("", false)are not the same. The first one means that there was actually an entry in the map with the specified key, which value is an empty string. The second one means that there were no value associated with that key.
This is why checking if m["key"] != 0 can be ambiguous, the key might not exist, or it might exist with a zero value. Always use the two-value form when you need to distinguish these cases.
As you add more key-value pairs to a map, performance would degrade if Go kept using the same number of buckets. With more entries squeezed into the same buckets, lookups would require scanning longer bucket chains, making operations slower.
\nThat's why maps grow dynamically. Go monitors the map's health and triggers growth when performance would suffer.
\n\nGo runtime decides on whether to grow the map or not is based on load factor, which is the average number of key-value pairs per bucket.
\n
A map grows when either of these conditions is met:
\nThe average number of key-value pairs per bucket exceeds 6.5 (load factor threshold). This is calculated as:
\n1loadFactor := count / (1 << B) // count / numberOfBuckets\n2if loadFactor > 6.5 {\n3 // Trigger growth\n4}\nWhy 6.5? That number comes from 80% threshold, as it's defined in Go runtime source code. It's a balance between memory usage and performance, a sweet spot where you don't waste too much memory but also don't have too many overflow buckets yet.
\nLet's see an example. Say we have a map with 4 buckets that has accumulated 28 entries:
\nBefore Growth (load factor = 7.0):
\n
The load factor is 28 / 4 = 7.0, which exceeds the 6.5 threshold. Many buckets are overflowing, creating long chains that slow down lookups.
\nIn this case, the runtime performs a regular growth by doubling the bucket count. Here's what happens:
\nGo increments B by 14, which doubles the bucket count. If you had 4 buckets (B == 2), you now have 8 buckets (B == 3). The new bucket array is allocated, but the old buckets remain in place.
The hmap.buckets pointer is updated to point to the new bucket array, and hmap.oldbuckets is set to point to the old buckets. The map is now in a growing state.
Runtime starts an incremental evacuation process, where it doesn't move all key-value pairs immediately. Instead, it evacuates entries gradually during normal map operations. We'll discuss this in detail in the next section.
\nAfter Growth (load factor = 3.5):
\n
Now with 8 buckets, the load factor drops to 28 / 8 = 3.5, which is well below the threshold. The entries are redistributed more evenly, and overflow chains are reduced or eliminated.
\nEven with a low load factor, if there are too many overflow buckets relative to the base bucket count, Go triggers a different type of growth. This happens when keys cluster unevenly, some buckets overflow while others stay mostly empty.
\nIn this case, Go performs a same-size growth which reorganizes the data without adding more buckets. Runtime allocates a new bucket array with the same size as the current one (B stays the same). As entries are evacuated from old buckets to new ones, overflow chains are broken up. Keys that previously shared a bucket with many others might now be distributed more evenly, reducing overflow buckets. This process eliminates overflow buckets that accumulated over time due to deletions or poor initial distribution, making the map more cache-friendly.
\nWhy would a map grow without increasing bucket count?
\n
When keys cluster in specific buckets due to hash patterns, you might have many overflow buckets even with a low overall load factor. For example, you might have a map with 100 entries across 16 buckets (load factor = 6.25, below the threshold), but if 80 entries landed in just 2 buckets, you'd have many overflow buckets. Same size growth reorganizes the data to improve performance without allocating more buckets.
\nHere's how this logic is implemented in Go source code:
\n 1// incrnoverflow increments h.noverflow.\n 2// noverflow counts the number of overflow buckets.\n 3// This is used to trigger same-size map growth.\n 4// See also tooManyOverflowBuckets.\n 5// To keep hmap small, noverflow is a uint16.\n 6// When there are few buckets, noverflow is an exact count.\n 7// When there are many buckets, noverflow is an approximate count.\n 8func (h *hmap) incrnoverflow() {\n 9 ...\n10}\nHere's one of the cool aspects of Go's map implementation: incremental evacuation. Instead of moving all entries at once when the map grows (which could freeze your program for milliseconds on a large map), Go runtime spreads the work across many operations.
\nHere's how it works:
\nhmap in the oldbuckets and buckets fields. When you perform any operation on the map, Go checks if the map is growing:1if h.growing() {\n2 growWork(t, h, bucket) // Evacuate some pairs\n3}\nWhenever a value is inserted to, deleted from, or looked up in the map, Go evacuates at least one bucket (and possibly more) from the old array to the new array. The nevacuate field in the hmap structure tracks which old buckets have been evacuated.
During the growth period, when you look up a key, Go might need to check the old buckets if that bucket hasn't been evacuated yet. The evacuation state markers in the tophash array (we'll see these shortly) tell Go whether to look in the old or new buckets.
An example code snippet from the Go source code, shows how the runtime handles an edge case of iterating through key/value pairs in the map during a grow:
\n 1 if checkBucket != noCheck && !h.sameSizeGrow() {\n 2 // Special case: iterator was started during a grow to a larger size\n 3 // and the grow is not done yet. We're working on a bucket whose\n 4 // oldbucket has not been evacuated yet. Or at least, it wasn't\n 5 // evacuated when we started the bucket. So we're iterating\n 6 // through the oldbucket, skipping any keys that will go\n 7 // to the other new bucket (each oldbucket expands to two\n 8 // buckets during a grow).\n 9\n10 ...\n11 }\noldBucketIndex + oldBucketCountThis incremental approach prevents "stop-the-world" pauses. Even with millions of entries, each individual operation only pays a small, bounded cost for evacuation. The map remains responsive throughout the growth process.
\n\nMaps with interface{} (or any) values are interesting because Go doesn't know the size of the values at compile time. An interface{} can hold anything: an int (which might be 8 bytes), a string (which is 16 bytes: pointer + length), or a complex struct.
Let's see an example:
\n1data := make(map[string]interface{})\n2data["number"] = 42\n3data["text"] = "hello"\n4data["slice"] = []int{1, 2, 3}\nHow does Go handle this? The values stored in the bucket aren't the actual data, they're interface values, which are represented with two pointers in runtime: one for the type, and one for the value. This means Go can still calculate a fixed bucket layout even though the concrete types can change.
\nThe actual data (42, "hello", and the []int{1, 2, 3} slice) is stored elsewhere in memory, and the interface values in the bucket point to them. This adds a level of abstraction, which is why interface maps can be slightly slower than maps with concrete types.
Here's something crucial to understand if you're writing concurrent Go code: maps are not safe for concurrent access.
\nThere are two specific panics in Go runtime:
\n"concurrent map read and map write""concurrent map writes" 1// UNSAFE - Will cause fatal error\n 2m := make(map[string]int)\n 3\n 4go func() {\n 5 for i := 0; i < 1000; i++ {\n 6 m["key"] = i // Write\n 7 }\n 8}()\n 9\n10go func() {\n11 for i := 0; i < 1000; i++ {\n12 _ = m["key"] // Read\n13 }\n14}()\n15\n16go func() {\n17 for i := 1000; i < 2000; i++ {\n18 m["key"] = i // Write again\n19 }\n20}()\nThis code will likely crash within the first few iterations.
\nNotice that concurrent reading only in a map is not a panic-invoking action. However most of the time you will need to use maps for both read and write in different goroutines concurrently.
\nSolution: Use sync.Map, mutex protection, or channels to coordinate access!
The most common solution is to protect the map with a sync.Mutex:
1// SAFE - Using sync.Mutex\n 2var mu sync.Mutex\n 3m := make(map[string]int)\n 4\n 5// Writer goroutine\n 6go func() {\n 7 for i := 0; i < 1000; i++ {\n 8 mu.Lock()\n 9 m["key"] = i\n10 mu.Unlock()\n11 }\n12}()\n13\n14// Reader goroutine\n15go func() {\n16 for i := 0; i < 1000; i++ {\n17 mu.Lock()\n18 _ = m["key"]\n19 mu.Unlock()\n20 }\n21}()\nBy holding the lock using mu.Lock() before entering the critical sections, we ensure only one goroutine accesses the map at a time.
We can also use defer to automatically release the lock at the return of the goroutine. This is also a well-known pattern in Go:
1// SAFE - Using sync.Mutex with defer\n 2func writer(m map[string]int, mu *sync.Mutex, start, stop int) {\n 3 mu.Lock()\n 4 defer mu.Unlock() // This function will be called right before the return\n 5\n 6 for i := start; i < stop; i++ {\n 7 key := fmt.Sprintf("key-%d", i)\n 8 m[key] = i\n 9 }\n10}\n11\n12var mu sync.Mutex\n13m := make(map[string]int)\n14\n15// Writer goroutines\n16go writer(m, &mu, 0, 100)\n17go writer(m, &mu, 100, 200)\n18go writer(m, &mu, 200, 300)\nFor read-heavy workloads, you can use sync.RWMutex to allow multiple concurrent readers (but still only one writer at a time). For specialized use cases, sync.Map provides a concurrent map implementation optimized for specific access patterns.
Now that we've explored both slices and maps in detail, let's compare these two fundamental Go data structures. While they both store collections of data, they're designed for very different use cases.
\n| Feature | \nSlices | \nMaps | \n
|---|---|---|
| Index Type | \nInteger only | \nAny comparable type | \n
| Ordering | \nOrdered by index | \nNo guaranteed order | \n
| Growth | \nAppend to end | \nHash-based placement | \n
| Memory Layout | \nContiguous array | \nHash table with buckets | \n
| Lookup Time | \nO(1) by index | \nO(1) average by key | \n
| Concurrent Access | \nSafe for read-only | \nSafe for read-only | \n
\n\n\nBoth slices and maps grow dynamically, but maps are more complex internally due to hash collision handling and bucket management. Slices give you predictable iteration order, while maps give you flexible key types.
\n
runtime/map.go (map internals)See The Go Blog for more information. ↩︎
\nThe actual hash function used by Go is runtime/aeshash for most types, which uses AES instructions when available for security and performance. Different types may use optimized hash functions (e.g., strings have their own specialized hash). ↩︎
\nThe bucket count is always a power of 2, which allows Go to use bitwise operations for bucket selection instead of expensive modulo operations. This is why B represents the power (2^B buckets) rather than the count directly. ↩︎
\nI was at WeAreDevelopers World Congress 2024 in Berlin, Germany. Talked about the basics of eBPF, why you may need to use it, and how to write simple eBPF programs with Go.
\nThis talk was a polished version of my previous talk at GoConf Istanbul 2024.
\n", "url": "https://sazak.io/talks/an-applied-introduction-to-ebpf-with-go-2024-07-18", "title": "An Applied Introduction to eBPF with Go", "summary": "I was at WeAreDevelopers World Congress 2024 in Berlin, Germany. I talked about the basics of eBPF, why you may need it, and how to write eBPF programs with Go.", "image": "https://sazak.io/assets/talks/an-applied-introduction-to-ebpf-with-go-2024-07-18.jpeg", "date_modified": "2024-07-18T00:00:00.000Z", "author": { "name": "Ozan Sazak", "url": "https://sazak.io" } }, { "id": "https://sazak.io/articles/an-applied-introduction-to-ebpf-with-go-2024-06-06", "content_html": "\n\nThis article is from the talk I gave at the Go Konf Istanbul '24 conference.
\n
Most of the time we are developing software or even using software, we are playing within the safe boundaries of the operating system. We don't even know how that IP packet was welcomed from the network interface, or those inodes were handled by the filesystem when we save a file.
\nThat boundary is called the user space, and it's where we write our applications, libraries, and tools. But there's another world, which is the kernel space. It's where the operating system's kernel resides, and it's responsible for managing the system's resources, such as memory, CPU, and I/O devices.
We usually don't need to go below the sockets or file descriptors, but sometimes we need to. Let's say you want to profile an application to see how much resource it consumes.
\nIf you profile the application from the user space, you will not only miss too many useful details, but also you will consume a significant amount of resources for profiling itself, because each layer on top of the CPU or memory introduces some overhead.
\nLet's say you want to go down the stack and somehow insert your custom code into the kernel to profile the application, or to trace the system calls, or to monitor the network packets. How would you do that?
\nTraditionally you have two options.
\nIf you want to change the Linux kernel source code and then ship the same kernel to you customer's machine, you will need to convince the Linux kernel community that the change is required. Then, you will need to wait for several years for the new kernel version to be adopted by the Linux distributions.
\nThis is not a practical approach for most of the cases, and it's also a little
\nmuch for just profiling an application, or monitoring the network packets.
You can write a kernel module, which is a piece of code that can be loaded into the kernel and executed. This is a more practical approach, but it has its own risks and downsides.
\nFirst, you need to write a kernel module, which is not an easy task. Then, you need to maintain it regularly, because the kernel is a living thing, and it changes over time. If you don't maintain your kernel module, it will be outdated and won't work with the new kernel versions.
\nSecond, you are risking corrupting your Linux kernel, because kernel modules don't have security boundaries. If you write a kernel module that has a bug, it can crash the whole system.
\neBPF (Extended Berkeley Packet Filter) is a revolutionary technology that allows you to reprogram the Linux kernel within minutes, even without rebooting the system.
\neBPF allows you to trace system calls, user space functions, library functions, network packets, and much more. It's a powerful tool for systems performance, monitoring, securtiy, and much more.
\neBPF is a system consisting of several components:
\n\n\nNote that I have used the term "BPF" and "eBPF" interchangeably. eBPF stands for "Extended Berkeley Packet Filter". BPF was originally introduced to Linux to filter network packets, but eBPF extends the original BPF to allow it to be used for other purposes. Today it's not related to Berkeley, and it's not only for filtering packets.
\n
Below is an illustration of how eBPF works in both user space and under the hood. eBPF programs are written in a high-level language, such as C, and then compiled to eBPF bytecode. Then, the eBPF bytecode is loaded into the kernel and executed by the eBPF virtual machine.
An eBPF program is attaced to a specific code path in the kernel, such as a system call. These code paths are called "hooks". When the hook is triggered, the eBPF program is executed and now it performs the custom logic you have written. This way we can run our custom code in the kernel space.
![\"Operating](\"/assets/articles/ebpf-go/ebpf.png\")
Before moving on to the details, let's write a simple eBPF program to trace the execve system call. We will write the program in C, the user space program in Go, and then run the user space program which will load the eBPF program into the kernel, and poll the custom events we will emit from the eBPF program, right before the actual execve system call is executed.
Let's start writing the eBPF program first. I will write part by part to explain the details better, but you can find the whole program in my GitHub repo: ozansz/intro-ebpf-with-go.
\n 1#include "vmlinux.h"\n 2#include <bpf/bpf_helpers.h>\n 3\n 4struct event {\n 5 u32 pid;\n 6 u8 comm[100];\n 7};\n 8\n 9struct {\n10\t__uint(type, BPF_MAP_TYPE_RINGBUF);\n11\t__uint(max_entries, 1000);\n12} events SEC(".maps");\nHere we import the vmlinux.h header file, which contains the kernel's data structures and function prototypes. Then we include the bpf_helpers.h header file, which contains helper functions for eBPF programs.
Then we define a struct to hold the event data, and then we define a BPF map to store the events. We will use this map to communicate the events between the eBPF program, which will run in kernel space, and the user space program.
\n\nWe will go into the details of BPF maps later, so don't worry if you don't understand why we used
\nBPF_MAP_TYPE_RINGBUF, or what isSEC(".maps")for.
We are now ready to write our first program and define the hook that it will be attached to:
\n 1SEC("kprobe/sys_execve")\n 2int hello_execve(struct pt_regs *ctx) {\n 3 u64 id = bpf_get_current_pid_tgid();\n 4 pid_t pid = id >> 32;\n 5 pid_t tid = (u32)id;\n 6\n 7 if (pid != tid)\n 8 return 0;\n 9\n10 struct event *e;\n11\n12\te = bpf_ringbuf_reserve(&events, sizeof(struct event), 0);\n13\tif (!e) {\n14\t\treturn 0;\n15\t}\n16\n17\te->pid = pid;\n18\tbpf_get_current_comm(&e->comm, 100);\n19\n20\tbpf_ringbuf_submit(e, 0);\n21\n22\treturn 0;\n23}\nHere we define a function, hello_execve, and attach it to the sys_execve system call using the kprobe hook. kprobe is one of many hooks that eBPF provides, and it's used to trace kernel functions. This hook will trigger our hello_execve function right before the sys_execve system call is executed.
Inside the hello_execve function, we first get the process ID and the thread ID, and then we check if they are the same. If they are not the same, that means we are in a thread, and we don't want to trace threads, so we exit the eBPF program by returning zero.
We then reserve space in the events map to store the event data, and then we fill the event data with the process ID and the command name of the process. Then we submit the event to the events map.
It's pretty simple until now, right?
\nBefore starting to write the user space program, let me briefly explain what the program needs to do in user space. We need a user space program to load the eBPF program into the kernel, create the BPF map, attach to the BPF map, and then read the events from the BPF map.
\nTo perform these operations, we need to use a specific system call. This system call is called bpf(), and it's used to perform several eBPF-related operations, such as reading the contents of a BPF map.
We can call this system call ourselves from the user space as well, but it means too many low-level operations. Thankfully there are libraries that provide a high-level interface to the bpf() system call. One of them is Cilium's ebpf-go package, which we will use in this example.
Let's dive into some Go code.
\n 1//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -type event ebpf hello_ebpf.c\n 2\n 3func main() {\n 4\tstopper := make(chan os.Signal, 1)\n 5\tsignal.Notify(stopper, os.Interrupt, syscall.SIGTERM)\n 6\n 7\t// Allow the current process to lock memory for eBPF resources.\n 8\tif err := rlimit.RemoveMemlock(); err != nil {\n 9\t\tlog.Fatal(err)\n10\t}\n11\n12\tobjs := ebpfObjects{}\n13\tif err := loadEbpfObjects(&objs, nil); err != nil {\n14\t\tlog.Fatalf("loading objects: %v", err)\n15\t}\n16\tdefer objs.Close()\n17\n18\tkp, err := link.Kprobe(kprobeFunc, objs.HelloExecve, nil)\n19\tif err != nil {\n20\t\tlog.Fatalf("opening kprobe: %s", err)\n21\t}\n22\tdefer kp.Close()\n23\n24\trd, err := ringbuf.NewReader(objs.Events)\n25\tif err != nil {\n26\t\tlog.Fatalf("opening ringbuf reader: %s", err)\n27\t}\n28\tdefer rd.Close()\n29\n30 ...\nThe first line is a Go compiler directive, go:generate. Here we say to the Go compiler to run the bpf2go tool from the github.com/cilium/ebpf/cmd/bpf2go package, and generate a Go file from the hello_ebpf.c file.
The generated Go files will include the Go representation of the eBPF program, the types and structs we have defined in the eBPF program, etc. We then will use these representations inside our Go code to load the eBPF program into the kernel, and to interact with the BPF map.
\nWe then use the generated types to load the eBPF program (loadEbpfObjects), attach to the kprobe hook (link.Kprobe), and read the events from the BPF map (ringbuf.NewReader). All of these functions use the generated types.
It's time to interact with the kernel side:
\n 1 ...\n 2\n 3\tgo func() {\n 4\t\t<-stopper\n 5\n 6\t\tif err := rd.Close(); err != nil {\n 7\t\t\tlog.Fatalf("closing ringbuf reader: %s", err)\n 8\t\t}\n 9\t}()\n10\n11\tlog.Println("Waiting for events..")\n12\n13\tvar event ebpfEvent\n14\tfor {\n15\t\trecord, err := rd.Read()\n16\t\tif err != nil {\n17\t\t\tif errors.Is(err, ringbuf.ErrClosed) {\n18\t\t\t\tlog.Println("Received signal, exiting..")\n19\t\t\t\treturn\n20\t\t\t}\n21\t\t\tlog.Printf("reading from reader: %s", err)\n22\t\t\tcontinue\n23\t\t}\n24\n25\t\tif err := binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event); err != nil {\n26\t\t\tlog.Printf("parsing ringbuf event: %s", err)\n27\t\t\tcontinue\n28\t\t}\n29\n30\t\tprocName := unix.ByteSliceToString(event.Comm[:])\n31\t\tlog.Printf("pid: %d\\tcomm: %s\\n", event.Pid, procName)\n32\t}\n33}\nWe start a goroutine to listen to the stopper channel, which we defined in the previous Go snippet. This channel will be used to stop the program gracefully when we receive an interrupt signal.
We then start a loop to read the events from the BPF map. We use the ringbuf.Reader type to read the events, and then we parse the event data using the binary.Read function, into the ebpfEvent type, which is generated from the eBPF program.
We then print the process ID and the command name of the process to the standard output.
\nNow we are ready to run the program. First, we need to compile the eBPF program, and then run the user space program.
\n1$ go generate\n2Compiled /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x01-helloworld/ebpf_bpfel.o\n3Stripped /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x01-helloworld/ebpf_bpfel.o\n4Wrote /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x01-helloworld/ebpf_bpfel.go\n5Compiled /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x01-helloworld/ebpf_bpfeb.o\n6Stripped /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x01-helloworld/ebpf_bpfeb.o\n7Wrote /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x01-helloworld/ebpf_bpfeb.go\n8\n9$ go build -o hello_ebpf\nWe first run the go generate command to compile the eBPF program, and then we run the go build command to compile the user space program.
Then we run the user space program:
\n1sudo ./hello_ebpf\n2hello_ebpf: 01:20:54 Waiting for events..\nI'm running this program inside a VM in Lima, why not open another shell and see what will happen?
\n1limactl shell intro-ebpf\n2\n3$\nMeanwhile in the first shell:
\n1hello_ebpf: 01:22:22 pid: 3360\tcomm: sshd\n2hello_ebpf: 01:22:22 pid: 3360\tcomm: bash\n3hello_ebpf: 01:22:22 pid: 3361\tcomm: bash\n4hello_ebpf: 01:22:22 pid: 3362\tcomm: bash\n5hello_ebpf: 01:22:22 pid: 3363\tcomm: bash\n6hello_ebpf: 01:22:22 pid: 3366\tcomm: bash\n7hello_ebpf: 01:22:22 pid: 3367\tcomm: lesspipe\n8hello_ebpf: 01:22:22 pid: 3369\tcomm: lesspipe\n9hello_ebpf: 01:22:22 pid: 3370\tcomm: bash\nAs expected, we are seeing that the sshd process is starting, and then the bash process is starting, and then the lesspipe process is starting, and so on.
This is a simple example of how we can use eBPF to trace the execve system call, and then read the events from the BPF map in the user space. We wrote a fairly simple yet powerful program, and we intercepted the execve system call without modifying the kernel source code or restarting the system.
So, what actually happened in the previous example? We attached the eBPF program to the sys_execve system call using the kprobe hook, to run the hello_execve function wheneve the sys_execve system call is called, right before the original system call is code is executed.
![\"\"](\"/assets/articles/ebpf-go/hello-world.png\")
eBPF is event driven, meaning that it expects us to attach the eBPF program to a specific code path in the kernel. These code paths are called "hooks", and there are several types of hooks that eBPF provides. The most common ones are:
\nkprobe, kretprobe: Trace kernel functionsuprobe, uretprobe: Trace user space functionstracepoint: Trace predefined tracepoints in the kernelxdp: eXpress Data Path, used to filter and redirect network packetsusdt: User Statically Defined Tracing, used to trace user space functions in a more efficient wayThe hooks kprobe and uprobe are used to invoke the attached eBPF programs before the function/syscall execution, and kretprobe and uretprobe are used to invoke the attached eBPF programs after the function/syscall execution.
We also used a BPF map to store the events. BPF maps are data structures to store and communicate different kinds of data. We also use them for state management. There are too many types of BPF maps supported, and we use different types of maps for different purposes. Some of the most common BPF map types are:
\nBPF_MAP_TYPE_HASH: A hash mapBPF_MAP_TYPE_ARRAY: An arrayBPF_MAP_TYPE_RINGBUF: A ring bufferBPF_MAP_TYPE_STACK: A stackBPF_MAP_TYPE_QUEUE: A queueBPF_MAP_TYPE_LRU_HASH: A least recently used hash mapSome of these map types also have per-CPU variants, such as BPF_MAP_TYPE_PERCPU_HASH, which is a hash map with a separate hash table for each CPU core.
Let's take a step further and write a more complex eBPF program. This time we will use the XDP hook to invoke the eBPF program right after the network interface sends a network packet to the kernel, even before the kernel processes the packet.
![\"\"](\"/assets/articles/ebpf-go/xdp.png\")
We will write an eBPF program to count the number of incoming IP packets by the source IP address and port number, and then we will read the counts from the BPF map in the user space. We will parse the ethernet, IP and TCP/UDP headers of each packet, and store the counts of the valid TCP/UDP packets in the BPF map.
\nFirst, the eBPF program:
\n 1#include "vmlinux.h"\n 2#include <bpf/bpf_helpers.h>\n 3#include <bpf/bpf_endian.h>\n 4\n 5#define MAX_MAP_ENTRIES 100\n 6\n 7/* Define an LRU hash map for storing packet count by source IP and port */\n 8struct {\n 9\t__uint(type, BPF_MAP_TYPE_LRU_HASH);\n10\t__uint(max_entries, MAX_MAP_ENTRIES);\n11\t__type(key, u64); // source IPv4 addresses and port tuple\n12\t__type(value, u32); // packet count\n13} xdp_stats_map SEC(".maps");\nLike the first example, we will include the vmlinux.h and BPF helper headers. We also define a map, xdp_stats_map, to store the IP:ports and packet count information. We wil then populate this map inside the hook function and read the contents in the user space program.
What I mean by IP:ports is basically a u64 value, packed with the source IP, source port and the destination port. The IP address (IPv4, specifically) is 32-bits long, and each port number is 16-bits long, so we need exactly 64 bits to store all three - that's why we are using u64 here. We are only processing the ingress (incoming packets) here so we won't need to store the destination IP address.
Different from the last example we now used BPF_MAP_TYPE_LRU_HASH as the map type. This type of map lets us store a (key, value) pair as a hashmap with LRU variant.
See how we defined the map here, we explicitly set the number of maximum entries, and the types of the map key and values. For key we are using a 64-bit unsigned integer and for value a 32-bit unsigned integer.
\n\n\nThe maximum value of
\nu32is2^32 - 1, which is more than enough packets for the sake of this example.
To learn the IP address and port number, we first need to parse the packet and read the ethernet, IP and then TCP/UDP headers.
\nAs XDP is placed right after the network interface card, we will be given the raw packet data in bytes, so we will need to manually walk on the byte array and unmarshal the ethernet, IP and TCP/UDP headers.
\nHopefully, we have all the header definitions (struct ethhdr, struct iphdr, struct tcphdr, and struct udphdr), inside vmlinux.h header file. We will use these structs to extract the IP address and port number information in a separate function, parse_ip_packet:
1#define ETH_P_IP\t\t0x0800\t/* Internet Protocol packet\t*/\n 2\n 3#define PARSE_SKIP \t\t\t0\n 4#define PARSED_TCP_PACKET\t1\n 5#define PARSED_UDP_PACKET\t2\n 6\n 7static __always_inline int parse_ip_packet(struct xdp_md *ctx, u64 *ip_metadata) {\n 8\tvoid *data_end = (void *)(long)ctx->data_end;\n 9\tvoid *data = (void *)(long)ctx->data;\n10\n11\t// First, parse the ethernet header.\n12\tstruct ethhdr *eth = data;\n13\tif ((void *)(eth + 1) > data_end) {\n14\t\treturn PARSE_SKIP;\n15\t}\n16\n17\tif (eth->h_proto != bpf_htons(ETH_P_IP)) {\n18\t\t// The protocol is not IPv4, so we can't parse an IPv4 source address.\n19\t\treturn PARSE_SKIP;\n20\t}\n21\n22\t// Then parse the IP header.\n23\tstruct iphdr *ip = (void *)(eth + 1);\n24\tif ((void *)(ip + 1) > data_end) {\n25\t\treturn PARSE_SKIP;\n26\t}\n27\n28\tu16 src_port, dest_port;\n29\tint retval;\n30\n31\tif (ip->protocol == IPPROTO_TCP) {\n32\t\tstruct tcphdr *tcp = (void*)ip + sizeof(*ip);\n33\t\tif ((void*)(tcp+1) > data_end) {\n34\t\t\treturn PARSE_SKIP;\n35\t\t}\n36\t\tsrc_port = bpf_ntohs(tcp->source);\n37\t\tdest_port = bpf_ntohs(tcp->dest);\n38\t\tretval = PARSED_TCP_PACKET;\n39\t} else if (ip->protocol == IPPROTO_UDP) {\n40\t\tstruct udphdr *udp = (void*)ip + sizeof(*ip);\n41\t\tif ((void*)(udp+1) > data_end) {\n42\t\t\treturn PARSE_SKIP;\n43\t\t}\n44\t\tsrc_port = bpf_ntohs(udp->source);\n45\t\tdest_port = bpf_ntohs(udp->dest);\n46\t\tretval = PARSED_UDP_PACKET;\n47\t} else {\n48\t\t// The protocol is not TCP or UDP, so we can't parse a source port.\n49\t\treturn PARSE_SKIP;\n50\t}\n51\n52\t// Return the (source IP, destination IP) tuple in network byte order.\n53\t// |<-- Source IP: 32 bits -->|<-- Source Port: 16 bits --><-- Dest Port: 16 bits -->|\n54\t*ip_metadata = ((u64)(ip->saddr) << 32) | ((u64)src_port << 16) | (u64)dest_port;\n55\treturn retval;\n56}\nThe function:
\nh_proto of struct ethhdr and protocol of struct iphdr. Each header stores the protocol of the inner packet it wraps.IP:ports tuple inside a 64-bit unsigned integer (u64)PARSE_SKIP)Notice the __always_inline at the beginning of the function signature. This tells the compiler to always inline this funcion as static code, which saves us from performing a function call.
Now it's time to write the hook function and use parse_ip_packet:
1SEC("xdp")\n 2int xdp_prog_func(struct xdp_md *ctx) {\n 3\tu64 ip_meta;\n 4\tint retval = parse_ip_packet(ctx, &ip_meta);\n 5\t\n 6\tif (retval != PARSED_TCP_PACKET) {\n 7\t\treturn XDP_PASS;\n 8\t}\n 9\n10\tu32 *pkt_count = bpf_map_lookup_elem(&xdp_stats_map, &ip_meta);\n11\tif (!pkt_count) {\n12\t\t// No entry in the map for this IP tuple yet, so set the initial value to 1.\n13\t\tu32 init_pkt_count = 1;\n14\t\tbpf_map_update_elem(&xdp_stats_map, &ip_meta, &init_pkt_count, BPF_ANY);\n15\t} else {\n16\t\t// Entry already exists for this IP tuple,\n17\t\t// so increment it atomically.\n18\t\t__sync_fetch_and_add(pkt_count, 1);\n19\t}\n20\n21\treturn XDP_PASS;\n22}\nThe xdp_prog_func is fairly simple as we already coded most of the program logic inside parse_ip_packet. What we do here is:
parse_ip_packetXDP_PASSIP:ports tuple in BPF map keys using the bpf_map_lookup_elem helper functionIP:ports tuple is seen the first time, else increment it by one. The __sync_fetch_and_add is an LLVM built-in hereFinally we attach this function to the XDP subsystem by using the SEC("xdp") macro.
It's time to dive into Go code again.
\n 1//go:generate go run github.com/cilium/ebpf/cmd/bpf2go ebpf xdp.c\n 2\n 3var (\n 4 ifaceName = flag.String("iface", "", "network interface to attach XDP program to")\n 5)\n 6\n 7func main() {\n 8\tlog.SetPrefix("packet_count: ")\n 9\tlog.SetFlags(log.Ltime | log.Lshortfile)\n10 flag.Parse()\n11\n12\t// Subscribe to signals for terminating the program.\n13\tstop := make(chan os.Signal, 1)\n14\tsignal.Notify(stop, os.Interrupt, syscall.SIGTERM)\n15\n16\tiface, err := net.InterfaceByName(*ifaceName)\n17\tif err != nil {\n18\t\tlog.Fatalf("network iface lookup for %q: %s", *ifaceName, err)\n19\t}\n20\n21\t// Load pre-compiled programs and maps into the kernel.\n22\tobjs := ebpfObjects{}\n23\tif err := loadEbpfObjects(&objs, nil); err != nil {\n24\t\tlog.Fatalf("loading objects: %v", err)\n25\t}\n26\tdefer objs.Close()\n27\n28\t// Attach the program.\n29\tl, err := link.AttachXDP(link.XDPOptions{\n30\t\tProgram: objs.XdpProgFunc,\n31\t\tInterface: iface.Index,\n32\t})\n33\tif err != nil {\n34\t\tlog.Fatalf("could not attach XDP program: %s", err)\n35\t}\n36\tdefer l.Close()\n37\n38\tlog.Printf("Attached XDP program to iface %q (index %d)", iface.Name, iface.Index)\n39\n40 ...\nHere we first load the generated eBPF program and map using the loadEbpfObjects function. Then we attach the program to the specified network interface using the link.AttachXDP function. Similar to the previous example we used a channel to listen to the interrupt signal and close the program gracefully.
Next, we will read the map contents in every second and print the packet counts to the standard output:
\n 1 ...\n 2\n 3 ticker := time.NewTicker(time.Second)\n 4\tdefer ticker.Stop()\n 5\tfor {\n 6\t\tselect {\n 7\t\tcase <-stop:\n 8\t\t\tif err := objs.XdpStatsMap.Close(); err != nil {\n 9\t\t\t\tlog.Fatalf("closing map reader: %s", err)\n10\t\t\t}\n11\t\t\treturn\n12\t\tcase <-ticker.C:\n13\t\t\tm, err := parsePacketCounts(objs.XdpStatsMap, excludeIPs)\n14\t\t\tif err != nil {\n15\t\t\t\tlog.Printf("Error reading map: %s", err)\n16\t\t\t\tcontinue\n17\t\t\t}\n18\t\t\tlog.Printf("Map contents:\\n%s", m)\n19\t\t\tsrv.Submit(m)\n20\t\t}\n21\t}\n22}\nWe will use a utility function, parsePacketCounts, to read the map contents and parse the packet counts. This function will read the map contents in a loop.
As we will be given raw bytes from the map, we will need to parse the bytes and convert them into a human-readable format. We will define a new type PacketCounts to store the parsed map contents.
1type IPMetadata struct {\n 2\tSrcIP netip.Addr\n 3\tSrcPort uint16\n 4\tDstPort uint16\n 5}\n 6\n 7func (t *IPMetadata) UnmarshalBinary(data []byte) (err error) {\n 8\tif len(data) != 8 {\n 9\t\treturn fmt.Errorf("invalid data length: %d", len(data))\n10\t}\n11\tif err = t.SrcIP.UnmarshalBinary(data[4:8]); err != nil {\n12\t\treturn\n13\t}\n14\tt.SrcPort = uint16(data[3])<<8 | uint16(data[2])\n15\tt.DstPort = uint16(data[1])<<8 | uint16(data[0])\n16\treturn nil\n17}\n18\n19func (t IPMetadata) String() string {\n20\treturn fmt.Sprintf("%s:%d => :%d", t.SrcIP, t.SrcPort, t.DstPort)\n21}\n22\n23type PacketCounts map[string]int\n24\n25func (i PacketCounts) String() string {\n26\tvar keys []string\n27\tfor k := range i {\n28\t\tkeys = append(keys, k)\n29\t}\n30\tsort.Strings(keys)\n31\n32\tvar sb strings.Builder\n33\tfor _, k := range keys {\n34\t\tsb.WriteString(fmt.Sprintf("%s\\t| %d\\n", k, i[k]))\n35\t}\n36\n37\treturn sb.String()\n38}\nWe defined a new type, IPMetadata, to store the IP:ports tuple. We also defined a UnmarshalBinary method to parse the raw bytes and convert them into a human-readable format. We also defined a String method to print the IP:ports tuple in a human-readable format.
We then defined a new type, PacketCounts, to store the parsed map contents. We also defined a String method to print the map contents in a human-readable format.
Finally, we will use the PacketCounts type to parse the map contents and print the packet counts:
1func parsePacketCounts(m *ebpf.Map, excludeIPs map[string]bool) (PacketCounts, error) {\n 2\tvar (\n 3\t\tkey IPMetadata\n 4\t\tval uint32\n 5\t\tcounts = make(PacketCounts)\n 6\t)\n 7\titer := m.Iterate()\n 8\tfor iter.Next(&key, &val) {\n 9\t\tif _, ok := excludeIPs[key.SrcIP.String()]; ok {\n10\t\t\tcontinue\n11\t\t}\n12\t\tcounts[key.String()] = int(val)\n13\t}\n14\treturn counts, iter.Err()\n15}\nWe first need to compile the eBPF program and then run the user space program.
\n1$ go generate\n2Compiled /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x03-packet-count/ebpf_bpfel.o\n3Stripped /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x03-packet-count/ebpf_bpfel.o\n4Wrote /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x03-packet-count/ebpf_bpfel.go\n5Compiled /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x03-packet-count/ebpf_bpfeb.o\n6Stripped /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x03-packet-count/ebpf_bpfeb.o\n7Wrote /Users/sazak/workspace/gocode/src/github.com/ozansz/intro-ebpf-with-go/0x03-packet-count/ebpf_bpfeb.go\n8\n9$ go build -o packet_count\nNow we can run it:
\n1$ sudo ./packet_count --iface eth0\n2packet_count: 22:11:10 main.go:107: Attached XDP program to iface "eth0" (index 2)\n3packet_count: 22:11:10 main.go:132: Map contents:\n4192.168.5.2:58597 => :22\t| 51\n5packet_count: 22:11:11 main.go:132: Map contents:\n6192.168.5.2:58597 => :22\t| 52\n7packet_count: 22:11:11 main.go:132: Map contents:\n8192.168.5.2:58597 => :22\t| 53\nThe packets coming to the port 22 from the IP address 192.168.5.2 are the SSH packets, as I am running this program inside a VM and I am SSHing into.
Let's run curl inside the VM in another terminal, and see what will happen:
1$ curl https://www.google.com/\nMeanwhile in the first terminal:
\n 1packet_count: 22:14:07 main.go:132: Map contents:\n 2172.217.22.36:443 => :38324\t| 12\n 3192.168.5.2:58597 => :22\t| 551\n 4packet_count: 22:14:08 main.go:132: Map contents:\n 5172.217.22.36:443 => :38324\t| 12\n 6192.168.5.2:58597 => :22\t| 552\n 7packet_count: 22:14:08 main.go:132: Map contents:\n 8172.217.22.36:443 => :38324\t| 30\n 9192.168.5.2:58597 => :22\t| 570\n10packet_count: 22:14:09 main.go:132: Map contents:\n11172.217.22.36:443 => :38324\t| 30\n12192.168.5.2:58597 => :22\t| 571\nWe are seeing the packets coming to the port 38324 from the IP address 172.217.22.36 are the packets coming from the curl command.
eBPF is powerful in many ways and I think it's a good technology to invest time in, especially you are in systems programming, observability or security. In this article we have seen what eBPF is, how it works, and how we can start using it with Go.
\nI hope you enjoyed this article and learned something new. If you have any questions, feel free to ping me.
\nDistributed Paper Stacks is the second iteration of Reading Stacks, a tech book reading club we started with a few friends from METU. We meet every two weeks to discuss a distributed systems paper.
\nWe are a group of friends from METU, Turkey.
\n\n", "url": "https://sazak.io/reading-groups/distributed-paper-stacks-2024-04-30", "title": "Distributed Paper Stacks", "summary": "We are oranizing the second iteration of Reading Stacks by reading a distributed systems paper bi-weekly.", "date_modified": "2024-04-30T00:00:00.000Z", "author": { "name": "Ozan Sazak", "url": "https://sazak.io" } }, { "id": "https://sazak.io/talks/homeops-building-a-gitops-driven-homelab-infrastructure-from-scratch-2024-04-20", "content_html": "", "url": "https://sazak.io/talks/homeops-building-a-gitops-driven-homelab-infrastructure-from-scratch-2024-04-20", "title": "HomeOps: Building a GitOps-driven Homelab Infrastructure From Scratch", "summary": "I gave an ignite talk at DevOpsDays Istanbul 2024 about building a GitOps-driven homelab infrastructure from scratch, which tools you can use, and how you can automate everything.", "image": "https://sazak.io/assets/talks/homeops-building-a-gitops-driven-homelab-infrastructure-from-scratch-2024-04-20.jpeg", "date_modified": "2024-04-20T00:00:00.000Z", "author": { "name": "Ozan Sazak", "url": "https://sazak.io" } }, { "id": "https://sazak.io/talks/openfaas-for-the-curious-hows-and-whys-2024-04-16", "content_html": "", "url": "https://sazak.io/talks/openfaas-for-the-curious-hows-and-whys-2024-04-16", "title": "OpenFaaS for the Curious: How's and Why's", "summary": "I talked at a local cloud meetup about starting with OpenFaaS, the internals of OpenFaaS runtimes, and showcased a demo on setting up OpenFaaS on Kubernetes.", "image": "https://sazak.io/assets/talks/openfaas-for-the-curious-hows-and-whys-2024-04-16.jpeg", "date_modified": "2024-04-16T00:00:00.000Z", "author": { "name": "Ozan Sazak", "url": "https://sazak.io" } }, { "id": "https://sazak.io/articles/visual-guide-to-slices-in-go-2024-03-25", "content_html": "We use slices everywhere in Go. With maps, they are fundamental data types we use to store data. Today we will dive into the internals and see how slices work under the hood.
\n\nAt the end of the article, you will find exercises to test your knowledge. Enjoy!
\n
In Go, slice is an abstraction over arrays. It consists of a pointer to the backing array, the length of the slice, and its capacity. The length is the number of elements in the slice, while the capacity is the number of elements in the underlying array.
\nHere's how a slice is defined in Go runtime:
\n1type slice struct {\n2\tarray unsafe.Pointer\n3\tlen int\n4\tcap int\n5}\nThe array field is a pointer to the (first element of the) backing array. The len field is the length of the slice, denoting the number of items in the slice, and the cap field is the capacity of the slice. Capacity stores the number of elements in the backing array, and it is the maximum number of elements the slice can hold without reallocating the backing array.
There are multiple ways to create a slice, and some of them differ in how they set the length and capacity of the slice. Let's say we have the following slice definition:
\n1s1 := []int32{1, 2, 3, 4, 5}\nThis creates an int32 slice s1 with some initial values. The length of the slice is 5, as you can see from the number of elements in the slice definition. The capacity is also set to 5 by default (we did not specify a different capacity), and the backing array is filled with the given values.
Below is how the backing array looks like in memory. Numbers inside the cells represent values of items in the array, and the gray cells represent the memory that does not belong to the slice. See how both length and capacity is 5, and slice points (ptr = 0x0100) at the address of the first element of the array (0x0100).
make-ing of a sliceWe can specify different lengths and capacities for a slice by using the make built-in function. The make function accepts two to three arguments for slice allocation: the type of the slice, the length, and the capacity. If the capacity is not given, it defaults to the length.
1s2 := make([]int, 5, 10)\nThe code above allocates a 10-capacity array and creates a slice s2, which points to this array with a length of 5. The capacity of the slice is also set to 10.
make also fills the array with the zero values1 of the item type. In this case, the zero value of int is 0.
\nNotice that all items of the backing array are filled with zero values, not only the
\nlenitems of the slice. From now on I won't show the zeroes in the unused parts of the backing array to keep the visuals more readable.
If we want to populate this slice with some values, we can set specific indexes:
\n1for i := range s2 {\n2 s2[i] = i * 2\n3}\nThis will set the values of the slice to 0, 2, 4, 6, 8, by using range2 to iterate over the indexes of the slice.
Notice how the slice is still pointing to the same underlying array, but the initial values are changed. Also the length and capacity of the slice are still the same.
\n\nWhy didn't
\nrangeiterate over all indexes within the capacity (10) of the array?
Good question. Let's see what happens if we try to set a value to an index that is out of the slice's length?
\n1s2[8] = 16\nIf we run the assignment above, Go runtime will panic with the message below:
\n1panic: runtime error: index out of range [8] with length 5\nThe capacity is not the space Go gives us to use, it's the maximum number of the elements slice can hold without reallocating the backing array. If we try to access or set a value to an index that is out of the slice's length, Go will panic.
\nOkay, so what if we add more elements by simply appending them to the slice?
1s2 = append(s2, []int{10, 12, 14, 16, 18}...)\nThe built-in function append works by pushing these items to the end of the slice, starting by the len-th item. Both the length and the capacity of the slice are 10 now.
What do you think will happen if we add one more item to the end of the slice now, as it's capacity is already full?
\nIf we have a full slice (len == cap), then Go runtime will reallocate the backing array, copy the existing elements to the new array, and append the new element.
\n\nThis was what I meant by "slices are abstraction over arrays". Go abstracts the memory management from the developer so you don't need to think if the backing array is full, or how much of an extra capacity should I allocate every time. In C for example, you would need to use
\ncallocorrealloc3 to handle the reallocation of the memory, manually.
The new capacity will be double the previous capacity until the slice reaches a certain capacity threshold. After that, the capacity will increase monotonically with different growth factors.
\nYou can see in other (older) resources that the capacity increases by 25% after 1024 items, but in 2021 the growth formula was changed to be smoother.
\nBelow is the theoretical4 growth factors by certain capacity thresholds:
\n| Starting Capacity | \nGrowth Factor | \n
|---|---|
| 256 | \n2.0 | \n
| 512 | \n1.63 | \n
| 1024 | \n1.44 | \n
| 2048 | \n1.35 | \n
| 4096 | \n1.30 | \n
So if we add one more item at the end of the slice s2 we had, the append function will allocate a new array with a capacity of 20, copy the existing elements to the new array, and append the new element.
1s2 = append(s2, 20)\nNotice how s2 now points to a different backing array with a capacity of 20. The old array at memory address 0x1234 will be garbage collected by the Go runtime5 if no other slice points to it (references it).
\nKeep in mind that the memory addresses I use in examples are just for demonstration purposes. In reality, the memory addresses are longer 32-bit or 64-bit integers and can change between runs.
\n
In many places you will see the append function used with the same slice as the first argument, like sl = append(sl, "hello").
This is because when the slice given exceeds the capacity with the given new elements and append needs to internally reallocate a new backing array, the new slice structure pointing to the new array will be returned from the function. The slice given in the first argument, however, will not be updated in-place.
That's why if you want the update to be reflected to the slice, you need to assign the result of the append function to the slice, as how append is used for most of the time.
It's good to know how using different variables for the result of the append function can affect the slices. Consider the following example:
1dogs := []string{"bulldog", "poodle", "beagle"}\n2moreDogs := append(dogs, "husky")\n3moreAndMoreDogs := append(moreDogs, "shiba inu")\nBefore the append, dogs has both length and capacity of 3, and points to the backing array with the elements "bulldog", "poodle", and "beagle". As dogs is at full capacity, the first append call will return a new slice, pointing to a new backing array filled with the initial elements and the new element "husky", with a capacity of 6.
Now moreDogs points to the new backing array, and dogs still points to the old backing array. In the second append call, we would not expect another reallocation happening, as the moreDogs slice has enough capacity to hold the new element "shiba inu". So the moreAndMoreDogs slice will point to the same backing array as moreDogs.
We can create a new slice from an existing slice by using the slicing expression. It's syntactically similar to what Python does with lists.
\nWhen we slice another slice, the new slice will point to the same backing array, but the length and capacity will be different, depending on the range we specify. The range is inclusive for the start index and exclusive for the end index.
\n1s1 := []byte{16, 32, 48, 64, 80}\n2s2 := s1[1:3]\n3\n4fmt.Printf("s1: %v, len: %d, cap: %d\\n", s1, len(s1), cap(s1))\n5fmt.Printf("s2: %v, len: %d, cap: %d\\n", s2, len(s2), cap(s2))\nThe code above will output:
\n1s1: [16 32 48 64 80], len: 5, cap: 5\n2s2: [32 48], len: 2, cap: 4\nLength of s2 is 2, as the range [1:3] includes the elements at indexes 1 and 2. The capacity will be the capacity of s1 minus the start index, which is 4.
If we look at the pointers of the slices, we will see that they don't point to the same memory address:
\n1fmt.Printf("s1: %p\\n", s1)\n2fmt.Printf("s2: %p\\n", s2)\n1s1: 0x0010\n2s2: 0x0011\nNotice how the pointer of s2 is one byte ahead of the pointer of s1. This is because the slice s2 starts from the second element of the slice s1, and as each element in slice s1 is one byte long (byte), the second item is one byte ahead of the first item.
\n
Let's play a little bit on our new slice s2. Can you guess the outcome of the following code?
1s2 = append(s2, 100, 101)\nRemember that s2 points to the same backing array as s1, and has enough capacity (4) to hold two more elements. So the new slice s2 will be:
And, as the slice s1 still points to the same backing array, it will look like the following in memory:
Notice how the slice s1 is also updated, as it points to the same backing array as s2.
Now that s2 is at full capacity, you know that if we append one more element to s2, the backing array will be reallocated, and the slice will point to a new array.
1s2 = append(s2, 102)\ns2 now points to a new backing array in a different memory address (0x0050), with a capacity of 8.
\nWhat will happen to
\ns1in this case?
Normally the old array would be garbage collected, but in this case, s1 still points to it, so it will not be garbage collected.
See how appending to a slice which shares the same backing array with another slice can affect both slices, until one exceeds its capacity and it has a new reallocated backing array.
\nDifferent slices using the same backing array may cause implicit problems sometimes. The new slice used in a function or in another part of the code appending new elements to the slice may cause the original slice to change its contents, which may not be the desired behavior.
\n\nHow can I prevent this behavior then?
\n
Go allows us to specify the capacity of the new slice by using a third argument in the slicing expression. This is useful when we want to limit the capacity of the new slice, to prevent the slice to change the contents out of the specified bounds of the backing array.
\nIf we would like to create a new slice from s1 starting from the second element and ending at the third element, with a capacity of 2, we can do it like this:
1s3 := s1[1:3:3]\nThis will create a new slice s3 with a length of 2 (indexes 1 and 2), a capacity of 2 as specified, and it will point to the second item (index 1) of the same backing array as s1. Notice that the third argument is not the capacity of the new slice, but in which index the capacity should end.
Now if we append a to s3, as s3 is at capacity (len(s3) == cap(s3) == 2) it will cause a reallocation of the used parts of the backing array, but it will not affect the contents of s1.
1s1 = []byte{16, 32, 48, 64, 80}\n2s3 := s1[1:3:3]\n3s3 = append(s3, 200, 201)\n
\n
Notice how the ptr of s3 is different as it now points to a new backing array with length of 4 (2 old elements copied and two newly appended), and capacity of 4 (twice as the old capacity). The contents of s1 after the index 2 are not affected by the append operation on s3.
The whole code snippet is below:
\n 1s1 := []byte{16, 32, 48, 64, 80}\n 2s2 := s1[1:3]\n 3\n 4s2 = append(s2, 100, 101)\n 5s2 = append(s2, 102)\n 6\n 7fmt.Printf("s1: %v, len: %d, cap: %d\\n", s1, len(s1), cap(s1))\n 8fmt.Printf("s2: %v, len: %d, cap: %d\\n\\n", s2, len(s2), cap(s2))\n 9\n10s1 = []byte{16, 32, 48, 64, 80}\n11s3 := s1[1:3:3]\n12\n13fmt.Printf("s3 initially: %v, len: %d, cap: %d\\n\\n", s3, len(s2), cap(s3))\n14\n15s3 = append(s3, 200, 201)\n16\n17fmt.Printf("s1: %v, len: %d, cap: %d\\n", s1, len(s1), cap(s1))\n18fmt.Printf("s3: %v, len: %d, cap: %d\\n", s3, len(s2), cap(s3))\n1s1: [16 32 48 100 101], len: 5, cap: 5\n2s2: [32 48 100 101 102], len: 5, cap: 8\n3\n4s3 initially: [32 48], len: 5, cap: 2\n5\n6s1: [16 32 48 64 80], len: 5, cap: 5\n7s3: [32 48 200 201], len: 5, cap: 8\n\n\nSpecifying the slice capacity is an option we can use to limit the new slice's capacity and actions on the items of backing array, to prevent unexpected changes in the original slice.
\n
Slices are a powerful abstraction in Go, allowing us to work with arrays in a more flexible way. Understanding how slices work internally helps us write more efficient and less error-prone code. I hope this article helped you gain insight on how slices work under the hood, and when to pay attention to implementation details to prevent unexpected behavior.
\nThere will be more of these visual guides to Go internals in the future, and there may be a part two to this article.
\nIf you have any questions or feedback, feel free to reach out to me on Twitter. Until next time, stay tuned.
\nZero value is the default value of an uninitialized type in Go. For example, the zero value of a bool is false, integer types are 0, and slices, functions, maps, etc. are nil. ↩︎
The range keyword is used to iterate over the indexes and (optionally) the values of an array or slice. It returns the index and the value of each element. If the result is decomposed into one variable such as i := range slice, it will only return the index. If it is decomposed into two variables such as i, v := range slice, it will return both the index and the value. Similarly, if you only need the value, you can use _ to discard the index, such as _, v := range slice. The range iteration behaves differently on maps and channels. ↩︎
The growth factors are not constant, in practice. To see the actual calculation of the "smooth" growth factors, you can check the nextslicecap function in Go runtime source code. ↩︎
For more info about the garbage collector, see tip.golang.org/doc/gc-guide ↩︎
\nOne of the things I like about Go is that when you want to use someone else's code in your project, you can just import it by its GitHub repo URL.
\nLiterally, you can just import the package by writing:
\n1import "github.com/google/go-cmp/cmp"\nAnd that's it. Although these URLs don't have to be GitHub URLs. Sometimes you see import paths like go.uber.org/atomic, or golang.org/x/crypto/ssh.
These are also valid import paths and they are called "vanity import paths". They are called "vanity" because they are custom domain names, which redirect to the actual repository.
\nThe work behind the scenes is pretty simple: When you run go get, or go install, Go tooling sends an HTTP request to this URL to get the package info, then downloads the source code under the vendor directory. The package info is provided in meta tags in the HTML response, which are:
go-import: Where to find the repository. It has three parts: the import path, the VCS type (git, hg, etc.), and the repository URL.
\n![\"\"](\"/assets/articles/go-sazak-io/go-import.png\")
go-source: Optional information for pkg.go.dev to link to the source code.
\n![\"\"](\"/assets/articles/go-sazak-io/go-source.png\")
The idea is, by simply returning these meta tags in the HTML response, you can make your own Go package index.
\nWhy not? I like building stuff and putting them in public repos. I also use my own packages/tools later in other projects. So, I thought it would be cool to step it up a bit and have my own domain for my Go packages.
\nIt's better to use this:
\n1import "go.sazak.io/my-package"\nInstead of this:
\n1import "github.com/ozansz/my-package"\nThat's how go.sazak.io was born. It's a simple Go package index for my own public Go packages. It's basically a static site built by using a JSON file that contains the package info, which is also generated by a simple Go tool.
\nWhen you think of designing an index which lists a number of packages, which link to their respective GitHub repositories, you might think of using a database, a backend server, and a frontend client.
\nWe tend to overcomplicate things as developers, but in my opinion it's the best solution to keep things as simple as possible while preserving the functionality and developer experience.
\n\n"Simple is better than complex." - Zen of Python
\n
The whole systems consists of two parts: a Go tool called sync-github-repos that collects and compiles the data, and the static site that uses the JSON file as the data source to build a static site.
![\"\"](\"/assets/articles/go-sazak-io/overall.png\")
The tool uses GitHub API to list my public repositories, then filters the ones that have a go.mod file which content starts with module go.sazak.io/. The data is then stored inside a Repo Go struct to later be marshaled into a JSON file:
1type Repo struct {\n 2\tOwner string `json:"owner"`\n 3\tName string `json:"name"`\n 4\tStars uint `json:"stars"`\n 5\tDescription string `json:"desc"`\n 6\tGoPackage string `json:"go_package"`\n 7\tLatestTag string `json:"latest_tag"`\n 8\tAlphaRelease bool `json:"alpha_release"`\n 9\tHasCLIApp bool `json:"has_cli_app"`\n10\tPackages map[string]string `json:"packages"`\n11\tMasterBranch string `json:"master_branch"`\n12}\nThen it checks if there are any public packages (containers) of the repository, are there any releases/tags, and if the latest tag is a pre-release. It also checks the name of the default branch, which is usually main or master.
The HasCLIApp field is a boolean that tells if the repository includes a CLI app in the cmd directory. This field is set if the repo has a cli tag.
Also the AlphaRelease field is a boolean that tells if the latest tag is a pre-release or if the tag is below version v1.0.0.
The JSON file is then used as the data source for the static site. The site is built with Next.js1, Shadcn UI2 and deployed to Vercel.
\nAs I explained before, the website only needs to serve two meta tags in the HTML response. For a basic requirement like this I could just use plain HTML, but I thought it would be better to have a clean and informative UI for the human visitors.
\nThat's why I decided to use Shadcn UI and design a one-page package listing site. I used the Card component to display the package info for all packages. The basic information is same for all packages, for example the package name, latest tag badge, instructions to clone the repository, etc.
![\"\"](\"/assets/articles/go-sazak-io/card-1.png\")
Cards also contain different sections, based on what package provides. If it's a CLI app, it shows the instructions to install the app. If it has packages (containers), it also shows the instructions to pull the containers using docker pull.
![\"\"](\"/assets/articles/go-sazak-io/card-2.png\")
After putting it all together, the site looks like this:
\n![\"\"](\"/assets/articles/go-sazak-io/website.png\")
Which, I think, is pretty cool for a simple Go package index, and also for a backend/platform engineer like me to build.
\nTo make it easier to deploy the website with the synced data, I created a GitHub Actions workflow pre-commit hook that runs the sync-github-repos tool to generate the JSON file and put it inside the UI project:
1#!/bin/sh\n 2\n 3set -eo pipefail\n 4\n 5go run cmd/sync-github-repos/main.go --username ozansz --hostname go.sazak.io --out ui/src/repos.json --force-authenticated\n 6if [ $? -ne 0 ]; then\n 7 echo "Pre-commit hook failed (content generation script)"\n 8 exit 1\n 9fi\n10\n11git update-index --add ui/src/repos.json\nAfter I push the changes, the Vercel deployment pipeline is triggered and the site is updated with the new data.
\n![\"\"](\"/assets/articles/go-sazak-io/deploy.png\")
It's pretty simple -and cool- to have your own Go package index. It's a good way to keep track of your own packages publicly.
\nIf you liked the idea, you can also build your own package index. My implementation for go.sazak.io is open source and available on GitHub. I tried to make it as simple as possible to use and extend.
\nFeel free to fork it and build your own package index!
\nI had the opportunity to speak at Go Konf Istanbul about developing eBPF programs with Go, and how we use eBPF at Edge Delta. Go Konf was organized by Istanbul Gophers, a local Go community in Istanbul, and was the first in-person Go conference in Istanbul, as I recall. There were many great talks (I especially liked Fatih's talk on the new expeimental function iterators).
\n", "url": "https://sazak.io/talks/an-applied-introduction-to-ebpf-with-go-2024-02-17", "title": "An Applied Introduction to eBPF with Go", "summary": "I had the opportunity to speak at Go Konf Istanbul about developing eBPF programs with Go, and how we use eBPF at Edge Delta. Go Konf was the first in-person Go conference in Istanbul, as I recall, and there were many great talks. Happy to be a part of it.", "image": "https://sazak.io/assets/talks/an-applied-introduction-to-ebpf-with-go-2024-02-17.jpeg", "date_modified": "2024-02-17T00:00:00.000Z", "author": { "name": "Ozan Sazak", "url": "https://sazak.io" } }, { "id": "https://sazak.io/reading-groups/street-coder---sedat-kapanoglu-2023-11-07", "content_html": "Street Coder is the first book we read in our tech book reading club, Reading Stacks. We meet every two weeks to discuss a chapter or two from a tech book. We started with 'Street Coder' by Sedat Kapanoglu. The book is written for self-taught developers, talking about foundational concepts and best practices in software development.
\nSome of the chapters were a little much opinionated, but it was a good read overall. I wouldn't say I learned something new, but it was a good refresher on some of the topics, and also I felt challenged to think about some of the topics from a different perspective.
\nWe are a group of friends from METU, Turkey.
\n\n", "url": "https://sazak.io/reading-groups/street-coder---sedat-kapanoglu-2023-11-07", "title": "Street Coder - Sedat Kapanoglu", "summary": "Reading Stacks is a tech book reading club we started with a few friends from METU. We meet every two weeks to discuss a chapter or two from a tech book. We started with 'Street Coder' by Sedat Kapanoglu.", "date_modified": "2023-11-07T00:00:00.000Z", "author": { "name": "Ozan Sazak", "url": "https://sazak.io" } }, { "id": "https://sazak.io/articles/fantastic-side-projects-and-where-to-find-them-2023-09-03", "content_html": "So you are a newbie developer looking for a project idea to apply what you've learned in that Udemy course, or maybe an experienced engineer searching for the next side project to work on for fun. Either way, you need a creative idea to bring to life.
\nIt's a fun way to apply what you have learned in the courses/books and an excellent skill to craft the tool you just need to automate that boring stuff you need to do every day.
\nHere, I will list a few sources/techniques I used to find cool project ideas while learning programming and now to craft useful tools for my needs. This blog post is a little bit opinionated. Everyone has a unique learning curve, so if you find my suggestions harder or easier than expected, it's okay to modify them to fit your level.
\n![\"\"](\"/assets/articles/project-complexity-by-time-graph.png\")
...and looking for project ideas. You probably know Python or JavaScript a little bit, or maybe C/Go, and want to dive into interesting real-life stuff.
\nIn this stage, you should not dive right into big and complex projects. I know it's exciting, but hear my reasons:
\nHowever, the most beneficial thing you can do is code simple yet not tedious stuff using the programming topics you learned to jump to advanced levels where you can work on complex projects as fast as possible.
\nI suggest coding simple versions of those complex programs, such as creating little terminal-based games. You don't need to learn GUI, a self-made hangman game is still fun after all. Also, while building one, you will need to use most of the topics you've learned/are learning: fundamental data types, variables, control flow (conditionals, loops), terminal input/output, etc…
\nAnother one may be a simple chatbot in the terminal. It gives you a list of prompt options, you select one, and it prints a defined answer for the fixed prompt. It's like a super simple version of ChatGPT - you don't need to design and code all the frontend, backend, and ML stuff.
\nPerfect! Now you are ready to step up your game a little bit and start learning and applying CS fundamentals in your programs to reach a higher level of complexity:
\nIn this stage, you can add more functionality to the programs you have created until now, adding dynamism by implementing:
\n\nThe idea is forcing yourself to use a new data structure/algorithm in every project you create.
\n
Let's say you have a program that gives you the information of students when you enter the name of one. You can enhance this program to run in a loop that takes the student name, gives the info, then asks the user if they want to update the information. To do this, you will need to use a data structure (linked list, for example) instead of hardcoded data. Then, you will need to use a search algorithm to find specific students given the name.
\nAdding more functionality pushes you to utilize new stuff in your code, which is an excellent way to learn and practice new concepts in programming.
\n\nThis is the time you need to start learning more advanced stuff. Such as:
\nIn this stage, you can try to design and implement your own version of existing stuff you use, such as creating your own programming language and writing a compiler for it, trying to write your own database engine with indexes, basic queries, and maybe aggregations.
\nYou don't have to go with a fully functional clone; for example, creating a programming language with just integer types as the native type is a good exercise, and it's still challenging!
\nHere are some practical resources and project ideas:
\nint, list, dict, etc.) are implemented, how did they handle memory management, and so on.Some good resources worth reading/checking for compiler/interpreter design:
\nThis blog post became longer than I expected, and it's been a while since I started writing, so I will write another one just for this subtopic 🤷♂️
\nI hope you found this content useful. I will add sequels to this post, writing about my side projects or other resources to find fun side project ideas in the future.
\nI've started reading technical books after my graduation from CS to broaden my knowledge on many topics, such as system design, low-level CS, cloud native, etc. Nowadays, I'm reading Cloud Native Go by Matthew Titmus, which gives a smooth introduction to the cloud native world using the benefits of Go. The book starts with Go basics and cloud native patterns, and that's where I thought "I should write about these patterns!".
\nIn this series of blog posts, I will write about the cloud native patterns I've learnt – also with illustrations using Manim.
\nLet's say we have a constant data source (such as a Go channel), and we need to process the data and put the processed data into a destination channel with a latency as low as possible.
\nIn the general case, we have a processor function which is in the middle of source and destinations, which processes the data packets (packet here is an abstraction for simplicity) one by one.
\n![\"\"](\"/assets/articles/fan-in-fan-out-1.png\")
However, every processing function have latency. This may be because of network or CPU usage, blocking syscalls, etc. If we send enough packets per second to the processor function, voilà! We now have a bottleneck!
\n![\"\"](\"/assets/articles/fan-in-fan-out-2.png\")
There's a fairly simple solution to this problem: using multiple processors inside the data pipeline. This way, we can process the data stream concurrently, which drops the overall latency and decreases the pipeline congestion.
\n![\"\"](\"/assets/articles/fan-in-fan-out-3.png\")
We can implement this solution by utilizing shared memory, such as message queues.
\nWith this approach, we will split the incoming data packets into different input queues. Then, each queue's own processor will take the packets one by one, process the data, and put the processed data into its related output queue. Finally, the destination (another processor, queue, or some other system) will take the processed data packets from each of the output queues, and aggregate them into a single data source.
The first approach, splitting the data source (input) to multiple data sources (input queues) is called the Fan-out pattern. The second one, aggregating multiple data sources (output queues) into a single data source (destination) is called the Fan-in pattern.
\n\nFor simplicity, we've specified one input queue and one output queue for each processor in our pipeline. We could use multiple input/output queues per processor, or a shared queue between a few processors, depending on the overall system requirements.
\n
![\"\"](\"/assets/articles/fan-in-fan-out-4.png\")
Let's get our hands dirty with some concurrent Go! First, let's define our data sources:
\n 1func stringGenerator(prefix string, count uint) <-chan string {\n 2 ch := make(chan string)\n 3\n 4 go func() {\n 5 defer close(ch)\n 6 for i := uint(0); i < count; i++ {\n 7 ch <- fmt.Sprintf("%s-%d", prefix, i)\n 8 }\n 9 }()\n10\n11 return ch\n12}\nThe function stringGenerator creates a receive-only string channel, creates a goroutine that puts prefixed strings into the channel, and returns the channel. We read from this channel later in the Fan-out code.
Our processor function will be fairly simple too:
\n 1func stringProcessor(id int, source <-chan string) <-chan string {\n 2 ch := make(chan string)\n 3\n 4 go func() {\n 5 defer close(ch)\n 6 for data := range source {\n 7 ch <- fmt.Sprintf("P-%d processed message: %s", id, data)\n 8 millisecs := rand.Intn(901) + 100\n 9 time.Sleep(time.Duration(millisecs * int(time.Millisecond)))\n10 }\n11 }()\n12\n13 return ch\n14}\nInside the processor function, we will wait for a random while to simulate the processor latency.
\nIn the Fan-out implementation, we will take a receive-only channel, and return a slice of receive-only channels:
\n 1// Splitter implements the Fan-out pattern\n 2func Splitter(source <-chan string, numDests uint) []<-chan string {\n 3 destinations := make([]<-chan string, 0, numDests)\n 4\n 5 for i := uint(0); i < numDests; i++ {\n 6 dest := make(chan string)\n 7\n 8 go func() {\n 9 defer close(dest)\n10 for data := range source {\n11 dest <- data\n12 }\n13 }()\n14\n15 destinations = append(destinations, dest)\n16 }\n17\n18 return destinations\n19}\nGoroutines created inside the Splitter function will handle the data routing logic. Notice that inside the goroutines, we used a single range statement to receive from the source channel:
1for data := range source {\n2 dest <- data\n3}\nThis means each goroutine will try to read from the channel in the loop, and the first one to do the read will receive the next item. In other words, each goroutine will fight over the next data instance.
\nWe could use a centralized solution for Fan-out instead of competing goroutines. In that case, we would define a master process to distribute each incoming data instance (string in our case) among all the output channels.
In Fan-in, we will basically do the reverse of Fan-out, with a few differences:
\n 1// Aggregator implements the Fan-in pattern\n 2func Aggregator(sources []<-chan string) <-chan string {\n 3 destination := make(chan string)\n 4\n 5 var wg sync.WaitGroup\n 6 wg.Add(len(sources))\n 7\n 8 for _, source := range sources {\n 9 go func(src <-chan string) {\n10 defer wg.Done()\n11 for data := range src {\n12 destination <- data\n13 }\n14 }(source)\n15 }\n16 go func() {\n17 wg.Wait()\n18 close(destination)\n19 }()\n20\n21 return destination\n22}\nThe Aggregator function takes a slice of receive-only input sources and returns a single receive-only output channel. Inside, we created a goroutine for each input source, which reads from the source continuously and populates the output channel (destination) with the data it received.
Notice that we've used a sync.WaitGroup to wait for the aggregator goroutines to finish. After an input source (channel) is closed, the for loop inside the respective goroutine will end, and it will finish its job.
After all input sources are closed, we're done with aggregating, and we now can close the destination channel. This is a crucial step, and if we don't close the channel we have created, the Go runtime will complain with a fatal error:
\n1fatal error: all goroutines are asleep - deadlock!\nPutting all of our functions together, we're ready to run our code:
\n 1const (\n 2 processorCount = 10\n 3)\n 4\n 5func main() {\n 6 splitSources := Splitter(stringGenerator("test", 100), processorCount)\n 7 procSources := make([]<-chan string, 0, processorCount)\n 8\n 9 for i, src := range splitSources {\n10 procSources = append(procSources, stringProcessor(i, src))\n11 }\n12\n13 for out := range Aggregator(procSources) {\n14 fmt.Println(out)\n15 }\n16}\nThank you for reading!
\n", "url": "https://sazak.io/articles/cloud-native-patterns-illustrated-fan-in-and-fan-out-2022-10-30", "title": "Cloud Native Patterns Illustrated: Fan-in and Fan-out", "summary": "Fan-in and fan-out patterns explained with fancy animations.", "image": "https://sazak.io/assets/articles/cloud-native-patterns-illustrated-fan-in-and-fan-out-2022-10-30.jpeg", "date_modified": "2022-10-30T00:00:00.000Z", "author": { "name": "Ozan Sazak", "url": "https://sazak.io" } }, { "id": "https://sazak.io/articles/gls-file-manager-on-terminal-with-go-2022-07-19", "content_html": "This week my last term in college is finished and my brain started to come up with creative ides, so I started working on one of the fancy ones: developing a file manager with GUI that I can use in terminal, with Go.
\nBefore diving into my design and development process, here’s a little backstory: I use a 2019 model MacBook Pro storage as my development machine. Since it only has 256 GB storage, I frequently run out of space and try to find which files take up most of the space and which of them to delete. Here I have a few options and the one I do is basically using du command to find out the biggest files on terminal:
1$ du -shc ~/* | sort -rh\nThis basically works out, but I find myself consistently bragging about du not having an interactive interface that also lists the filesystem in tree format with file/folder sizes.
And that’s where I said out loud: I should do this!
\nAfter deciding to develop a program for this idea, I thought of the main features this program should provide for my daily use:
\nfile command doestouch-like), move, copy, remove filesMany other features could be added, but these are the trivial ones :) I also decided to name my program gls, a mimicking “graphical ls”.
![\"Initial](\"/assets/articles/Screen%20Shot%202022-07-10%20at%2010.40.33.png\")
Before diving deep into coding the UI, we first need to build a virtual filesystem representation of the original folder structure with the path we’re currently interested in. This representation will make us able to show the real folder structure in the UI in tree view. It will also help us perform specific file- and folder-related operations, such as copying and removing, by using the virtual representation to get info of specific files and folders instead of using system calls to get the same information each time.
\nSpeaking of which, most of the operating systems hold the whole filesystem structure as a one-rooted tree data structure, where all folders and files are child nodes of the root node or another node with a higher-level parent node. This type of data structure is also called “directed acyclic graph”.
\n![\"An](\"/assets/articles/Screen%20Shot%202022-07-10%20at%2010.39.22.png\")
\nExtra information: in Linux and UNIX-like systems, every node in the filesystem tree is a file1. Even directories, drivers (files under
\n/dev), terminals (eg./dev/tty), etc. are files with special properties.
Coming back to our virtual filesystem representation, it will be sufficient to construct a basic tree data structure that you’ve probably coded from scratch in a DS course:
\n 1type Node struct {\n 2 Name string\n 3 Mode os.FileMode\n 4 Size int64\n 5 SizeOnDisk int64\n 6 IsDir bool\n 7 LastModification time.Time\n 8 Children []*Node\n 9 Parent *Node\n10}\n\nYou may be asking “Why
\nSizeis a signed integer while it’s used for the size of an entity which can’t be negative?”. Well, the answer isos.FileInfo’s size property is alsoint64and that’s also probably because of whylen()returns a signed integer, to prevent integer underflows in arithmetic calculations2. Also, storing the size inint64makes us able to properly store the size of files/folders up to 16777216 TB, so it’s actually not a downside to use signed integers here.
The reason that we have both Size and SizeOnDisk is that, Size shows the actual number of bytes a specific file consists of. But as the operating systems allocates a number of blocks on the storage device for each file, the real size on the disk is a multiple of block_size3 (Even if the file content is a single byte, the size of the storage area allocated for the file will be block_size bytes4). So we calculate SizeOnDisk by block_size * number_of_blocks.
Now it’s time to walk on the file tree. To do this, we need to use a tree traversal algorithm5, which are pretty straightforward. I will explain the traversal basics in the next subtitle, so you can pass it if it’s too basic for you.
\nThe most trivial tree traversal algorithm is depth-first search, where we walk down the tree from a node as deep as possible before hopping to its siblings. DFS can be used as pre-order, post-order and in-order. The only difference is when do we process the root (or current) node, other than that, what DFS does is iteratively applying the same traversal to the child subtrees of the specific node.
\nIn pre-order traversal, we visit the root node and then the children one by one (root-Left-Right for example if we have a binary tree). In post-order, we visit the children one by one and then the root node eventually (e.g L-R-root). Likewise in in-order traversal we visit the root node in between visiting the child nodes (e.g L-root-R).
\n![\"Node](\"/assets/articles/IMG_B1947371A873-1.jpeg\")
For the sake of implementation simplicity, I decided to use pre-order traversal to walk in the file tree and construct the *Node tree on our side.
1func Walk(path string) (*Node, error) {\n 2 f, err := os.Lstat(path)\n 3 if err != nil {\n 4 log.Warningf("%s: %v", path, err)\n 5 return nil, nil\n 6 }\n 7 st, ok := f.Sys().(*syscall.Stat_t)\n 8 if !ok {\n 9 return nil, fmt.Errorf("could not cast %T to syscall.Stat_t", f.Sys())\n10 }\n11 root := &Node{\n12 Name: f.Name(),\n13 Mode: f.Mode(),\n14 Size: st.Size,\n15 SizeOnDisk: st.Blocks * 512,\n16 IsDir: f.IsDir(),\n17 LastModification: f.ModTime(),\n18 }\n19 if root.IsDir {\n20 names, err := readDirNames(path)\n21 if err != nil {\n22 log.Warningf("%s: %v", path, err)\n23 return root, nil\n24 }\n25 for _, name := range names {\n26 child, err := Walk(path + "/" + name)\n27 if err != nil {\n28 return nil, err\n29 }\n30 child.Parent = root\n31 root.Size += child.Size\n32 root.SizeOnDisk += child.SizeOnDisk\n33 root.Children = append(root.Children, child)\n34 }\n35 }\n36 return root, nil\n37}\nWe call os.Lstat for each node in the file tree to get the file/directory information. The os.Lstat function uses the lstat6 system call internally, which returns struct stat structure. Then, os.Lstat uses struct stat to return os.FileInfo, which is an easily usable variant of struct stat.
After getting the file information, we construct the root node of type *Node, and then call Walk for each of its children if root is a directory, and lastly update the child nodes’ root, the root’s size and so on. Children of the root are found by the readDirNames function, which basically reads the directories under the specified path and sorts them by their name7. The SizeOnDisk calculation is done by using Stat_t.Blocks. The Stat_t structure is the Go equivalent of Linux kernel’s struct stat, and Stat_t.Blocks is the number of 512-byte blocks that the file uses.
\nPlease note that this code we’ve written will only work in Linux and UNIX-like systems!
\n
We can now call Walk and test it on a real directory:
1import (\n 2 "flag"\n 3 "fmt"\n 4 "os"\n 5 "sort"\n 6 "strings"\n 7 "syscall" \n 8 "time"\n 9\n 10 "github.com/ozansz/gls/log"\n 11)\n 12\n 13type Node struct {\n 14 Name string\n 15 Mode os.FileMode\n 16 Size int64\n 17 SizeOnDisk int64\n 18 IsDir bool\n 19 LastModification time.Time\n 20 Children []*Node\n 21 Parent *Node\n 22}\n 23\n 24func (n *Node) Print() {\n 25 n.printWithLevel(0)\n 26}\n 27\n 28func (n *Node) printWithLevel(level int) {\n 29 fmt.Printf("%s%s [%s] [%d]\\n", strings.Repeat(" ", level),\n 30 n.Name, n.Mode.String(), n.SizeOnDisk)\n 31 for _, child := range n.Children {\n 32 child.printWithLevel(level + 1)\n 33 }\n 34}\n 35\n 36var (\n 37 path = flag.String("path", "", "path to walk")\n 38)\n 39\n 40func main() {\n 41 flag.Parse()\n 42 if *path == "" {\n 43 flag.Usage()\n 44 return\n 45 }\n 46\n 47}\n 48\n 49func Walk(path string) (*Node, error) {\n 50 f, err := os.Lstat(path)\n 51 if err != nil {\n 52 log.Warningf("%s: %v", path, err)\n 53 return nil, nil\n 54 }\n 55 st, ok := f.Sys().(*syscall.Stat_t)\n 56 if !ok {\n 57 return nil, fmt.Errorf("could not cast %T to syscall.Stat_t", f.Sys())\n 58 }\n 59 root := &Node{\n 60 Name: f.Name(),\n 61 Mode: f.Mode(),\n 62 Size: st.Size,\n 63 SizeOnDisk: st.Blocks * 512,\n 64 IsDir: f.IsDir(),\n 65 LastModification: f.ModTime(),\n 66 }\n 67 if root.IsDir {\n 68 names, err := readDirNames(path)\n 69 if err != nil {\n 70 log.Warningf("%s: %v", path, err)\n 71 return root, nil\n 72 }\n 73 for _, name := range names {\n 74 child, err := Walk(path + "/" + name)\n 75 if err != nil {\n 76 return nil, err\n 77 }\n 78 child.Parent = root\n 79 root.Size += child.Size\n 80 root.SizeOnDisk += child.SizeOnDisk\n 81 root.Children = append(root.Children, child)\n 82 }\n 83 }\n 84 return root, nil\n 85}\n 86\n 87// Taken directly from the Go stdlib\n 88func readDirNames(dirname string) ([]string, error) {\n 89 f, err := os.Open(dirname)\n 90 if err != nil {\n 91 return nil, err\n 92 }\n 93 names, err := f.Readdirnames(-1)\n 94 f.Close()\n 95 if err != nil {\n 96 return nil, err\n 97 }\n 98 sort.Strings(names)\n 99 return names, nil\n100}\nWhen we save this code as main.go and run it by specifying a path,
1$ go run main.go --path ~/root-folder\n…and assume that the file tree is something like that:
\n![\"Example](\"/assets/articles/Screen%20Shot%202022-07-10%20at%2010.38.48.png\")
…we will see an output like that:
\n1root-folder [drwxr-xr-x] [10010624]\n2 subfolder [drwxr-xr-x] [10002432]\n3 file_3.mp4 [-rw-r--r--] [9998336]\n4 file_1.c [-rw-rw-r] [8192]\n5 file_2.sh [-rw-r---] [4096]\nI talked about continuous profiling with Go, and how to use the pprof package to profile Go programs at Ankara Gophers's 8th meetup.
\nAnkara Gophers is a local Go community in Ankara, Turkey. They organize meetups to bring Go developers together, and to share knowledge and experiences since 2022.
\n\n\n", "url": "https://sazak.io/talks/continuous-profiling-with-go-2022-06-17", "title": "Continuous Profiling with Go", "summary": "Talked about continuous profiling with Go at Ankara Gophers meetup. Ankara Gophers is a local Go community in Ankara, Turkey.", "image": "https://sazak.io/assets/talks/continuous-profiling-with-go-2022-06-17.jpeg", "date_modified": "2022-06-17T00:00:00.000Z", "author": { "name": "Ozan Sazak", "url": "https://sazak.io" } } ] }Unfortunately the event recording or slides are not available.
\n