How I built my own Go package index - go.sazak.io

One of the things I like about Go is that when you want to use someone else's code in your project, you can just import it by its GitHub repo URL.

Literally, you can just import the package by writing:

1import "github.com/google/go-cmp/cmp"

And that's it. Although these URLs don't have to be GitHub URLs. Sometimes you see import paths like go.uber.org/atomic, or golang.org/x/crypto/ssh.

These are also valid import paths and they are called "vanity import paths". They are called "vanity" because they are custom domain names, which redirect to the actual repository.

How does it work?

The work behind the scenes is pretty simple: When you run go get, or go install, Go tooling sends an HTTP request to this URL to get the package info, then downloads the source code under the vendor directory. The package info is provided in meta tags in the HTML response, which are:

go-import: Where to find the repository. It has three parts: the import path, the VCS type (git, hg, etc.), and the repository URL.

go-source: Optional information for pkg.go.dev to link to the source code.

The idea is, by simply returning these meta tags in the HTML response, you can make your own Go package index.

But why?

Why not? I like building stuff and putting them in public repos. I also use my own packages/tools later in other projects. So, I thought it would be cool to step it up a bit and have my own domain for my Go packages.

It's better to use this:

1import "go.sazak.io/my-package"

Instead of this:

1import "github.com/ozansz/my-package"

That's how go.sazak.io was born. It's a simple Go package index for my own public Go packages. It's basically a static site built by using a JSON file that contains the package info, which is also generated by a simple Go tool.

Designing the system

When you think of designing an index which lists a number of packages, which link to their respective GitHub repositories, you might think of using a database, a backend server, and a frontend client.

We tend to overcomplicate things as developers, but in my opinion it's the best solution to keep things as simple as possible while preserving the functionality and developer experience.

"Simple is better than complex." - Zen of Python

The whole systems consists of two parts: a Go tool called sync-github-repos that collects and compiles the data, and the static site that uses the JSON file as the data source to build a static site.

The Backend

The tool uses GitHub API to list my public repositories, then filters the ones that have a go.mod file which content starts with module go.sazak.io/. The data is then stored inside a Repo Go struct to later be marshaled into a JSON file:

 1type Repo struct {
 2	Owner        string            `json:"owner"`
 3	Name         string            `json:"name"`
 4	Stars        uint              `json:"stars"`
 5	Description  string            `json:"desc"`
 6	GoPackage    string            `json:"go_package"`
 7	LatestTag    string            `json:"latest_tag"`
 8	AlphaRelease bool              `json:"alpha_release"`
 9	HasCLIApp    bool              `json:"has_cli_app"`
10	Packages     map[string]string `json:"packages"`
11	MasterBranch string            `json:"master_branch"`
12}

Then it checks if there are any public packages (containers) of the repository, are there any releases/tags, and if the latest tag is a pre-release. It also checks the name of the default branch, which is usually main or master.

The HasCLIApp field is a boolean that tells if the repository includes a CLI app in the cmd directory. This field is set if the repo has a cli tag.

Also the AlphaRelease field is a boolean that tells if the latest tag is a pre-release or if the tag is below version v1.0.0.

The JSON file is then used as the data source for the static site. The site is built with Next.js¹, Shadcn UI² and deployed to Vercel.

The UI

As I explained before, the website only needs to serve two meta tags in the HTML response. For a basic requirement like this I could just use plain HTML, but I thought it would be better to have a clean and informative UI for the human visitors.

That's why I decided to use Shadcn UI and design a one-page package listing site. I used the Card component to display the package info for all packages. The basic information is same for all packages, for example the package name, latest tag badge, instructions to clone the repository, etc.

Cards also contain different sections, based on what package provides. If it's a CLI app, it shows the instructions to install the app. If it has packages (containers), it also shows the instructions to pull the containers using docker pull.

After putting it all together, the site looks like this:

Which, I think, is pretty cool for a simple Go package index, and also for a backend/platform engineer like me to build.

The Pipeline

To make it easier to deploy the website with the synced data, I created a GitHub Actions workflow pre-commit hook that runs the sync-github-repos tool to generate the JSON file and put it inside the UI project:

 1#!/bin/sh
 2
 3set -eo pipefail
 4
 5go run cmd/sync-github-repos/main.go --username ozansz --hostname go.sazak.io --out ui/src/repos.json --force-authenticated
 6if [ $? -ne 0 ]; then
 7    echo "Pre-commit hook failed (content generation script)"
 8    exit 1
 9fi
10
11git update-index --add ui/src/repos.json

After I push the changes, the Vercel deployment pipeline is triggered and the site is updated with the new data.

Conclusion

It's pretty simple -and cool- to have your own Go package index. It's a good way to keep track of your own packages publicly.

If you liked the idea, you can also build your own package index. My implementation for go.sazak.io is open source and available on GitHub. I tried to make it as simple as possible to use and extend.

Feel free to fork it and build your own package index!