prefect

Flow code for the Prefect server on the LAN cluster (prefect.lan-k8s.kutzlab.io).

How this works

Flow code is pulled from this repo at run time, not baked into an image:

1. You push to main.
2. A flow run is scheduled; the worker in the prefect namespace creates a Kubernetes Job.
3. Inside that pod, the pull step in prefect.yaml clones this repo.
4. Your flow runs.

So a code change is a git push — no image rebuild, no redeploy. You only re-run prefect deploy when you change deployment config (the entrypoint, the schedule, job variables), not when you change flow code.

Dependencies

Runtime deps are baked into a custom image, not pip-installed per run. Dockerfile is prefecthq/prefect:3.8.3-python3.11 plus requirements.txt installed with uv; .forgejo/workflows/build.yaml builds it and pushes to harbor.rss.kutzlab.io/library/prefect:3.8.3 on any push to main that touches Dockerfile, requirements.txt, or the workflow itself.

That tag moves, and the deployment sets image_pull_policy: Always, so adding a dependency is also just a git push — wait for the build, then the next flow run picks it up. No pf deploy.

prefect itself must not go in requirements.txt; it's in the base image and the Dockerfile constrains it to 3.8.3 so a transitive dep can't drag it forward. Bumping the Prefect version means changing the tag in three places: FROM and the constraint in Dockerfile, the tags in the workflow, and prefect-version / job_variables.image in prefect.yaml.

The Harbor library/ project allows anonymous pulls, so the cluster needs no imagePullSecret. HARBOR_USERNAME / HARBOR_TOKEN for the push side are inherited org-wide from Forgejo — nothing to set per-repo.

Prerequisites

prefect isn't installed system-wide; use uv and pin the client to the same version as the server so you don't hit client/server mismatch:

export PREFECT_API_URL=https://prefect.lan-k8s.kutzlab.io/api
alias pf="uvx --from 'prefect==3.8.3' prefect"

The server has no authentication yet, so there's no API key to set. That will change — when it does, add PREFECT_API_AUTH_STRING.

Deploying

Push first. The pull step clones from the remote, so anything not pushed does not exist as far as a flow run is concerned. Deploying local-only code gives you a run that fails on a missing entrypoint, which is a confusing way to learn this.

git push
pf deploy --all

Then trigger it:

pf deployment run 'hello/hello'

Watch it at https://prefect.lan-k8s.kutzlab.io — or from the cluster side:

kubectl -n prefect get jobs,pods -w

The first run is slow (image pull on a cold node). Later runs start in seconds.

Adding a flow

1. Drop a new module under flows/.
2. Add an entry to deployments: in prefect.yaml pointing at flows/<module>.py:<flow_function>.
3. git push && pf deploy --all.

Subsequent edits to that flow's body need only a push.

Repo is public

kutzlab/prefect on Forgejo is public, which is why the pull step needs no credentials. Don't commit anything sensitive. If you make it private, add an access_token to the git_clone step in prefect.yaml backed by a Prefect Secret block — there's a commented example in place.