Skip to main content

MLflow Dashboard

The lora-finetune skill's own "Watching Progress" step already tunnels MLflow over SSH for the length of one training run — that's enough for "let me watch this run finish." The mlflow-dashboard-setup skill is for a dashboard that outlives a single SSH session: reopen it later without retraining, or reach it straight from a browser with no SSH client at all.

Set up a persistent MLflow dashboard for the training instance
Prerequisites

training-machine-provisioning must already be applied — SSH access and mlflow inside ~/training/.venv both need to already exist. Works even before any lora-finetune run: the SQLite tracking store is created on first launch if it doesn't exist yet.

Two Ways to Reach It​

MLflow's UI ships with no authentication — same tradeoff already true of this repo's TensorBoard setup. Whoever can reach the port can read every run's params and metrics.

  • SSH tunnel (default). mlflow ui binds to localhost only on the instance; only whoever holds the .pem key can ever reach it. Nothing to open on AWS.
  • Public IP. Opens TCP 5000 on the JoyAI VL Interaction Training security group and binds the UI to 0.0.0.0, so http://<instance-ip>:5000 works straight from any browser — no SSH client needed, but also no login screen. Only worth it if you specifically want to share a link, and only when there's no domain available — see below if there is.

Either way, the dashboard reads from the same sqlite:////home/ubuntu/training/mlflow.db store that every lora-finetune run (report_to: mlflow) writes to — every fine-tune ever run on the instance shows up as its own experiment/run, comparable side by side.

Confirmed the hard way

Two non-obvious things had to be worked out against a real deploy: mlflow's --allowed-hosts flag needs the port included (<instance-ip>:5000, not just the bare IP) or every request 403s as a suspected DNS-rebinding attempt, and the process needs to be launched with setsid, fully detached from the SSH session, or it can die the moment the connection that started it closes. Both are handled by the skill already.

Result​

A dashboard, reachable either as http://localhost:5000 (over a tunnel you keep open) or http://<instance-ip>:5000 (public path), accumulating every training run on that instance — filterable by experiment name — until the instance is stopped or the process is killed.

Adding HTTPS​

If a real domain's DNS already points at the training instance, set that env var and run the follow-up skill:

export MLFLOW_UI_DOMAIN=mlflow.example.com
Set up HTTPS for the MLflow dashboard domain

This invokes the mlflow-https-setup skill. It installs nginx + certbot, issues a real Let's Encrypt certificate, and — unlike the "Public IP" path above — rebinds mlflow ui to localhost and closes the direct port-5000 rule, so the trusted, encrypted nginx endpoint on 443 becomes the only way in rather than an addition to the unencrypted one. Prefer this over the plain "Public IP" path whenever a domain is available.

It also enables gzip compression for MLflow's frontend assets — without it, the ~4 MB JavaScript bundle ships uncompressed on every page load, which is the difference between a page that renders in a few seconds and one that takes nearly 30. If a deployed dashboard still feels slow to load, that's the first thing to check.

Confirmed the hard way

Non-interactive certbot renew (exactly how the real auto-renewal timer runs it) intentionally sleeps for a random delay — often several minutes — before actually renewing, to avoid every server on earth hitting Let's Encrypt at the same instant. Checking it manually can look exactly like a hung process for minutes at a stretch; the skill uses --no-random-sleep-on-renew for manual verification specifically to avoid that false alarm, while leaving the real automatic renewal timer's jitter untouched.