Fine-Tuning

The lora-finetune skill prepares training data and runs the actual LoRA fine-tune of JoyAI-VL-Interaction-Preview into a custom proactive speak/silent specialist for a target event you describe. The pipeline is scenario-agnostic — nothing about the target event is baked in, it's defined by editing one prompt (see below).

Prepare training data and fine-tune for <target event>, using <local video path>

Prerequisites

training-machine-provisioning must already be applied — SSH access, the ~/training/.venv environment, and the cloned base checkpoint all need to already exist.

Preparing Training Data

Only two things here have to come from you: a local video file (or files), and a one-line description of the target event to specialize for. Given those, the agent handles the rest directly — no commands to type yourself: copying this skill's scripts/ onto the instance, uploading your video to ~/training/data/videos/, and writing your event description to ~/training/config/target_event.txt. That file is reused verbatim at both training and inference time — see the skill's references/dataset-and-framework.md for why the two have to match.

The one thing that stays manual

manifest.jsonl — when the target event actually happens in your footage — isn't something that can be inferred automatically. Auto-generating it would mean an unfine-tuned model labeling its own training data, which defeats the point of fine-tuning it. You write this one by hand, one JSON line per uploaded video, marking the time windows where the target event occurs:

{"video": "session-04.mp4", "fps_sample": 1, "events": [
  {"start_sec": 128.0, "end_sec": 131.5, "decision": "response", "text": "Whoa, that jump nearly missed the rail!"}
]}

Every second not covered by an events window is auto-labeled silence — don't hand-annotate negatives. Full schema, class-balance guidance, and how much footage a first real run needs are in the skill's references/dataset-and-framework.md.

Once the manifest exists, the agent builds the labeled, frame-extracted example set (prepare_dataset.py) that the trainer reads from below — class-balance ratio and other tuning knobs are documented in the skill's references/dataset-and-framework.md.

Running Fine-Tuning

No separate invocation needed — this continues automatically from the same request at the top of this page. The agent first smoke-tests the pipeline on a tiny slice (checking loss is finite and trending down, memory has headroom, and a checkpoint saves cleanly), then kicks off the full run in the background once that looks healthy. Nothing to run yourself here either.

note

Silence dominates raw 1fps-labeled footage. If a smoke-test run's loss drops to near-zero suspiciously fast, that usually means the model has learned to always predict silence rather than the target event — worth flagging to the agent so it checks the silence ratio before trusting the run.

Watching Progress in MLflow

The agent starts a live MLflow dashboard tracking loss, learning rate, and epoch as training progresses, and gives you a link to open in your browser — how that link reaches you depends on the setup:

• Most commonly, the agent tunnels the instance's MLflow UI port over SSH and hands you a http://localhost:5000 link.
• Alternatively, if the training security group's port 5000 has been opened, the agent can bind the MLflow UI directly and give you a plain http://<instance-ip>:5000 link instead — worth asking the agent to close that port again once you're done, since MLflow's UI has no built-in login and that link is reachable by anyone who has it.

Either way, just open the link you're given. Because every run under the instance's MLflow tracking store is recorded to the same backing store, this dashboard accumulates every fine-tune you've ever run there — not just the current one — so you can compare hyperparameters and loss curves across runs (e.g. a smoke test alongside a full run, or two different target events) side by side, with no need to restart it.

This link stops working once the SSH session that started it closes. For a dashboard that stays up on its own — reopen it later, or share a link with no SSH client needed — see MLflow Dashboard.

The result is a LoRA adapter directory (tens of MB), not a full model copy. Ask the agent to load it via peft's PeftModel.from_pretrained for evaluation, or merge it into a plain checkpoint for inference-machine-provisioning to serve.

Registering Models in MLflow

Beyond tracking runs, the base checkpoint and every fine-tuned adapter can be registered as versioned MLflow Models — visible in the MLflow UI's Models tab, not just a run's metrics:

Register the base model and the fine-tuned adapter in MLflow

The base checkpoint registers once per instance (a real, full ~17GB copy into MLflow's own storage — the registry has no lightweight reference mode that still shows up in the Models tab, so this is a deliberate tradeoff, not a bug) and is slow — expect on the order of an hour, not minutes. Each fine-tuned adapter registers under its own target event's name, versioned per run, and is fast — just tens of MB. Either one can be pulled back later via MLflow (mlflow.transformers.load_model("models:/<name>/<version>")) as the starting point for a further fine-tune on new data, or for evaluation.