Deploying LLMs in production?

Here's a hands-on guide:

Local experimentation only takes you so far. At some point, the model needs to leave your machine. It needs to be fine-tuned on proper compute, exported in the right format, and deployed behind an endpoint that can handle real requests.

This is the workflow we use at our company:

- RunPod for on-demand GPU infra

- Unsloth for efficient fine-tuning

- SGLang for model serving Here's how it works step by step: - Spin up a RunPod Pod with a GPU (RTX 4090 works great). Your laptop just becomes the UI.

- Open Jupyter inside the Pod. All training and deployment code runs directly on the GPU.

- Load gpt-oss-20B with Unsloth.

The optimizations kick in at import time, making a 20B model actually practical to work with.

- Attach LoRA adapters. Instead of updating all 20B parameters, you train a small set of weights while keeping the base model frozen. - Run supervised fine-tuning. Unsloth's training loop is optimized for large models. Training stays fast, memory stays low.

- Export the model. Save a merged 16-bit checkpoint that combines the base model and LoRA adapters into one artifact.

- Launch SGLang server. It loads your checkpoint and starts an OpenAI-compatible inference endpoint.

- Send requests using the standard OpenAI client. No custom tooling needed.

This setup takes gpt-oss-20B from fine-tuning to real inference, all running on an on-demand GPU compute. Everything above ran on RunPod. Fine-tuning, export, and deployment, all on the same infrastructure, and I worked with the team to put this together.

What I appreciate about it is that the infrastructure stays out of the way. You rent the GPU, do your work, and pay by the second. When you’re prototyping, you use a cheaper GPU.

When you’re ready to scale, the higher-end options are there. The flexibility to move between these without dealing with quotas or approvals makes iteration much faster.

Infrastructure should disappear into the background. RunPod gets close to that ideal.
Source Reference - @akshay_pachaar