{"id":1795,"date":"2026-08-19T13:08:31","date_gmt":"2026-08-19T13:08:31","guid":{"rendered":"https:\/\/blog.openzeka.com\/en\/?p=1795"},"modified":"2026-08-19T13:12:33","modified_gmt":"2026-08-19T13:12:33","slug":"local-gpt-oss-120b-serving-on-dgx-spark-with-sparkrun","status":"publish","type":"post","link":"https:\/\/blog.openzeka.com\/en\/local-gpt-oss-120b-serving-on-dgx-spark-with-sparkrun\/","title":{"rendered":"Local GPT-OSS 120B Serving on DGX Spark with sparkrun"},"content":{"rendered":"

In this tutorial, you will run a large language model on a single or multiple DGX Sparks using sparkrun.<\/p>\n

Spark can be used directly as a computer by connecting a monitor and keyboard, or as a server accessed remotely from another computer. In this tutorial, we will connect to the Spark remotely, install the necessary software, and run the model.<\/p>\n

We will use GPT-OSS 120B as the language model (a Mixture-of-Experts model with 117 billion parameters, 5.1 billion active parameters, mxfp4 quantization), vLLM as the inference engine, and sparkrun as the management tool. vLLM will load the model’s trained weights into GPU memory and run them, serving an API that accepts external requests. sparkrun will manage Docker containers and model distribution across Sparks via the command line. This automates image synchronization, model transfer, and cluster configuration. Both will run on the Main Spark, inside Docker containers.<\/p>\n

This tutorial consists of three parts:<\/p>\n

    \n
  • Setup:<\/strong> sparkrun installation, Docker image build, cluster configuration, and recipe creation<\/li>\n
  • Running with a Single Spark:<\/strong> Starting, monitoring, and testing the model on a single Spark<\/li>\n
  • Running with Two Sparks:<\/strong> Running the model split across multiple Sparks (tensor parallelism) and comparing performance<\/li>\n<\/ul>\n

    If you have a single Spark, review the “Setup” and “Running with a Single Spark” sections. If you are using multiple Sparks, you can proceed directly to the “Running with Two Sparks” section after “Setup”.<\/p>\n

    If you are using multiple Sparks, note that throughout the tutorial the primary device will be referred to as the Main Spark, and the other devices as Worker Sparks.<\/p>\n

    \n

    Setup<\/h2>\n

    1. Connecting to the Spark<\/h3>\n

    If you are connecting to the Spark remotely for the first time, you need to find its IP address. Connect a monitor and keyboard to the Spark, log in, and run the following command from the terminal:<\/p>\n<\/div>