In this tutorial, you will run a large language model locally on a DGX Spark and chat with it from another computer’s browser.

The Spark can be used as a standalone computer by connecting a monitor and keyboard, or as a remote server accessed from another computer. In this tutorial, we will connect to the Spark remotely and install the necessary software to run the model.

We will use Qwen3.6-35B-A3B-FP8 (a 35-billion parameter Mixture-of-Experts model, 3 billion parameters active, FP8 quantization) as the language model, vLLM as the inference engine, and Open WebUI as the web interface. vLLM will load the model’s trained weights into GPU memory and expose an API that accepts external requests. Open WebUI will connect to this API and display a chat interface in the browser. Both will run on the Spark, inside Docker containers.

Prerequisites: Finding the Spark’s IP Address

If you are connecting to the Spark remotely for the first time, you need to find its IP address. Connect a monitor and keyboard to the Spark, log in, and run the following command in the terminal:

The command outputs the IP address of the Spark’s default network interface:

Note this address; you will use it in place of throughout this tutorial. Alternatively, you can find the IP address by checking the NVIDIA Sync application.

1. Connecting to the Spark

Make sure your computer is connected to the same network as the Spark. Then, open a terminal on your computer and connect to the Spark via SSH:

On the first connection, you will see a fingerprint warning. Type yes and press Enter. Then, when prompted for a password, enter the Spark’s password:

Once connected, the Spark will start accepting commands sent from this terminal.

2. Clearing the Cache

Before starting vLLM, flush the filesystem cache:

The main reason for doing this is the DGX Spark’s unified memory architecture: the operating system caches model files read from disk in RAM. vLLM loads the model weights from here into GPU memory. After loading is complete, the cached data remains in RAM even though it will not be used again. On systems with separate memory, this is not significant — since inference runs in GPU memory, RAM utilization does not affect performance. On the Spark, however, the CPU and GPU share the same RAM, so the cache reduces the memory available to the GPU. This command frees the cache, providing maximum memory for vLLM.

When prompted for a password, enter the Spark’s password. The command produces no output and completes silently:

3. Setting Up the vLLM Server

Pull the vLLM Docker image:

During the download, layers will be pulled sequentially and you will see progress percentages on the screen. If the image is already on disk, Docker skips the download. Once the download is complete, you will see the following output:

This image contains everything needed to run vLLM — PyTorch, CUDA kernels, Python libraries…

Now, let’s start the vLLM server:

This command starts the downloaded image as a container. If the model is not already installed on the Spark, it will be automatically downloaded from Hugging Face. As the download progresses, you will see progress bars, and once complete, the files will be placed in the ~/.cache/huggingface/hub/ directory. If the model has been previously downloaded on the Spark (if you have followed this tutorial before), vLLM skips the download step and loads the downloaded weights directly into GPU memory.

When the command finishes, you will see a container ID; this indicates that the server has started in the background and the model is now being served on port 8000:

If a container with the same name (from your previous attempts) already exists, the command will return the following error:

In this case, check the container list — vllm-qwen36-35b-fp8 should be listed:

If the status column shows “Up”, the container is already running and ready to use — you can proceed to the next step:

If the status column shows “Exited”, the container has stopped:

You can restart the stopped container and proceed to the next step:

4. Monitoring the vLLM Startup Process

After the model download is complete, it may take a few minutes for vLLM to become ready for serving. During this process, vLLM loads the model weights into GPU memory, compiles GPU kernels, and allocates memory for inference. To monitor this process:

You will see the following in the logs:

Once you see the Application startup complete. line, the model server is ready. Press Ctrl+C to stop watching the logs. The server will continue running in the background.

5. Testing the vLLM Server

First, let’s verify that the server is running:

If you don’t receive any errors, the server is healthy:

Now test the model:

The simplified response is shown below. In the content field (the model’s response), you will see the number 204. This confirms that the model performed the multiplication correctly. Additionally, the response includes a reasoning field. This field contains the model’s thought process. As you can see, the model performed the calculation using two separate methods and verified the solution before arriving at the answer:

vLLM is now running and we can ask questions to Qwen3.6 and receive responses. However, sending each question as a terminal command like the one above is not practical. We need a web interface to chat from the browser.

6. Setting Up Open WebUI

Open WebUI is an interface that connects to vLLM and can be used from your browser. You can chat with your language model and upload images to ask questions about them. Responses start appearing on the screen as they are being generated — you don’t have to wait for the entire response to finish before you start reading. For each chat, you can adjust settings such as temperature, top_p, and max_tokens. On first setup, it downloads a small embedding model, which enables you to upload documents and ask questions about them (RAG — Retrieval-Augmented Generation). Your chat history, settings, and account information are stored on the Spark’s disk; even if the device or container is shut down, your data is not lost, and you can continue from where you left off when you restart.

First, pull the Open WebUI Docker image on the Spark:

If the image is already on disk, Docker skips the download. Once the download is complete, you will see the following output:

Once the download is complete, start the container:

The command returns a container ID, indicating that the container has started:

As mentioned in the vLLM setup section, if a container with the same name (from your previous attempts) already exists, the command will return the following error:

In this case, check the container list — open-webui should be listed:

If the status column shows “Up”, the container is already running and ready to use — you can proceed to the next step:

If the status column shows “Exited”, the container has stopped:

You can restart the stopped container and proceed to the next step:

Your Open WebUI server will then be ready for use.

7. Monitoring the Open WebUI Startup Process

To watch the logs:

You will see output similar to the following in the logs:

After you see the Scheduler worker started line in the logs, wait a few seconds. Press Ctrl+C to exit log monitoring.

8. Testing Open WebUI

To verify that the server is ready and healthy, run the following command:

If you receive an HTTP 200 response, Open WebUI is ready:

9. Accessing from the Browser

On your computer’s browser, which is on the same network as the Spark, navigate to:

On first launch, a login screen will appear. Click the “Sign up” link and enter your name, email, and password. The first account created automatically becomes the administrator account. The purpose of creating an account is to store chat history and settings. This information is stored on the Spark.

The model is automatically detected by Open WebUI and listed in the model selection menu. You will see the Qwen/Qwen3.6-35B-A3B-FP8 model in the menu. Select the model. If you are running multiple models at the same time, you can instantly switch between models from here. Additionally, the chat history tab on the left allows you to create multiple chat sessions and switch between them.

Type a question in the message box and press Enter. As the model responds, a collapsible reasoning section will appear at the top, and the response will appear below.

Here are a few experiments we conducted to demonstrate what the model and web interface can do:

First, to test the model’s Turkish comprehension skills, we asked the model to list Turkey’s three largest cities and provide brief information about each.

Next, we asked the model to explain the benefits of artificial intelligence in the healthcare sector in three paragraphs.

After that, to test the model’s programming skills, we asked it to create a Python function that checks whether a text is a palindrome.

Using the file upload button in the message area, we sent the model an image of nature. The model described elements in the uploaded image such as flowers, leaves, water drops, and the background.

From the same file upload section, this time we gave the model the DGX Spark datasheet and asked it to analyze it.

Your local LLM setup on the Spark is ready! You can now chat with Qwen3.6 from all your devices on the same network.

10. Shutdown

When you are done, stop both containers:

Although the stopped containers cease running, their data and configuration are preserved on disk. To restart the containers from where they left off:

To completely remove the containers:

Even if you remove the containers, the Docker images and model files remain on disk. Therefore, you do not need to re-download to restart — simply run the docker run commands from steps 3 and 6 again. Chat history and account information are stored in the open-webui-data volume; even if the container is removed, this data is preserved on the Spark. If you want to delete this data as well, you can use the docker volume rm open-webui-data command.

In this tutorial, you will run a large language model on a single or multiple DGX Sparks using sparkrun.

Spark can be used directly as a computer by connecting a monitor and keyboard, or as a server accessed remotely from another computer. In this tutorial, we will connect to the Spark remotely, install the necessary software, and run the model.

We will use GPT-OSS 120B as the language model (a Mixture-of-Experts model with 117 billion parameters, 5.1 billion active parameters, mxfp4 quantization), vLLM as the inference engine, and sparkrun as the management tool. vLLM will load the model’s trained weights into GPU memory and run them, serving an API that accepts external requests. sparkrun will manage Docker containers and model distribution across Sparks via the command line. This automates image synchronization, model transfer, and cluster configuration. Both will run on the Main Spark, inside Docker containers.

This tutorial consists of three parts:

• Setup: sparkrun installation, Docker image build, cluster configuration, and recipe creation
• Running with a Single Spark: Starting, monitoring, and testing the model on a single Spark
• Running with Two Sparks: Running the model split across multiple Sparks (tensor parallelism) and comparing performance

If you have a single Spark, review the “Setup” and “Running with a Single Spark” sections. If you are using multiple Sparks, you can proceed directly to the “Running with Two Sparks” section after “Setup”.

If you are using multiple Sparks, note that throughout the tutorial the primary device will be referred to as the Main Spark, and the other devices as Worker Sparks.

Setup

1. Connecting to the Spark

If you are connecting to the Spark remotely for the first time, you need to find its IP address. Connect a monitor and keyboard to the Spark, log in, and run the following command from the terminal:

The command returns the IP address of the Spark’s default network interface:

Note this address; throughout the tutorial you will use it in place of . Alternatively, you can find the IP address by checking the NVIDIA Sync application.

If you are using multiple Sparks, repeat this step for every other device and note the addresses. You will use them in place of throughout the tutorial.

If you are using multiple Sparks, ensure the same username is used on all Sparks. sparkrun requires matching usernames when setting up passwordless SSH between devices. The default DGX OS username is nvidia. Check your username on each Spark:

If the username is not nvidia, create this user on all Sparks:

These commands: create the user, add them to the sudo group, set a password, and switch to the new user. Ensure the same username exists on all Sparks.

If you are using multiple Sparks, connect the devices to each other with QSFP cables. The cable can be plugged into any of the CX-7 ports on each Spark. To verify the connection, log in to each device with a monitor and keyboard and run the following command:

Those shown as (Up) are the ports where the cable is connected. Those shown as (Down) are unused ports. If no interface shows as (Up), check the QSFP cable and restart the Sparks.

Ensure your computer is connected to the same network as the Spark. Then, open a terminal on your computer and connect to the Spark via SSH:

On the first connection you will see a fingerprint warning. Type yes and press Enter. Then, when prompted for a password, enter the Spark’s password:

Once connected, the Spark will begin accepting commands sent from this terminal. Throughout the tutorial, you will enter all commands you encounter into this terminal on your computer.

2. Installing sparkrun

First, download the uv package manager:

Add uv to your PATH:

PATH updated. The ~/.local/bin directory (where uv and sparkrun are installed) is now available and has been added to the PATH for all future terminal sessions. Now install sparkrun:

Let’s verify the installation:

sparkrun v0.2.40 is installed and accessible.

3. Building the Docker Image

To get the best performance from GPT-OSS 120B, we will build a Docker image that contains CUTLASS MoE and FlashInfer attention kernels compiled specifically for the Spark’s GPU architecture. This image also includes the --mxfp4-backend, --mxfp4-layers flags and GPT-OSS’s tiktoken encoding files.

We will build this image using the build-and-copy.sh script from the eugr community project. First, clone the eugr repository:

The Dockerfile.mxfp4 file in the repository determines the version of the source code that build-and-copy.sh will compile. Before starting the build, we must ensure this version is up to date. Therefore, let’s query the latest commit hash of the mxfp4_v2 branch in christopherowen’s community project vLLM fork:

This hash returned by the command will direct build-and-copy.sh to the latest version of the vLLM source code we want. Copy or note it. Then open the Dockerfile.mxfp4 file with nano:

Once the file is open, press Ctrl+W to search within the file, type VLLM_SHA and press Enter. The cursor will land on a line similar to:

Delete the old hash value and paste the updated hash you noted above. Then press Ctrl+O to save the file, confirm with Enter, and exit nano with Ctrl+X.

Let’s verify the change we made to the Dockerfile.mxfp4 file:

The returned hash value should be the updated hash you copied in the previous step.

Fallback: If you experience issues with the latest commit, you can use the hash 04f641e537e80a67db464c6e65b928dbfab5d647 (January 28, 2026), which we have tested and confirmed. You can also check for newer commits in the commit history .

Now build the image:

The first build takes approximately 50 minutes; subsequent builds complete in about 5 minutes thanks to caching. When the build completes, you will see the following output:

Verify the image:

Verify the image’s vLLM version:

If you are using multiple Sparks, building the image only on the Main Spark is sufficient. sparkrun will automatically synchronize the image to the Workers over the CX-7 network on first launch. No additional step is needed.

4. Setup Wizard

The sparkrun setup wizard configures all the infrastructure needed to run the Spark with a single interactive command:

Six phases run sequentially, and the phases are interactive. In the first phase, enter in the ‘Enter host IPs/hostnames’ field and press Enter. For the subsequent prompts, you can simply press Enter (confirming the default options) to continue:

If you are using multiple Sparks, enter all the Sparks’ IP addresses separated by commas in the ‘Enter host IPs/hostnames’ field .

The second phase sets up passwordless SSH. Type Y and press Enter:

If you are using multiple Sparks, the wizard connects to the Worker Sparks via SSH during this phase and asks for the password. Enter the password — this step is done once; after the wizard distributes the passwordless SSH keys, no password will be asked for in subsequent connections.

The third phase configures the CX-7 high-speed network interfaces. It is automatically skipped if a single Spark is used. Type Y and press Enter:

If you are using multiple Sparks, the wizard automatically detects CX-7 interfaces, selects non-overlapping subnets, assigns a static IP to each Spark, sets MTU 9000 (jumbo frame), and writes the /etc/netplan/40-cx7.yaml file and applies it with netplan apply. This operation requires root privileges — enter the sudo password.

After CX-7 configuration, the wizard automatically refreshes the SSH network with the new IPs. This phase requires no additional input.

The fourth phase verifies that the user is in the docker group. Type Y and press Enter:

The fifth phase installs restricted sudoers rules. The installed rules are for passwordlessly fixing HuggingFace cache ownership and passwordlessly clearing the Linux page cache. Broad sudo privileges are not granted. Type Y and press Enter:

The sixth phase installs earlyoom OOM protection. earlyoom terminates inference processes instead of locking up the system when memory limits are approached in the DGX Spark’s unified memory architecture. Type Y and press Enter:

5. Creating the Recipe

A recipe is a YAML file that defines how sparkrun runs the model. The model, Docker image, vLLM flags, and memory settings are consolidated in a single file. This recipe can be used for both a single Spark (TP=1) and multiple Sparks (TP>1) — the only difference is that the --tp value in the launch command is adjusted according to the number of Sparks.

Verify the file:

Key fields in the recipe and the reasons they were selected:

• container
  • Value: vllm-node-mxfp4
  • Why: Image built in step 3 — includes the CUTLASS/FlashInfer forks
• tensor_parallel
  • Value: 1 (default)
  • Why: Default for a single Spark; overridden with the --tp value when using multiple Sparks
• gpu_memory_utilization
  • Value: 0.7
  • Why: Safe value for DGX Spark UMA
• --distributed-executor-backend ray
  • Value: Ray
  • Why: sparkrun automatically detects Ray and sets up the Ray cluster. For TP=1, Ray runs on a single node.
• --mxfp4-backend CUTLASS
  • Value: CUTLASS
  • Why: Custom CUTLASS MXFP4 MoE GEMM kernel
• --mxfp4-layers moe,qkv,o,lm_head
  • Value: Full
  • Why: All layers are quantized to FP4
• --attention-backend FLASHINFER
  • Value: FlashInfer
  • Why: Custom FlashInfer kernel supporting GPT-OSS attention architecture
• --kv-cache-dtype fp8
  • Value: FP8
  • Why: Reduced memory usage for the KV cache
• --load-format fastsafetensors
  • Value: Fast loader
  • Why: ~41-second model loading time
• --reasoning-parser openai_gptoss
  • Value: Harmony
  • Why: Separates the reasoning process into the reasoning field
• --tool-call-parser openai
  • Value: Tool usage
  • Why: Required for --enable-auto-tool-choice

Important:
The --rootful flag is not specified in the recipe; it is passed in the launch command.
FlashInfer compiles CUTLASS attention kernels at runtime, and this compilation requires root privileges.
The launch commands are shown in the “Launch the Model” step of the relevant section.

6. Pre-Launch Checks

Verify that `sparkrun` correctly parses the recipe and that the memory budget is appropriate:

sparkrun correctly parsed the recipe, selected vllm-ray, and gave the ‘DGX Spark fit: YES’ confirmation.

Now preview the launch plan:

Dry run successful. The recipe is valid, the serve command includes the flags we added, and it is ready to launch. If the model is not installed on the Spark, sparkrun will automatically download it from Hugging Face on first launch. The GPT-OSS 120B model is approximately 183 GB; download time depends on your internet speed. The dry run does not trigger this download — downloading only happens on the actual launch.

If you are using multiple Sparks, set the --tp value according to the number of Sparks for the TP dry-run (for example, --tp 2 for two Sparks). You will see --tensor-parallel-size 2 in the serve command, and the mode will appear as cluster (2 nodes) instead of solo:

Running with a Single Spark

1. Clearing the Cache

Before starting vLLM, clear the filesystem cache:

The main reason for doing this is the DGX Spark’s unified memory architecture: The operating system caches the model files it reads from disk in RAM. vLLM loads the model weights from here into GPU memory. After loading is complete, the data remaining in the cache is not used again but is not cleaned up immediately. In systems with separate memory, this is not significant. Since inference runs in GPU memory, RAM fullness does not affect performance. On the Spark, however, since the CPU and GPU share the same RAM, the cache reduces the space available to the GPU. The command clears this cache, providing maximum memory for vLLM.

When prompted for a password, enter the Spark’s password. The command produces no output; it completes silently:

2. Launching the Model

Now let’s launch the model:

The --rootful flag runs the container with root privileges. FlashInfer compiles CUTLASS attention kernels at runtime and requires root privileges for this compilation. The --no-follow flag causes sparkrun to return to the command line after launching the containers.

sparkrun successfully completed all 6 steps. Image building was skipped, and the image we built during setup was used. Since the model files were already on disk, distribution completed quickly. All flags were correctly resolved within the serve command.

3. Monitoring the Startup Process

After the model download completes, it may take a few minutes for vLLM to become ready to serve. During this process, vLLM loads the model weights into GPU memory, compiles GPU kernels, and allocates memory for inference.

To monitor the vLLM logs:

You will see the following in the logs:

If you see the Application startup complete. line, the model server is ready. Press Ctrl+C to stop watching the logs. The server will continue running in the background.

4. Testing the Model

First, verify that the server is running:

An ‘HTTP 200’ response indicates the server is healthy. Now check the sparkrun container status:

List the registered models:

The model openai/gpt-oss-120b is registered.

Now test the model with the prompt “What is 12*17?”:

A simplified version of the response is below. In the content field (the model’s answer) you will see 204. From this, you can tell the model performed the multiplication correctly. Additionally, the response has a reasoning field. This field contains the model’s reasoning process. GPT-OSS’s Harmony format is working: the reasoning process and the final answer are presented in separate fields:

vLLM is now running and serving on the Spark’s port 8000. We can ask GPT-OSS 120B questions and get answers.

5. Shutdown

When you are done, stop the model:

This command stops the container and frees up memory. However, the container, the Docker image, and the model files remain on disk. This means you don’t need to re-download to start again. Simply re-run the sparkrun run command from step 2.

Running with Two Sparks

1. Clearing the Cache

Before starting vLLM, clear the filesystem cache on both Sparks. The main reason for doing this is the DGX Spark’s unified memory architecture (UMA): The operating system caches the model files it reads from disk in RAM. vLLM loads the model weights from here into GPU memory. After loading is complete, the data remaining in the cache is not used again but is not cleaned up immediately. In systems with separate memory, this is not significant. Since inference runs in GPU memory, RAM fullness does not affect performance. On the Spark, however, since the CPU and GPU share the same RAM, the cache reduces the space available to the GPU. The command clears this cache, providing maximum memory for vLLM.

Clear the cache on the Main Spark:

Apply the same cleanup on the Worker Spark via SSH from the Main Spark:

2. Launching the Model

Now, let’s launch the model on two Sparks:

The --tp 2 flag tells sparkrun to run the model split (tensor parallel) across two Sparks. The --rootful flag runs the container with root privileges — FlashInfer compiles CUTLASS attention kernels at runtime and requires root privileges for this compilation. The --no-follow flag causes sparkrun to return to the command line after launching the containers. sparkrun automatically synchronizes the image to the Worker (skips if the same ID), copies the model to the Worker (skips if already present), sets up the Ray cluster, configures NCCL for the CX-7 interfaces, and launches containers on both Sparks.

sparkrun successfully completed all 6 steps. The ‘Mode: cluster (2 nodes)’ section was seen. The image was synchronized to the Worker over CX-7. --tensor-parallel-size 2 appears in the serve command. sparkrun detected the --distributed-executor-backend ray flag, selected the vllm-ray runtime, and automatically set up the Ray cluster.

At TP=2, the model weights are split in two — each Spark loads ~30 GB (vs. 61 GB at TP=1). This leaves 35.97 GiB of memory for KV cache on each Spark (vs. 8.46 GiB at TP=1) — meaning more concurrent requests.

3. Monitoring the Startup Process

After the model download completes, it may take a few minutes for vLLM to become ready to serve. During this process, vLLM loads the model weights into GPU memory, compiles GPU kernels, and allocates memory for inference.

Monitor the logs:

You will see the following in the logs:

If you see the Application startup complete. line, the model server is ready. Press Ctrl+C to stop watching the logs. The server will continue running in the background.

4. Testing the Model

Check the container status. Containers should be running on both Sparks:

Both containers are running.

Let’s run a health check on the model:

The model is healthy. Now let’s test the model with a prompt:

The model answered 12 × 17 = 204. Additionally, the reasoning and reasoning_content fields contain the model’s reasoning processes.

The API is served only through the head node (Main Spark). That is, we communicate with the model via the Main Spark. The Worker Spark only participates in the inference computation.

5. Shutdown

When you are done, stop the model. Since you used --tp 2 when launching, you need to specify --tp 2 when stopping as well:

This command stops the container and frees up memory. However, the container, the Docker image, and the model files remain on disk. This means you don’t need to re-download to start again. Simply re-run the sparkrun run command from step 2.

Using with Open WebUI

In the previous tutorial (Local LLM Serving with vLLM on DGX Spark), we set up Open WebUI and connected it to vLLM’s port 8000. The GPT-OSS 120B model is also served from the same port 8000. If Open WebUI is running, it automatically detects the model and lists it as openai/gpt-oss-120b in the model selection menu on the chat screen. You can select the model and start using it from the browser.

Single vs Dual Spark Performance Comparison

We compared the single Spark (TP=1) and dual Spark (TP=2) configurations at different concurrency levels. The measurements recorded average TTFT (Time to First Token — the time until the first token starts being generated) and TPS (Tokens Per Second — the number of tokens generated per second) values.

The dual Spark setup provides 26% higher token throughput at concurrency 1 (single user), and as concurrency increases, this gap grows, exceeding 60% at concurrency 16, as shown.

The measurements were taken using the CordatusAI LLM Benchmark Tool. This tool is a benchmarking application developed by CordatusAI that tests LLM servers with OpenAI-compatible APIs. Below are screenshots of the benchmark results the application produced for our model. As can be seen, the application can test at concurrency levels from 1 to 64, and at each level it measures TTFT, ITL, TPS, latency, and throughput metrics, presenting the results as tables and graphs. It also calculates the recommended number of users the system can support. This calculation is a general estimate based on certain assumptions and does not reflect all usage scenarios. As seen in the image, the tool calculated a recommended user count of 55 for a single Spark and 120 for dual Sparks.

Single Spark benchmark results:

Double Spark benchmark results:

For large AI models, the limiting factor is often not just compute performance, but memory capacity. When a model no longer fits within the memory of a single system, the workload must be distributed across multiple nodes. However, multi-node deployments are not used solely to accommodate models that are too large for one system. Even when a model can run on a single DGX Spark, using two or four systems with the right parallelism strategy and runtime configuration can increase aggregate token throughput and concurrent request-processing capacity.

There is an important distinction here: adding more nodes does not automatically improve performance in every scenario. The model architecture, inference engine, parallelism strategy, request pattern, and inter-node communication overhead all have a direct impact on the outcome. The real benefit of a multi-node system therefore comes not from the hardware count alone, but from optimizing the compute and communication layers together.

In this project, four NVIDIA DGX Spark systems were connected over a high-speed network and configured as a single AI cluster. The goal was to run models that could not be hosted on one Spark alone, while also increasing token throughput for suitable workloads.

The 10GbE RJ45 management network was separated from the 200GbE compute network provided by the ConnectX-7 interfaces. When both ConnectX-7 interfaces on the compute fabric were used together, approximately 200 Gbps of aggregate RDMA bandwidth was achieved between the Spark systems, with latency measured at 1–3 microseconds. The infrastructure was then validated under a real workload by running GLM-5.2 across all four nodes.

Why Build a Cluster?

When a model does not fit within the memory of a single GPU or system, several parallelism strategies can be used.

Tensor parallelism splits large tensors and computations within an individual model layer across multiple GPUs. Each GPU processes part of the same layer, and the intermediate results are exchanged between devices. This approach is particularly useful when a single layer or the model weights cannot fit into the memory of one GPU.

Pipeline parallelism divides the model into groups of layers or sequential stages. The first layers run on one GPU or node, while later layers run on another. Data, typically organized into micro-batches, moves through these stages like items along a production pipeline. Pipeline parallelism is especially useful in multi-node deployments because different parts of the model can be placed on separate devices.

Tensor and pipeline parallelism can also be combined. The most appropriate strategy depends on the model architecture, memory requirements, and the capabilities of the inference stack.

Because GPUs continuously exchange data during distributed execution, compute performance alone is not enough. If the network is slow, GPUs spend time waiting for data from other nodes instead of performing calculations. Cluster performance must therefore be evaluated as a combination of GPU compute capability and inter-node communication performance.

Architecture

The cluster was built on two physically separate networks:

• Management network (10GbE): SSH access, system administration, software updates, and NAS traffic
• Compute network (200GbE): RDMA and distributed model communication

This separation prevents intensive compute traffic from interfering with routine management and storage operations. The main components and their roles are summarized below:

The 400G QSFP-DD ports on the CRS812 were split into two independent 200G QSFP56 links using passive breakout cables. This allowed a single physical switch port to provide separate 200G connections to two Spark systems.

Why RDMA and RoCEv2?

With conventional TCP/IP communication, data passes through the operating system networking stack. This process can introduce CPU overhead, memory copies, and additional latency. In large-model workloads, where high volumes of data are continuously exchanged between nodes, that overhead becomes more significant.

RDMA, or Remote Direct Memory Access, enables data to be transferred from the memory of one system to another while reducing the load on the CPU and operating system networking stack. This can deliver lower latency and higher effective bandwidth.

This cluster uses RoCEv2, or RDMA over Converged Ethernet version 2, to carry RDMA traffic over an Ethernet fabric. Ethernet is inherently a lossy network, however, so packet loss and congestion behavior must be carefully managed for RoCEv2 to operate efficiently.

The following settings were therefore applied to the compute network:

• MTU 9000: Reduces packet-processing overhead by carrying large transfers in fewer frames
• DSCP 26 → TC3: Places RoCE traffic into a dedicated traffic class
• PFC: Temporarily pauses the affected traffic class during congestion instead of dropping packets
• ECN and CNP: Detect congestion early and signal the sender to reduce its transmission rate

PFC was enabled only for TC3, the traffic class carrying RDMA traffic. SSH and ordinary TCP traffic were not placed in the lossless queue.

Why the MikroTik CRS812?

NVIDIA Spectrum switches are a well-established option for high-speed AI fabrics. This project, however, targeted a compact and cost-conscious four-node lab cluster.

The CRS812 provides the features required for this design, including 400G QSFP-DD ports, breakout support, hardware-offloaded QoS, PFC, ECN, and DCBX. With the correct configuration, approximately 100–111 Gbps of RDMA throughput per link was achieved.

This does not mean that the CRS812 can replace a data-center-class switch in every deployment. For this four-node DGX Spark cluster, however, it provides a practical balance between cost and performance.

Preparing the Nodes

Before cluster setup, all four Spark systems were brought to the same operating system and firmware level. This step is more important than it may appear. Outdated firmware on high-speed network adapters can prevent the expected performance from being reached even when the network configuration is otherwise correct.

The operating system packages and firmware were updated, and the DGX Dashboard was checked for any remaining updates. On the Docker side, all nodes were configured to use the same storage backend, and overlayfs through the containerd snapshotter was verified.

For the management network, the 10GbE port on each Spark was connected to the CRS312 and assigned a static IP address. SSH access, system updates, sparkrun management, and NAS connectivity were all provided through this network.

What Does sparkrun Simplify?

Although four systems can be managed individually, repeatedly running the same commands on every node, configuring SSH keys, and manually launching distributed workloads quickly becomes error-prone.

sparkrun is a toolkit that creates an SSH mesh between DGX Spark nodes and simplifies multi-node model execution. In this deployment, sparkrun was used to:

• Configure passwordless SSH access between nodes
• Prepare the ConnectX-7 networks
• Configure user and Docker permissions
• Launch the model from a single recipe file

Although sparkrun automates many of the setup steps, the host-side DCB configuration was applied separately. DSCP 26 traffic was mapped to priority 3 on the ConnectX-7 interfaces, and PFC was enabled for that priority. A systemd service was created to persist these settings across reboots.

Validating the Network

The network was validated layer by layer before any model workload was launched.

Connectivity between the compute IP addresses was tested first. The ping -M do -s 8972 command was then used to confirm that MTU 9000 worked end to end.

At the TCP layer, iperf3 measured approximately 100–120 Gbps of throughput. The RDMA tests produced the following results on each ConnectX-7 subnet:

• 100–111 Gbps write bandwidth
• 95–110 Gbps read bandwidth

Across both subnets, the Spark systems achieved approximately 200 Gbps of aggregate RDMA bandwidth. The ib_write_lat test measured latency at approximately 1–3 microseconds.

These figures matter beyond network benchmarking. When tensor or pipeline parallelism is used, the time required to transfer data between nodes directly affects overall model performance.

Running a Real Workload with GLM-5.2

After the network tests were completed, GLM-5.2 was deployed across four Spark systems using a tensor parallel size of 4.

The model was launched through sparkrun using a single recipe:

sparkrun run glm52-qt-dcp4-4spark.yaml

In addition to the model definition, the recipe specifies the container image, memory limits, KV cache settings, quantization format, and vLLM runtime parameters. This makes the deployment reproducible without requiring the full command line to be reconstructed each time.

The Application startup complete. message confirmed that all four nodes were participating in the same distributed model deployment.

In a benchmark performed after the service was ready, the GLM-5.2-Int4 model achieved 20.16 output tokens per second at concurrency 1. This result confirmed that the distributed system could serve a real end-to-end inference request, rather than merely loading the model successfully.

Shared Storage

The ASUSTOR NAS was used as shared storage so that model files did not need to be copied separately to every node. Its two 10GbE ports were connected to the CRS312 using LACP, and the NFS share was mounted on all Spark nodes.

LACP does not automatically increase a single file transfer to 20 Gbps. Its benefit comes from distributing multiple connections and traffic flows across the two physical links.

Model weights, datasets, and benchmark outputs were stored in the shared NAS location.

Conclusion

This project transformed four independent DGX Spark systems into an integrated AI cluster running on shared networking and storage infrastructure. Separating management and compute traffic, implementing RoCEv2/RDMA end to end, centralizing node management with sparkrun, and using a shared NFS volume resulted in an infrastructure that is both manageable and stable.

The resulting platform can run models that exceed the memory capacity of a single Spark. Even when a model already fits on one system, multiple Spark nodes can be used with the right parallelism and serving configuration to increase aggregate token throughput and support more concurrent requests.

The approximately 200 Gbps of aggregate RDMA bandwidth, 1–3 microsecond latency, and successful four-node GLM-5.2 deployment demonstrate that the architecture works not only in theory, but under a real large-model inference workload. With the networking, storage, and cluster-management layers designed as a single system, DGX Spark can serve as the foundation for a compact yet capable distributed AI platform.

All commands, MikroTik switch settings, DCB configuration, network tests, NAS setup, and troubleshooting steps are available in the detailed DGX Spark 4-Node AI Cluster Deployment Guide.

To deploy a similar DGX Spark cluster, the complete system can be ordered from openzeka.com. Cluster networking, node configuration, and basic commissioning are included at no additional cost with the system purchase.