{"id":1410,"date":"2025-09-12T10:44:53","date_gmt":"2025-09-12T10:44:53","guid":{"rendered":"https:\/\/blog.aetherix.com\/?p=1410"},"modified":"2026-03-27T13:39:49","modified_gmt":"2026-03-27T13:39:49","slug":"how-to-run-llama-cpp-server-on-jetson-agx-thor","status":"publish","type":"post","link":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/","title":{"rendered":"How to Run Llama.cpp Server on Jetson AGX Thor?"},"content":{"rendered":"<div class=\"fusion-fullwidth fullwidth-box fusion-builder-row-1 fusion-flex-container nonhundred-percent-fullwidth non-hundred-percent-height-scrolling\" style=\"--awb-border-radius-top-left:0px;--awb-border-radius-top-right:0px;--awb-border-radius-bottom-right:0px;--awb-border-radius-bottom-left:0px;--awb-flex-wrap:wrap;\" ><div class=\"fusion-builder-row fusion-row fusion-flex-align-items-flex-start fusion-flex-content-wrap\" style=\"max-width:1331.2px;margin-left: calc(-4% \/ 2 );margin-right: calc(-4% \/ 2 );\"><div class=\"fusion-layout-column fusion_builder_column fusion-builder-column-0 fusion_builder_column_1_1 1_1 fusion-flex-column\" style=\"--awb-bg-blend:overlay;--awb-bg-size:cover;--awb-width-large:100%;--awb-margin-top-large:0px;--awb-spacing-right-large:1.92%;--awb-margin-bottom-large:0px;--awb-spacing-left-large:1.92%;--awb-width-medium:100%;--awb-spacing-right-medium:1.92%;--awb-spacing-left-medium:1.92%;--awb-width-small:100%;--awb-spacing-right-small:1.92%;--awb-spacing-left-small:1.92%;\"><div class=\"fusion-column-wrapper fusion-flex-justify-content-flex-start fusion-content-layout-column\"><div class=\"fusion-title title fusion-title-1 fusion-sep-none fusion-title-text fusion-title-size-four\"><h4 class=\"fusion-title-heading title-heading-left\" style=\"margin:0;\">Llama.cpp Server on Jetson AGX Thor: Unlocking Edge AI with Large Language Models<\/h4><\/div><div class=\"fusion-text fusion-text-1\"><p><strong>Llama.cpp Server<\/strong> is a lightweight, high-performance runtime for large language models (LLMs), designed to run efficiently on both CPU and GPU. Built in C++, it eliminates unnecessary overhead and delivers deep hardware-level optimizations. By supporting the <strong>GGUF model format,<\/strong> it allows for quantization, drastically reducing memory requirements while maintaining accuracy. Through its<strong> REST API,<\/strong> Llama.cpp Server can be seamlessly integrated into applications, enabling developers to bring advanced LLM capabilities directly to devices\u2014without relying on the cloud.<\/p>\n<p>When deployed on <strong>NVIDIA Jetson AGX Thor<\/strong>, the advantages become even more compelling:<\/p>\n<ul>\n<li>GPU acceleration with<strong> CUDA<\/strong> ensures that the Thor\u2019s compute power is fully utilized, bringing real-time inference to the edge.<\/li>\n<li>Optimized for edge AI use cases such as robotics, autonomous systems, and industrial automation, it provides ultra-low latency decision-making.<\/li>\n<li>Resource efficiency via quantization makes it possible to run models from 7B up to 13B parameters within the limited memory budgets typical of embedded devices.<\/li>\n<\/ul>\n<p>By combining <strong>Llama.cpp Server<\/strong> with Jetson<strong> AGX Thor<\/strong>, organizations gain a powerful platform for on-device AI that is private, fast, and cost-effective. No data needs to leave the device, latency is minimized, and the system remains fully adaptable to both prototyping and production scenarios. Supported by an open-source ecosystem, this pairing represents a breakthrough for deploying large language models securely and efficiently at the edge.<\/p>\n<\/div><\/div><\/div><div class=\"fusion-layout-column fusion_builder_column fusion-builder-column-1 fusion_builder_column_1_1 1_1 fusion-flex-column\" style=\"--awb-bg-size:cover;--awb-width-large:100%;--awb-margin-top-large:0px;--awb-spacing-right-large:1.92%;--awb-margin-bottom-large:20px;--awb-spacing-left-large:1.92%;--awb-width-medium:100%;--awb-order-medium:0;--awb-spacing-right-medium:1.92%;--awb-spacing-left-medium:1.92%;--awb-width-small:100%;--awb-order-small:0;--awb-spacing-right-small:1.92%;--awb-spacing-left-small:1.92%;\"><div class=\"fusion-column-wrapper fusion-column-has-shadow fusion-flex-justify-content-flex-start fusion-content-layout-column\"><div class=\"fusion-title title fusion-title-2 fusion-sep-none fusion-title-text fusion-title-size-three\"><h3 class=\"fusion-title-heading title-heading-left\" style=\"margin:0;\">Requirements<\/h3><\/div><div class=\"fusion-text fusion-text-2\"><ul>\n<li>JetPack 7 (<span style=\"color: #76b900;\"><a style=\"color: #76b900;\" href=\"https:\/\/blog.openzeka.com\/en\/what-is-nvidia-jetpack-beginner-friendly-guide\/\">Learn more about JetPack<\/a><\/span>)<\/li>\n<li>CUDA 13<\/li>\n<li>At least 10 GB of free disk space<strong> (Only for the Llama Server image, not for the models.)<\/strong><\/li>\n<li>A stable and fast internet connection<\/li>\n<\/ul>\n<\/div><div class=\"fusion-title title fusion-title-3 fusion-sep-none fusion-title-text fusion-title-size-four\"><h4 class=\"fusion-title-heading title-heading-left\" style=\"margin:0;\">How to use Llama.cpp Server ?<\/h4><\/div><div class=\"fusion-text fusion-text-3\"><p>Firstly download the image ;<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-1 > .CodeMirror, .fusion-syntax-highlighter-1 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-1 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_1\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_1\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_1\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">docker run --gpus all -it --rm \\\n  -p 8080:8080 \\\n  -v \/workspace\/models:\/models \\\n  ghcr.io\/nvidia-ai-iot\/llama_cpp:r38.2.arm64-sbsa-cu130-24.04 \\\n  \/bin\/bash<\/textarea><\/div><div class=\"fusion-text fusion-text-4\" style=\"--awb-margin-top:20px;\"><p>Then, download the model from Hugging Face. If the model requires access, log in with your token by running:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-2 > .CodeMirror, .fusion-syntax-highlighter-2 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-2 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_2\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_2\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_2\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\"># huggingface-cli login\nhf download Qwen\/Qwen3-4B-Instruct-2507<\/textarea><\/div><div class=\"fusion-text fusion-text-5\" style=\"--awb-margin-top:20px;\"><p>Then, install the required Python dependencies with the following command:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-3 > .CodeMirror, .fusion-syntax-highlighter-3 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-3 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_3\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_3\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_3\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">pip install transformers torch mistral_common sentencepiece<\/textarea><\/div><div class=\"fusion-text fusion-text-6\" style=\"--awb-margin-top:20px;\"><p>This command set downloads the <strong>NVIDIA NVPL local repository package<\/strong>, installs it, adds the signing key to the system, and then installs the NVPL library via apt-get.<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-4 > .CodeMirror, .fusion-syntax-highlighter-4 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-4 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_4\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_4\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_4\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">export NVPL_VERSION=25.5\nexport DISTRO=ubuntu2404\n\nwget https:\/\/developer.download.nvidia.com\/compute\/nvpl\/${NVPL_VERSION}\/local_installers\/nvpl-local-repo-${DISTRO}-${NVPL_VERSION}_1.0-1_arm64.deb\n\ndpkg -i nvpl-local-repo-ubuntu2404-25.5_1.0-1_arm64.deb\n\ncp \/var\/nvpl-local-repo-ubuntu2404-25.5\/nvpl-local-52E38D21-keyring.gpg \/usr\/share\/keyrings\/\n\napt-get update && apt-get install -y nvpl<\/textarea><\/div><div class=\"fusion-text fusion-text-7\" style=\"--awb-margin-top:20px;\"><p>This command takes the Qwen2.5-VL-3B-Instruct model downloaded from Hugging Face (inside the snapshot folder identified by ), and uses the convert_hf_to_gguf.py tool to convert the Hugging Face weights (safetensors\/PyTorch) into GGUF format, saving the output as \/data\/models\/Qwen3-4B-Instruct-2507-f16.gguf.<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-5 > .CodeMirror, .fusion-syntax-highlighter-5 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-5 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_5\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_5\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_5\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">python3 \/opt\/llama_cpp_python\/vendor\/llama.cpp\/convert_hf_to_gguf.py \\\n  \/data\/models\/huggingface\/models--Qwen--Qwen2.5-VL-3B-Instruct\/snapshots\/<hash> \\\n  --outfile \/data\/models\/Qwen3-4B-Instruct-2507-f16.gguf<\/textarea><\/div><div class=\"fusion-text fusion-text-8\" style=\"--awb-margin-top:20px;\"><p>This command takes the full-precision GGUF model (Qwen3-4B-Instruct-2507-f16.gguf) and runs it through llama-quantize to produce a quantized version <strong>(Qwen3-4B-Instruct-2507-q4_k_m.gguf)<\/strong> using the<strong> q4_k_m quantization method.<\/strong><\/p>\n<ul>\n<li><strong>Input file:<\/strong> \/data\/models\/Qwen3-4B-Instruct-2507-f16.gguf (the FP16 model converted from Hugging Face).<\/li>\n<li><strong>Output file:<\/strong> \/data\/models\/Qwen3-4B-Instruct-2507-q4_k_m.gguf (smaller, quantized model).<\/li>\n<li><strong>Quantization type:<\/strong> q4_k_m \u2192 a 4-bit quantization scheme optimized for speed and memory efficiency.<\/li>\n<\/ul>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-6 > .CodeMirror, .fusion-syntax-highlighter-6 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-6 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_6\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_6\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_6\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">llama-quantize \/data\/models\/Qwen3-4B-Instruct-2507-f16.gguf \\\n  \/data\/models\/Qwen3-4B-Instruct-2507-q4_k_m.gguf q4_k_m<\/textarea><\/div><div class=\"fusion-text fusion-text-9\" style=\"--awb-margin-top:20px;\"><p>This command launches the llama.cpp server so the quantized model can be served via an<strong> HTTP API.<\/strong><\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-7 > .CodeMirror, .fusion-syntax-highlighter-7 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-7 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_7\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_7\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_7\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">llama-server \\\n  -m \/data\/models\/Qwen3-4B-Instruct-2507-q4_k_m.gguf \\\n  --host 0.0.0.0 --port 8080 \\\n  -c 8192 \\\n  --n-gpu-layers 35<\/textarea><\/div><div class=\"fusion-text fusion-text-10\" style=\"--awb-margin-top:20px;\"><p>And thats it ! You can start chatting .<\/p>\n<\/div><div class=\"fusion-image-element \" style=\"--awb-caption-title-font-family:var(--h2_typography-font-family);--awb-caption-title-font-weight:var(--h2_typography-font-weight);--awb-caption-title-font-style:var(--h2_typography-font-style);--awb-caption-title-size:var(--h2_typography-font-size);--awb-caption-title-transform:var(--h2_typography-text-transform);--awb-caption-title-line-height:var(--h2_typography-line-height);--awb-caption-title-letter-spacing:var(--h2_typography-letter-spacing);\"><span class=\" fusion-imageframe imageframe-none imageframe-1 hover-type-zoomin\"><img decoding=\"async\" width=\"1024\" height=\"568\" title=\"Screenshot from 2025-09-12 13-31-53\" src=\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-12-13-31-53-1024x568.png\" alt class=\"img-responsive wp-image-1422\" srcset=\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-12-13-31-53-200x111.png 200w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-12-13-31-53-400x222.png 400w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-12-13-31-53-600x333.png 600w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-12-13-31-53-800x444.png 800w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-12-13-31-53-1200x666.png 1200w\" sizes=\"(max-width: 640px) 100vw, 1024px\" \/><\/span><\/div><\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":3,"featured_media":1602,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[],"class_list":["post-1410","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-generative-ai"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.3.1 (Yoast SEO v25.3.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to Run Llama.cpp Server on Jetson AGX Thor? - OpenZeka EN Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Run Llama.cpp Server on Jetson AGX Thor?\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/\" \/>\n<meta property=\"og:site_name\" content=\"OpenZeka EN Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/profile.php?id=61576911356211\" \/>\n<meta property=\"article:published_time\" content=\"2025-09-12T10:44:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-27T13:39:49+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Enhar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Aetherixnl\" \/>\n<meta name=\"twitter:site\" content=\"@Aetherixnl\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Enhar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/\"},\"author\":{\"name\":\"Enhar\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/62c964376839cf2c4b2eb682bf14d3cb\"},\"headline\":\"How to Run Llama.cpp Server on Jetson AGX Thor?\",\"datePublished\":\"2025-09-12T10:44:53+00:00\",\"dateModified\":\"2026-03-27T13:39:49+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/\"},\"wordCount\":1979,\"publisher\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/#organization\"},\"image\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg\",\"articleSection\":[\"Generative AI\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/\",\"url\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/\",\"name\":\"How to Run Llama.cpp Server on Jetson AGX Thor? - OpenZeka EN Blog\",\"isPartOf\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg\",\"datePublished\":\"2025-09-12T10:44:53+00:00\",\"dateModified\":\"2026-03-27T13:39:49+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#primaryimage\",\"url\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg\",\"contentUrl\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg\",\"width\":1920,\"height\":1080},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.openzeka.com\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Run Llama.cpp Server on Jetson AGX Thor?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#website\",\"url\":\"https:\/\/blog.openzeka.com\/en\/\",\"name\":\"Aetherix B.V.\",\"description\":\"NVIDIA Jetson Developer Kits &amp;Edge Devices\",\"publisher\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.openzeka.com\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#organization\",\"name\":\"Aetherix B.V.\",\"url\":\"https:\/\/blog.openzeka.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/06\/aetherix-site-icon.webp\",\"contentUrl\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/06\/aetherix-site-icon.webp\",\"width\":421,\"height\":398,\"caption\":\"Aetherix B.V.\"},\"image\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/profile.php?id=61576911356211\",\"https:\/\/x.com\/Aetherixnl\",\"https:\/\/www.instagram.com\/aetherixnl\/\",\"https:\/\/www.tiktok.com\/@aetherixnl\"],\"description\":\"Aetherix provides a full range of NVIDIA Jetson-based edge AI solutions\u2014including Developer Kits, AI Kits, industrial-grade Carrier Boards, and fully integrated Boxed AI Systems.\",\"email\":\"info@aetherix.com\",\"legalName\":\"Aetherix B.V.\",\"vatID\":\"NL867727688B01\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/62c964376839cf2c4b2eb682bf14d3cb\",\"name\":\"Enhar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/189d567adce3bb0c8d438b4586bf861ec04980f2e451003975e3cf871781d0f4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/189d567adce3bb0c8d438b4586bf861ec04980f2e451003975e3cf871781d0f4?s=96&d=mm&r=g\",\"caption\":\"Enhar\"}}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How to Run Llama.cpp Server on Jetson AGX Thor? - OpenZeka EN Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/","og_locale":"en_US","og_type":"article","og_title":"How to Run Llama.cpp Server on Jetson AGX Thor?","og_url":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/","og_site_name":"OpenZeka EN Blog","article_publisher":"https:\/\/www.facebook.com\/profile.php?id=61576911356211","article_published_time":"2025-09-12T10:44:53+00:00","article_modified_time":"2026-03-27T13:39:49+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg","type":"image\/jpeg"}],"author":"Enhar","twitter_card":"summary_large_image","twitter_creator":"@Aetherixnl","twitter_site":"@Aetherixnl","twitter_misc":{"Written by":"Enhar","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#article","isPartOf":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/"},"author":{"name":"Enhar","@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/62c964376839cf2c4b2eb682bf14d3cb"},"headline":"How to Run Llama.cpp Server on Jetson AGX Thor?","datePublished":"2025-09-12T10:44:53+00:00","dateModified":"2026-03-27T13:39:49+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/"},"wordCount":1979,"publisher":{"@id":"https:\/\/blog.openzeka.com\/en\/#organization"},"image":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg","articleSection":["Generative AI"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/","url":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/","name":"How to Run Llama.cpp Server on Jetson AGX Thor? - OpenZeka EN Blog","isPartOf":{"@id":"https:\/\/blog.openzeka.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#primaryimage"},"image":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg","datePublished":"2025-09-12T10:44:53+00:00","dateModified":"2026-03-27T13:39:49+00:00","breadcrumb":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#primaryimage","url":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg","contentUrl":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/b.jpg","width":1920,"height":1080},{"@type":"BreadcrumbList","@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-llama-cpp-server-on-jetson-agx-thor\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.openzeka.com\/en\/"},{"@type":"ListItem","position":2,"name":"How to Run Llama.cpp Server on Jetson AGX Thor?"}]},{"@type":"WebSite","@id":"https:\/\/blog.openzeka.com\/en\/#website","url":"https:\/\/blog.openzeka.com\/en\/","name":"Aetherix B.V.","description":"NVIDIA Jetson Developer Kits &amp;Edge Devices","publisher":{"@id":"https:\/\/blog.openzeka.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.openzeka.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/blog.openzeka.com\/en\/#organization","name":"Aetherix B.V.","url":"https:\/\/blog.openzeka.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/06\/aetherix-site-icon.webp","contentUrl":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/06\/aetherix-site-icon.webp","width":421,"height":398,"caption":"Aetherix B.V."},"image":{"@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/profile.php?id=61576911356211","https:\/\/x.com\/Aetherixnl","https:\/\/www.instagram.com\/aetherixnl\/","https:\/\/www.tiktok.com\/@aetherixnl"],"description":"Aetherix provides a full range of NVIDIA Jetson-based edge AI solutions\u2014including Developer Kits, AI Kits, industrial-grade Carrier Boards, and fully integrated Boxed AI Systems.","email":"info@aetherix.com","legalName":"Aetherix B.V.","vatID":"NL867727688B01"},{"@type":"Person","@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/62c964376839cf2c4b2eb682bf14d3cb","name":"Enhar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/189d567adce3bb0c8d438b4586bf861ec04980f2e451003975e3cf871781d0f4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/189d567adce3bb0c8d438b4586bf861ec04980f2e451003975e3cf871781d0f4?s=96&d=mm&r=g","caption":"Enhar"}}]}},"_links":{"self":[{"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/posts\/1410","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/comments?post=1410"}],"version-history":[{"count":12,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/posts\/1410\/revisions"}],"predecessor-version":[{"id":1603,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/posts\/1410\/revisions\/1603"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/media\/1602"}],"wp:attachment":[{"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/media?parent=1410"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/categories?post=1410"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/tags?post=1410"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}