{"id":1338,"date":"2025-09-09T10:16:57","date_gmt":"2025-09-09T10:16:57","guid":{"rendered":"https:\/\/blog.aetherix.com\/?p=1338"},"modified":"2026-03-27T13:44:14","modified_gmt":"2026-03-27T13:44:14","slug":"how-to-run-vllm-on-jetson-agx-thor","status":"publish","type":"post","link":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/","title":{"rendered":"How to Run vLLM on Jetson AGX Thor?"},"content":{"rendered":"<div class=\"fusion-fullwidth fullwidth-box fusion-builder-row-1 fusion-flex-container has-pattern-background has-mask-background nonhundred-percent-fullwidth non-hundred-percent-height-scrolling\" style=\"--awb-border-radius-top-left:0px;--awb-border-radius-top-right:0px;--awb-border-radius-bottom-right:0px;--awb-border-radius-bottom-left:0px;--awb-flex-wrap:wrap;\" ><div class=\"fusion-builder-row fusion-row fusion-flex-align-items-flex-start fusion-flex-content-wrap\" style=\"max-width:1331.2px;margin-left: calc(-4% \/ 2 );margin-right: calc(-4% \/ 2 );\"><div class=\"fusion-layout-column fusion_builder_column fusion-builder-column-0 fusion_builder_column_1_1 1_1 fusion-flex-column\" style=\"--awb-bg-size:cover;--awb-width-large:100%;--awb-margin-top-large:0px;--awb-spacing-right-large:1.92%;--awb-margin-bottom-large:20px;--awb-spacing-left-large:1.92%;--awb-width-medium:100%;--awb-order-medium:0;--awb-spacing-right-medium:1.92%;--awb-spacing-left-medium:1.92%;--awb-width-small:100%;--awb-order-small:0;--awb-spacing-right-small:1.92%;--awb-spacing-left-small:1.92%;\"><div class=\"fusion-column-wrapper fusion-column-has-shadow fusion-flex-justify-content-flex-start fusion-content-layout-column\"><div class=\"fusion-title title fusion-title-1 fusion-sep-none fusion-title-text fusion-title-size-three\"><h3 class=\"fusion-title-heading title-heading-left\" style=\"margin:0;\">What is vLLM and Why Does It Matter on Jetson AGX Thor?<\/h3><\/div><div class=\"fusion-text fusion-text-1\"><p><strong>vLLM<\/strong> is an open-source inference engine designed to run large language models (LLMs) with exceptional efficiency. Thanks to its innovative PagedAttention architecture, vLLM delivers both high throughput and low latency making it possible to deploy advanced AI models in real-time applications.<\/p>\n<p>On the other side, NVIDIA Jetson AGX Thor is a next-generation edge AI platform built for robotics, autonomous machines, and industrial systems. With its immense compute power and AI acceleration, Thor is the perfect hardware to unlock the full potential of LLMs at the edge.<\/p>\n<p>When combined, vLLM on Jetson AGX Thor enables:<\/p>\n<ul>\n<li><strong>Real-time LLM services (chatbots, assistants, summarization, translation)<\/strong><\/li>\n<li><strong>Vision + Language use cases (explaining camera input instantly)<\/strong><\/li>\n<li><strong>On-device inference with ultra-low latency and stronger data privacy<\/strong><\/li>\n<li><strong>Reduced reliance on cloud resources, with better energy efficiency<\/strong><\/li>\n<\/ul>\n<p>In short, vLLM provides the software intelligence, Thor provides the hardware muscle together they make cutting-edge LLM experiences possible directly on the device.<\/p>\n<\/div><div class=\"fusion-title title fusion-title-2 fusion-sep-none fusion-title-text fusion-title-size-three\"><h3 class=\"fusion-title-heading title-heading-left\" style=\"margin:0;\">Installing Process<\/h3><\/div><div class=\"fusion-text fusion-text-2\"><p>First, download the following Triton Inference Server container image.<br \/>\nThis image comes with vLLM version 0.9.2 pre-installed. The tag 25.08 refers to August 2025.<\/p>\n<p>If you\u2019d like to update to a newer version in the future, you can always visit the <strong><a style=\"color: #00dd37;\" href=\"https:\/\/catalog.ngc.nvidia.com\/?filters=&amp;orderBy=weightPopularDESC&amp;query=&amp;page=&amp;pageSize=\">NVIDIA NGC Catalog<\/a><\/strong> to find the latest container releases.<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-1 > .CodeMirror, .fusion-syntax-highlighter-1 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-1 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_1\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_1\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_1\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">docker run --name vllm_container -it \\\n  --gpus all \\\n  -p 8000:8000 \\\n  -v $HOME\/.cache\/huggingface:\/root\/.cache\/huggingface \\\n  nvcr.io\/nvidia\/tritonserver:25.08-vllm-python-py3 bash<\/textarea><\/div><div class=\"fusion-text fusion-text-3\" style=\"--awb-margin-top:20px;\"><p>You can verify the installed vLLM version directly with Python.<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-2 > .CodeMirror, .fusion-syntax-highlighter-2 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-2 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_2\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_2\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_2\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">python3 -c \"import vllm; print(vllm.__version__)\"<\/textarea><\/div><div class=\"fusion-text fusion-text-4\" style=\"--awb-margin-top:20px;\"><p>Next, you\u2019ll need to create an account on Hugging Face , generate an access token, and log in with it.<\/p>\n<p>This token will allow the container to securely download and run models directly from <a href=\"https:\/\/huggingface.co\/\"><strong style=\"color: #00e200;\">Hugging Face.<\/strong><\/a><\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-3 > .CodeMirror, .fusion-syntax-highlighter-3 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-3 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_3\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_3\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_3\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">huggingface-cli login<\/textarea><\/div><div class=\"fusion-text fusion-text-5\" style=\"--awb-margin-top:20px;\"><p>To download model run ;<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-4 > .CodeMirror, .fusion-syntax-highlighter-4 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-4 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_4\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_4\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_4\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">hf download <model><\/textarea><\/div><div class=\"fusion-text fusion-text-6\" style=\"--awb-margin-top:20px;\"><p>Once your environment is ready, you can launch the vLLM API server using the following command:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-5 > .CodeMirror, .fusion-syntax-highlighter-5 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-5 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_5\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_5\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_5\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">python3 -m vllm.entrypoints.openai.api_server \\\n  --model meta-llama\/Llama-3.1-8B-Instruct \\\n  --tensor-parallel-size 1 \\\n  --gpu-memory-utilization 0.90 \\\n  --max-model-len 8192 \\\n  --dtype float16<\/textarea><\/div><div class=\"fusion-text fusion-text-7\" style=\"--awb-margin-top:20px;\"><p>Here\u2019s what each parameter does:<\/p>\n<ul>\n<li><strong><em>&#8211;model<\/em> \u2192<\/strong> specifies which model to load (in this case, Llama-3.1-8B-Instruct from Hugging Face).<\/li>\n<li><em><strong>&#8211;tensor-parallel-size 1<\/strong> <\/em>\u2192 runs the model on a single GPU. If you have multiple GPUs, you can increase this value.<\/li>\n<li><em><strong>&#8211;gpu-memory-utilization 0.90<\/strong><\/em> \u2192 tells vLLM to use up to 90% of available GPU memory. Adjust this if you run into memory errors.<\/li>\n<li><em><strong>&#8211;max-model-len 8192 \u2192<\/strong><\/em> sets the maximum context length (in tokens) for the model.<\/li>\n<li><em><strong>&#8211;dtype float16 \u2192<\/strong> <\/em>runs the model in FP16 precision, which is more efficient on Jetson AGX Thor.<\/li>\n<\/ul>\n<\/div><div class=\"fusion-text fusion-text-8\"><p><em><strong>\u26a0\ufe0f Heads-up: If you encounter ;<\/strong><\/em><\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-6 > .CodeMirror, .fusion-syntax-highlighter-6 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-6 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_6\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_6\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_6\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}<\/textarea><\/div><div class=\"fusion-text fusion-text-9\" style=\"--awb-margin-top:20px;\"><p><em><strong>It usually means the engine couldn\u2019t reserve enough GPU memory. Try lowering the GPU memory utilization. For example try with &#8211;gpu-memory-utilization 0.75 .<\/strong><\/em><\/p>\n<\/div><div class=\"fusion-image-element \" style=\"--awb-caption-title-font-family:var(--h2_typography-font-family);--awb-caption-title-font-weight:var(--h2_typography-font-weight);--awb-caption-title-font-style:var(--h2_typography-font-style);--awb-caption-title-size:var(--h2_typography-font-size);--awb-caption-title-transform:var(--h2_typography-text-transform);--awb-caption-title-line-height:var(--h2_typography-line-height);--awb-caption-title-letter-spacing:var(--h2_typography-letter-spacing);\"><span class=\" fusion-imageframe imageframe-none imageframe-1 hover-type-none\"><img decoding=\"async\" width=\"1024\" height=\"617\" title=\"Screenshot from 2025-09-09 09-33-20\" src=\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-09-09-33-20-1024x617.png\" alt class=\"img-responsive wp-image-1345\" srcset=\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-09-09-33-20-200x120.png 200w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-09-09-33-20-400x241.png 400w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-09-09-33-20-600x361.png 600w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-09-09-33-20-800x482.png 800w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-09-09-33-20-1200x723.png 1200w, https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/Screenshot-from-2025-09-09-09-33-20.png 1393w\" sizes=\"(max-width: 640px) 100vw, 1024px\" \/><\/span><\/div><div class=\"fusion-text fusion-text-10\" style=\"--awb-margin-top:20px;\"><p>If you see a message like:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-7 > .CodeMirror, .fusion-syntax-highlighter-7 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-7 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_7\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_7\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_7\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">Starting vLLM API server 0 on http:\/\/0.0.0.0:8000<\/textarea><\/div><div class=\"fusion-text fusion-text-11\" style=\"--awb-margin-top:20px;\"><p>it means that vLLM is now serving on port 8000 and ready to accept requests.<br \/>\nAt this point, you can start testing it with a simple curl command. For example:<\/p>\n<\/div><style type=\"text\/css\" scopped=\"scopped\">.fusion-syntax-highlighter-8 > .CodeMirror, .fusion-syntax-highlighter-8 > .CodeMirror .CodeMirror-gutters {background-color:#000000;}<\/style><div class=\"fusion-syntax-highlighter-container fusion-syntax-highlighter-8 fusion-syntax-highlighter-theme-dark\" style=\"opacity:0;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;font-size:14px;border-width:1px;border-style:solid;border-color:rgba(242,243,245,0);\"><div class=\"syntax-highlighter-copy-code\"><span class=\"syntax-highlighter-copy-code-title\" data-id=\"fusion_syntax_highlighter_8\" style=\"font-size:14px;\">Copy to Clipboard<\/span><\/div><label for=\"fusion_syntax_highlighter_8\" class=\"screen-reader-text\">Syntax Highlighter<\/label><textarea class=\"fusion-syntax-highlighter-textarea\" id=\"fusion_syntax_highlighter_8\" data-readOnly=\"nocursor\" data-lineNumbers=\"\" data-lineWrapping=\"\" data-theme=\"oceanic-next\" data-mode=\"text\/x-sh\">curl http:\/\/localhost:8000\/v1\/chat\/completions \\\n  -H \"Content-Type: application\/json\" \\\n  -d '{\n    \"model\": \"meta-llama\/Llama-3.1-8B-Instruct\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello Jetson AGX Thor!\"}],\n    \"max_tokens\": 64\n  }'<\/textarea><\/div><div class=\"fusion-title title fusion-title-3 fusion-sep-none fusion-title-text fusion-title-size-four\"><h4 class=\"fusion-title-heading title-heading-left\" style=\"margin:0;\">Which Jetson should I choose for my LLM model?<\/h4><\/div><div class=\"fusion-text fusion-text-12\"><p>Below, you can find the RAM requirements of the most popular LLM models along with Jetson recommendations that meet the minimum specifications to run them. You can choose the one that best fits your needs.<\/p>\n<\/div>\n<div class=\"table-1\">\n<table width=\"100%\">\n<thead>\n<tr>\n<th align=\"left\">Model<\/th>\n<th align=\"left\">Parameters<\/th>\n<th align=\"left\">Quantization<\/th>\n<th align=\"left\">Required RAM (GB)<\/th>\n<th align=\"left\">Recommended Minimum Jetson<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td align=\"left\">deepseek-ai Deepseek-R1 Base<\/td>\n<td align=\"left\">684B<\/td>\n<td align=\"left\">Dynamic-1.58-bit<\/td>\n<td align=\"left\">162.11<\/td>\n<td align=\"left\">Not supported (\u2265128 GB and above)<\/td>\n<\/tr>\n<tr>\n<td align=\"left\">deepseek-ai Deepseek-R1 Distill-Qwen-1.5B<\/td>\n<td align=\"left\">1.5B<\/td>\n<td align=\"left\">Q4_K_M<\/td>\n<td align=\"left\">0.90<\/td>\n<td align=\"left\">Jetson Orin Nano 4 GB, Jetson Nano 4 GB<\/td>\n<\/tr>\n<tr>\n<td align=\"left\">deepseek-ai Deepseek-R1 Distill-Qwen-7B<\/td>\n<td align=\"left\">7B<\/td>\n<td align=\"left\">Q5_K_M<\/td>\n<td align=\"left\">5.25<\/td>\n<td align=\"left\">Jetson Orin Nano 8 GB, Jetson Orin NX 8 GB, Jetson Xavier NX 8 GB<\/td>\n<\/tr>\n<tr>\n<td align=\"left\">mistralai Mixtral 8x22B-Instruct-v0.1<\/td>\n<td align=\"left\">22B<\/td>\n<td align=\"left\">Q4_K_M<\/td>\n<td align=\"left\">13.20<\/td>\n<td align=\"left\">Jetson Orin NX 16 GB, Jetson AGX Orin 32 GB, Jetson AGX Xavier 32 GB<\/td>\n<\/tr>\n<tr>\n<td align=\"left\">mistralai Mathstral 7B-v0.1<\/td>\n<td align=\"left\">7B<\/td>\n<td align=\"left\">Q5_K_M<\/td>\n<td align=\"left\">5.25<\/td>\n<td align=\"left\">Jetson Orin Nano 8 GB, Jetson Orin NX 8 GB, Jetson Xavier NX 8 GB<\/td>\n<\/tr>\n<tr>\n<td align=\"left\">google gemma-3 12b-it<\/td>\n<td align=\"left\">12B<\/td>\n<td align=\"left\">Q4_K_M<\/td>\n<td align=\"left\">7.20<\/td>\n<td align=\"left\">Jetson Orin NX 8 GB, Jetson Orin Nano 8 GB, Jetson Xavier NX 8 GB<\/td>\n<\/tr>\n<tr>\n<td align=\"left\">meta-llama Llama-3.1 70B-Instruct<\/td>\n<td align=\"left\">70B<\/td>\n<td align=\"left\">Q5_K_M<\/td>\n<td align=\"left\">52.50<\/td>\n<td align=\"left\">Jetson AGX Orin 64 GB, Jetson AGX Xavier 64 GB, Jetson AGX Thor (T5000) 128 GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":3,"featured_media":1606,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[],"class_list":["post-1338","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-generative-ai"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.3.1 (Yoast SEO v25.3.1) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to Run vLLM on Jetson AGX Thor? - OpenZeka EN Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Run vLLM on Jetson AGX Thor?\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/\" \/>\n<meta property=\"og:site_name\" content=\"OpenZeka EN Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/profile.php?id=61576911356211\" \/>\n<meta property=\"article:published_time\" content=\"2025-09-09T10:16:57+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-27T13:44:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Enhar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Aetherixnl\" \/>\n<meta name=\"twitter:site\" content=\"@Aetherixnl\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Enhar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/\"},\"author\":{\"name\":\"Enhar\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/62c964376839cf2c4b2eb682bf14d3cb\"},\"headline\":\"How to Run vLLM on Jetson AGX Thor?\",\"datePublished\":\"2025-09-09T10:16:57+00:00\",\"dateModified\":\"2026-03-27T13:44:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/\"},\"wordCount\":2191,\"publisher\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/#organization\"},\"image\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg\",\"articleSection\":[\"Generative AI\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/\",\"url\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/\",\"name\":\"How to Run vLLM on Jetson AGX Thor? - OpenZeka EN Blog\",\"isPartOf\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg\",\"datePublished\":\"2025-09-09T10:16:57+00:00\",\"dateModified\":\"2026-03-27T13:44:14+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#primaryimage\",\"url\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg\",\"contentUrl\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg\",\"width\":1920,\"height\":1080},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.openzeka.com\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Run vLLM on Jetson AGX Thor?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#website\",\"url\":\"https:\/\/blog.openzeka.com\/en\/\",\"name\":\"Aetherix B.V.\",\"description\":\"NVIDIA Jetson Developer Kits &amp;Edge Devices\",\"publisher\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.openzeka.com\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#organization\",\"name\":\"Aetherix B.V.\",\"url\":\"https:\/\/blog.openzeka.com\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/06\/aetherix-site-icon.webp\",\"contentUrl\":\"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/06\/aetherix-site-icon.webp\",\"width\":421,\"height\":398,\"caption\":\"Aetherix B.V.\"},\"image\":{\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/profile.php?id=61576911356211\",\"https:\/\/x.com\/Aetherixnl\",\"https:\/\/www.instagram.com\/aetherixnl\/\",\"https:\/\/www.tiktok.com\/@aetherixnl\"],\"description\":\"Aetherix provides a full range of NVIDIA Jetson-based edge AI solutions\u2014including Developer Kits, AI Kits, industrial-grade Carrier Boards, and fully integrated Boxed AI Systems.\",\"email\":\"info@aetherix.com\",\"legalName\":\"Aetherix B.V.\",\"vatID\":\"NL867727688B01\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/62c964376839cf2c4b2eb682bf14d3cb\",\"name\":\"Enhar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/189d567adce3bb0c8d438b4586bf861ec04980f2e451003975e3cf871781d0f4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/189d567adce3bb0c8d438b4586bf861ec04980f2e451003975e3cf871781d0f4?s=96&d=mm&r=g\",\"caption\":\"Enhar\"}}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How to Run vLLM on Jetson AGX Thor? - OpenZeka EN Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/","og_locale":"en_US","og_type":"article","og_title":"How to Run vLLM on Jetson AGX Thor?","og_url":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/","og_site_name":"OpenZeka EN Blog","article_publisher":"https:\/\/www.facebook.com\/profile.php?id=61576911356211","article_published_time":"2025-09-09T10:16:57+00:00","article_modified_time":"2026-03-27T13:44:14+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg","type":"image\/jpeg"}],"author":"Enhar","twitter_card":"summary_large_image","twitter_creator":"@Aetherixnl","twitter_site":"@Aetherixnl","twitter_misc":{"Written by":"Enhar","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#article","isPartOf":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/"},"author":{"name":"Enhar","@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/62c964376839cf2c4b2eb682bf14d3cb"},"headline":"How to Run vLLM on Jetson AGX Thor?","datePublished":"2025-09-09T10:16:57+00:00","dateModified":"2026-03-27T13:44:14+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/"},"wordCount":2191,"publisher":{"@id":"https:\/\/blog.openzeka.com\/en\/#organization"},"image":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg","articleSection":["Generative AI"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/","url":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/","name":"How to Run vLLM on Jetson AGX Thor? - OpenZeka EN Blog","isPartOf":{"@id":"https:\/\/blog.openzeka.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#primaryimage"},"image":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg","datePublished":"2025-09-09T10:16:57+00:00","dateModified":"2026-03-27T13:44:14+00:00","breadcrumb":{"@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#primaryimage","url":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg","contentUrl":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/09\/e.jpg","width":1920,"height":1080},{"@type":"BreadcrumbList","@id":"https:\/\/blog.openzeka.com\/en\/how-to-run-vllm-on-jetson-agx-thor\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.openzeka.com\/en\/"},{"@type":"ListItem","position":2,"name":"How to Run vLLM on Jetson AGX Thor?"}]},{"@type":"WebSite","@id":"https:\/\/blog.openzeka.com\/en\/#website","url":"https:\/\/blog.openzeka.com\/en\/","name":"Aetherix B.V.","description":"NVIDIA Jetson Developer Kits &amp;Edge Devices","publisher":{"@id":"https:\/\/blog.openzeka.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.openzeka.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/blog.openzeka.com\/en\/#organization","name":"Aetherix B.V.","url":"https:\/\/blog.openzeka.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/06\/aetherix-site-icon.webp","contentUrl":"https:\/\/blog.openzeka.com\/en\/wp-content\/uploads\/2025\/06\/aetherix-site-icon.webp","width":421,"height":398,"caption":"Aetherix B.V."},"image":{"@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/profile.php?id=61576911356211","https:\/\/x.com\/Aetherixnl","https:\/\/www.instagram.com\/aetherixnl\/","https:\/\/www.tiktok.com\/@aetherixnl"],"description":"Aetherix provides a full range of NVIDIA Jetson-based edge AI solutions\u2014including Developer Kits, AI Kits, industrial-grade Carrier Boards, and fully integrated Boxed AI Systems.","email":"info@aetherix.com","legalName":"Aetherix B.V.","vatID":"NL867727688B01"},{"@type":"Person","@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/62c964376839cf2c4b2eb682bf14d3cb","name":"Enhar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.openzeka.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/189d567adce3bb0c8d438b4586bf861ec04980f2e451003975e3cf871781d0f4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/189d567adce3bb0c8d438b4586bf861ec04980f2e451003975e3cf871781d0f4?s=96&d=mm&r=g","caption":"Enhar"}}]}},"_links":{"self":[{"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/posts\/1338","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/comments?post=1338"}],"version-history":[{"count":18,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/posts\/1338\/revisions"}],"predecessor-version":[{"id":1565,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/posts\/1338\/revisions\/1565"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/media\/1606"}],"wp:attachment":[{"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/media?parent=1338"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/categories?post=1338"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.openzeka.com\/en\/wp-json\/wp\/v2\/tags?post=1338"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}