Jetson Thor 部(bù)署(shǔ) VSS（Video Search and Summarization）全(quán)教程
準備工作(zuò)
設備版本(běn)：
Jetson核心板：Jetson Thor T5000
JetPack版本(běn)：JetPack 7.1 (L4T 38.4.0)
安(ān)裝(zhuāng)Docker
1，# docker
2，sudo apt-get ｕｐｄａｔｅ
3，sudo apt-get install -y ca-certificates curl
4，sudo install -m 0755 -d /etc/apt/keyrings
5，sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
6，sudo chmod a+r /etc/apt/keyrings/docker.asc
7，
8，echo \
9， "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
10， $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
11， sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
12，sudo apt-get ｕｐｄａｔｅ
13，sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
14，
15，# nvidia-container
16，curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
17， && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
18， sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
19， sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
20，sudo apt-get ｕｐｄａｔｅ
21，export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
22，sudo apt-get install -y \
23， nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
24， nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
25， libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
26， libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
27，
28，sudo apt install -y jq
29，jq -n --indent 4 '{"runtimes": {"nvidia": {"args": [], "path": "nvidia-container-runtime"}}}' > daemon.json && \
30， sudo mv daemon.json /etc/docker/daemon.json
31，sudo systemctl daemon-reload && sudo systemctl restart docker
32，
33，sudo usermod -aG docker $USER
34，newgrp docker
獲取(qǔ)NGC API Key
- 進(jìn)入(rù)https://ngc.nvidia.com/並(bìng)登录(lù)賬号(hào)
- 在(zài)用(yòng)戶配置欄中(zhōng)進(jìn)入(rù)https://org.ngc.nvidia.com/setup/api-keys页(yè)面(miàn)
- 進(jìn)入(rù)Legacy API Key页(yè)面(miàn)，點(diǎn)擊Generate Legacy Key生(shēng)成(chéng)API Key
- 根(gēn)據(jù)提(tí)示複制保存生(shēng)成(chéng)的(de)API Key
獲取(qǔ) Hugging Face 令牌(pái)
- 登录(lù) Hugging Face 平台(tái)：訪問(wèn)网(wǎng)址 https://huggingface.co/，登录(lù)你的(de)賬号(hào)。
- 進(jìn)入(rù) API 密鑰页(yè)面(miàn)：跳轉(zhuǎn)至(zhì) API Keys 專屬页(yè)面(miàn)。
- 生(shēng)成(chéng) API 密鑰：如(rú)下(xià)图(tú)所(suǒ)示，點(diǎn)擊 “生(shēng)成(chéng)新(xīn)令牌(pái)（Generate New Token）” 按鈕。複制生(shēng)成(chéng)的(de)令牌(pái)，並(bìng)将其(qí)妥善保存在(zài)安(ān)全(quán)位(wèi)置，以備後(hòu)續使用(yòng)。
配置VSS Event Reviewer
- 下(xià)载(zài)Video Search and Summarization
1，git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git -b v2.4.1
2，cd video-search-and-summarization/
- 修改Event Reviewer功能(néng)的(de)API Key
1，cd deploy/docker/event_reviewer
2，vim .env
3，
4，# ｕｐｄａｔｅ to download Cosmos-Reason2 / other models from HuggingFace
5，HF_TOKEN=hf_***
6，
7，# ｕｐｄａｔｅ to download Cosmos-Reason1 from NGC
8，NGC_API_KEY=nvapi-***
- 修改docker compose配置（可(kě)選）
1，## 添加模型本(běn)地(dì)緩存
2，mkdir models
3，
4，# 打(dǎ)開(kāi)compose.yaml
5，vim compose.yaml
修改compose.yaml文(wén)件(jiàn)
1，services:
2，via-server:
3， image: ${VSS_IMAGE:-nvcr.io/nvidia/blueprint/vss-engine:2.4.1${IS_SBSA:+-sbsa}}
4， networks:
5， - vss-shared-network
6， shm_size: '16gb'
7，runtime: nvidia
8， ports:
9， - "${VSS_BACKEND_PORT:-8000}:${VSS_BACKEND_PORT:-8000}"
10， - "${VSS_FRONTEND_PORT:-9000}:${VSS_FRONTEND_PORT:-9000}"
11， volumes:
12， - "${MODEL_ROOT_DIR:-/dummy}${MODEL_ROOT_DIR:+:${MODEL_ROOT_DIR:-}}"
13， ## 注釋以下(xià)两(liǎng)行
14， # - via-ngc-model-cache:/root/.via/ngc_model_cache
15， # - via-hf-cache:/tmp/huggingface
16， ## 添加模型緩存目录(lù)
17， - ./models/via-ngc-model-cache:/root/.via/ngc_model_cache
18， - ./models/via-hf-cache:/tmp/huggingface
19， - "${ALERT_REVIEW_MEDIA_BASE_DIR:-/dummy}${ALERT_REVIEW_MEDIA_BASE_DIR:+:${ALERT_REVIEW_MEDIA_BASE_DIR:-}}"
20，
21， environment:
22， BACKEND_PORT: "${VSS_BACKEND_PORT:-8000}"
23， DISABLE_CA_RAG: "true"
24， DISABLE_FRONTEND: "true"
25， DISABLE_GUARDRAILS: "true"
26， DISABLE_CV_PIPELINE: "true"
27， FRONTEND_PORT: "${VSS_FRONTEND_PORT:-9000}"
28，MODEL_PATH: "${MODEL_PATH:-git:https://huggingface.co/nvidia/Cosmos-Reason2-8B}"
29，VLM_MODEL_TO_USE: "${VLM_MODEL_TO_USE:-cosmos-reason2}"
30， VLLM_GPU_MEMORY_UTILIZATION: "${VLLM_GPU_MEMORY_UTILIZATION:-}"
31，VLM_MAX_MODEL_LEN: "${VLM_MAX_MODEL_LEN:-}"
32， VSS_LOG_LEVEL: "${VSS_LOG_LEVEL:-}"
33，VSS_EXTRA_ARGS: "${VSS_EXTRA_ARGS:-}"
34， VLM_DEFAULT_NUM_FRAMES_PER_CHUNK: "${VLM_DEFAULT_NUM_FRAMES_PER_CHUNK:-}"
35， ALERT_REVIEW_MEDIA_BASE_DIR: "${ALERT_REVIEW_MEDIA_BASE_DIR:-}"
36，VLM_SYSTEM_PROMPT: "You are a helpful assistant. Answer the user's question. "
37， VLM_INPUT_WIDTH: "${VLM_INPUT_WIDTH:-}"
38， VLM_INPUT_HEIGHT: "${VLM_INPUT_HEIGHT:-}"
39， NVIDIA_VISIBLE_DEVICES: "${NVIDIA_VISIBLE_DEVICES:-}"
40， NGC_API_KEY: "${NGC_API_KEY:-}"
41， HF_TOKEN: "${HF_TOKEN:-}"
42， ## 添加科學(xué)上(shàng)网(wǎng)代(dài)理(lǐ)（如(rú)果(guǒ)需要(yào)）
43， HTTP_PROXY: "http://xxx.xxx.xxx.xxx:7890"
44，HTTPS_PROXY: "http://xxx.xxx.xxx.xxx:7890"
45，ulimits:
46， memlock:
47， soft: -1
48， hard: -1
49， stack: 67108864
50， ipc: host
51，healthcheck:
52，test: ["CMD", "curl", "-f", "http://localhost:${VSS_BACKEND_PORT:-8000}/health/live"]
53， interval: 30s
54，timeout: 20s
55，retries: 25
56， start_period: 90s
- 運行VSS Event Reviewer
1，# 創建VSS Event Reviewer网(wǎng)络
2，docker network create vss-shared-network
3，
4，# 如(rú)果(guǒ)升(shēng)級了(le)vss需清(qīng)空(kōng)vst數據(jù)
5，rm -rf vst/vst_volume/*
6，
7，# 運行VSS Event Reviewer，包(bāo)含Alert Bridge, VLM Pipeline, Alert Inspector UI和(hé)Video Storage Toolkit功能(néng)
8，ALERT_REVIEW_MEDIA_BASE_DIR=/tmp/alert-media-dir docker compose up -d
注意(yì)：
1.第(dì)一(yī)次(cì)運行需要(yào)下(xià)载(zài)docker鏡(jìng)像，可(kě)能(néng)会(huì)有(yǒu)些(xiē)docker鏡(jìng)像無法(fǎ)正(zhèng)常下(xià)载(zài)，需要(yào)手(shǒu)动下(xià)载(zài)对應(yìng)的(de)tar包(bāo)進(jìn)行加载(zài)，並(bìng)在(zài)compose.yaml中(zhōng)修改为當前(qián)使用(yòng)的(de)docker鏡(jìng)像名(míng)稱
2.進(jìn)入(rù)容器里(lǐ)会(huì)自(zì)动下(xià)载(zài)模型並(bìng)進(jìn)行加载(zài)
- 關(guān)閉VSS Event Reviewer
1，cd deploy/docker/event_reviewer
2，
3，ALERT_REVIEW_MEDIA_BASE_DIR=/tmp/alert-media-dir docker compose down
調用(yòng)大(dà)模型後(hòu)可(kě)能(néng)会(huì)造成(chéng)內(nèi)存占用(yòng)，需要(yào)運行命令進(jìn)行清(qīng)理(lǐ)，可(kě)執行脚本(běn)：
1，sudo sh deploy/scripts/sys_cache_cleaner.sh
配置cv-event-detector
- 進(jìn)入(rù)cv-event-detector目录(lù)
1，# change directory to example of CV detector
2，cd video-search-and-summarization/examples/cv-event-detector
- 修改docker compose配置（可(kě)選）
1，## 添加模型本(běn)地(dì)緩存
2，mkdir models
3，
4，# 打(dǎ)開(kāi)compose.yaml
5，vim compose.yaml
修改compose.yaml文(wén)件(jiàn)
1，services:
2，
3， nv-cv-event-detector:
4， image: ${NV_CV_EVENT_DETECTOR_IMAGE:-nvcr.io/nvidia/blueprint/nv-cv-event-detector:2.4.0${IS_SBSA:+-sbsa}}
5，networks:
6， - vss-shared-network
7，shm_size: '16gb'
8，runtime: nvidia
9， ports:
10， - "${NV_CV_EVENT_DETECTOR_API_PORT:-23491}:${NV_CV_EVENT_DETECTOR_API_PORT:-23491}"
11， entrypoint: ["/opt/nvidia/nv-cv-event-detector/start_nv_cv_event_detector.sh"]
12，
13，volumes:
14， ## 注釋以下(xià)两(liǎng)行
15， # - via-ngc-model-cache:/root/.via/ngc_model_cache
16， # - via-hf-cache:/tmp/huggingface
17，## 添加模型緩存目录(lù)
18， - ./models/via-ngc-model-cache:/root/.via/ngc_model_cache
19， - ./models/via-hf-cache:/tmp/huggingface
20， - "${ALERT_REVIEW_MEDIA_BASE_DIR:-/dummy}${ALERT_REVIEW_MEDIA_BASE_DIR:+:${ALERT_REVIEW_MEDIA_BASE_DIR:-}}"
21， - cv-input-dir:/tmp
22， - ./gdinoconfig_grpc.txt:/opt/nvidia/nv-cv-event-detector/gdinoconfig_grpc.txt
23，
24， environment:
25，NV_CV_EVENT_DETECTOR_API_PORT: ${NV_CV_EVENT_DETECTOR_API_PORT:-23491}
26， INSTALL_PROPRIETARY_CODECS: "true"
27， USE_GDINO: "${USE_GDINO:-true}"
28， DISABLE_SOM_OVERLAY: "${DISABLE_SOM_OVERLAY:-false}"
29， USE_TRACKER_CONFIG: "${USE_TRACKER_CONFIG:-/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml}"
30， CLIP_CACHE_PRE_EV_TIME: "${CLIP_CACHE_PRE_EV_TIME:-5}" #5s
31， CLIP_CACHE_POST_EV_TIME: "${CLIP_CACHE_POST_EV_TIME:-20}" #25s
32， ENABLE_FILE_STREAMING_MODE: "${ENABLE_FILE_STREAMING_MODE:-false}"
33，NVIDIA_VISIBLE_DEVICES: "${NVIDIA_VISIBLE_DEVICES:-}"
34， LD_LIBRARY_PATH: "/opt/nvidia/deepstream/deepstream/lib:/opt/tritonserver/lib/"
35， ## 添加科學(xué)上(shàng)网(wǎng)代(dài)理(lǐ)（如(rú)果(guǒ)需要(yào)）
36， HTTP_PROXY: "http://xxx.xxx.xxx.xxx:7890"
37， HTTPS_PROXY: "http://xxx.xxx.xxx.xxx:7890"
38， ulimits:
39， memlock:
40， soft: -1
41， hard: -1
42， nofile:
43， soft: 65535
44，hard: 65535
45， stack: 67108864
46，ipc: host
47， healthcheck:
48，test: ["CMD", "curl", "-f", "http://localhost:${NV_CV_EVENT_DETECTOR_API_PORT:-23491}/health"]
49，interval: 30s
50， timeout: 20s
51， retries: 50
52， start_period: 90s
- 運行cv-event-detector
1，# 運行cv-event-detector，包(bāo)含cv-ui功能(néng)
2，ALERT_REVIEW_MEDIA_BASE_DIR=/tmp/alert-media-dir docker compose -f compose.yaml -f compose.thor.yaml up -d
第(dì)一(yī)次(cì)運行需要(yào)下(xià)载(zài)docker鏡(jìng)像並(bìng)下(xià)载(zài)並(bìng)加载(zài)模型
- 關(guān)閉cv-event-detector
1，# change directory to example of CV detector
2，cd examples/cv-event-detector
3，
4，ALERT_REVIEW_MEDIA_BASE_DIR=/tmp/alert-media-dir docker compose down
進(jìn)入(rù)Inspector WebUI
設置视頻Pipeline
1. 打(dǎ)開(kāi)浏覽器，輸入(rù)http://0.0.0.0:7862進(jìn)入(rù)Computer Vision Pipeline Manager页(yè)面(miàn)
2. 選擇Examples中(zhōng)的(de)视頻文(wén)件(jiàn)，可(kě)修改Detection Parameters和(hé)Alert Prompts
3. 點(diǎn)擊Process Video開(kāi)始(shǐ)處(chù)理(lǐ)视頻
查看(kàn)Alert結果(guǒ)
1. 在(zài)浏覽器中(zhōng)輸入(rù)http://0.0.0.0:7860進(jìn)入(rù)Alert Inspector页(yè)面(miàn)
2. 在(zài)DASHBOARD中(zhōng)選擇想(xiǎng)進(jìn)行查看(kàn)的(de)视頻流，點(diǎn)擊条(tiáo)目最(zuì)右(yòu)側的(de)Chat图(tú)标(biāo)進(jìn)入(rù)CHAT功能(néng)並(bìng)播放(fàng)视頻
3. 可(kě)在(zài)聊天(tiān)框中(zhōng)輸入(rù)問(wèn)题點(diǎn)擊Ask獲取(qǔ)VLM的(de)結果(guǒ)