华为昇腾910B3搭建DeepSeek-R1-Distill-Llama-70B

## 平台介绍

**Atlas800训练服务器（型号：9000）**

- CPU: 4 × HUAWEI Kunpeng 920 5250 (2.6GHz, 48 cores)
- NPU: 8 × Ascend 910B3 64GB
- 架构：Arm
- 系统: CtyunOS 2.0.1

## 安装驱动

https://www.hiascend.com/hardware/firmware-drivers/community?product=1&model=30&cann=8.0.0.beta1&driver=1.0.28.alpha

![请输入图片描述](https://blog.tierl.com/usr/uploads/2025/02/481853396.png)

下载驱动，上传服务器，安装驱动

```
chmod +x Ascend-hdk-910b-npu-driver_24.1.rc3_linux-aarch64.run
./Ascend-hdk-910b-npu-driver_24.1.rc3_linux-aarch64.run --full --install-for-all
```

执行 `npu-smi info`，查看没有有信息输出，有代表驱动安装完成

```
(base) [root@host-sysf6h ~]# npu-smi info
+------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc3                 Version: 24.1.rc3                                             |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     910B3               | OK            | 92.2        37                0    / 0             |
| 0                         | 0000:C1:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 1     910B3               | OK            | 93.9        37                0    / 0             |
| 0                         | 0000:C2:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 2     910B3               | OK            | 88.1        37                0    / 0             |
| 0                         | 0000:81:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 3     910B3               | OK            | 92.3        37                0    / 0             |
| 0                         | 0000:82:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 4     910B3               | OK            | 91.5        42                0    / 0             |
| 0                         | 0000:01:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 5     910B3               | OK            | 94.5        41                0    / 0             |
| 0                         | 0000:02:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 6     910B3               | OK            | 97.2        41                0    / 0             |
| 0                         | 0000:41:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 7     910B3               | OK            | 86.8        40                0    / 0             |
| 0                         | 0000:42:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
```

## CANN Toolkit 和CANN 算子

```
wget wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.0.RC3.20/Atlas-A3-cann-kernels_8.0.RC3.20_linux-aarch64.run
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.0.RC3.20/Ascend-cann-toolkit_8.0.RC3.20_linux-aarch64.run
```

安装 Toolkit，按提示操作：

```shell
sudo sed -i 's/user=true/user=false/' ~/.pip/pip.conf
sudo chmod +x Ascend-cann-toolkit_8.0.RC3.20_linux-aarch64.run && sudo ./Ascend-cann-toolkit_8.0.RC3.20_linux-aarch64.run --install --install-for-all
```

安装 Kernel，按提示操作：

```shell
sudo chmod +x Atlas-A3-cann-kernels_8.0.RC3.20_linux-aarch64.run && sudo ./Atlas-A3-cann-kernels_8.0.RC3.20_linux-aarch64.run --install --install-for-all
```

配置环境变量：

```shell
sudo echo "source /usr/local/Ascend/ascend-toolkit/set_env.sh" >> /etc/profile
source /usr/local/Ascend/ascend-toolkit/set_env.sh
```

## 下载docker，拉取MindIE容器镜像

下载安装docker

```
yum install -y docker
docker info # 查看是否安装成功
```

**去华为申请MindIE容器镜像拉取链接**

```
# 这里只是示例，镜像需要去华为申请
docker pull mindie:1.0.0-800I-A2-py311-openeuler24.03-lts
```

## 下载模型

这里可以通过各种方法下载DeepSeek R1 模型，我使用的huggingface-cli

```
huggingface-cli download --resume-download DeepSeek-R1-Distill-Llama-70B --local-dir /data2/DeepSeek-R1-Distill-Llama-70B
```

## 量化模型部署

```
# 先提前创建好量化后的模型存放位置,这里/data2/mindie/DeepSeek-R1是量化前的源模型路径，/data1/deepseek-R1-bf16是量化后的模型路径，具体请看命令
docker run -it -d --net=host --shm-size=1g \
    --privileged \
    --name mindie \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --device=/dev/devmm_svm \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
    -v /usr/local/sbin:/usr/local/sbin:ro \
    -v /data2/mindie/DeepSeek-R1-Distill-Llama-70B:/DeepSeek-R1-Distill-Llama-70B:rw \
    -v /data1/DeepSeek-R1-Distill-Llama-70B-bf16:/DeepSeek-R1-Distill-Llama-70B-bf16:rw \
    mindie:py311-ubuntu22.04-arm64 bash

docker exec -it mindie bash
```

## 量化权重生成

- 生成量化权重依赖msModelSlim工具，安装方式见[此README](https://gitee.com/ascend/msit/tree/dev/msmodelslim)
- 量化权重统一使用${ATB_SPEED_HOME_PATH}/examples/convert/model_slim/quantifier.py脚本生成，以下提供Llama模型量化权重生成快速启动命令
- W8A8量化权重请使用以下指令生成
  
  - 注意该量化方式仅支持在Atlas 800I A2服务器上运行

```
cd ~
git clone https://gitee.com/ascend/msit.git
cd msit/msmodelslim
bash install.sh

# 设置CANN包的环境变量
source /usr/local/Ascend/ascend-toolkit/set_env.sh
# 关闭虚拟内存
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False
# 进入atb-models目录
cd ${ATB_SPEED_HOME_PATH}
sed -i '167s/m3/m4/' examples/models/llama3/generate_quant_weight.sh
# DeepSeek-R1-Distill-Llama-70B量化 bf16，有回退层，antioutlier使用m4算法配置，使用min-max量化方式，校准数据集使用50条BoolQ数据，在NPU上进行运算
bash examples/models/llama3/generate_quant_weight.sh -src /DeepSeek-R1-Distill-Llama-70B -dst /DeepSeek-R1-Distill-Llama-70B-bf16 -type llama3.1_70b_instruct_bf16_w8a8
```

## 使用MindIE容器部署推理

```
docker run -it -d --net=host --shm-size=1g \
    --privileged \
    --name mindie \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --device=/dev/devmm_svm \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
    -v /usr/local/sbin:/usr/local/sbin:ro \
    -v /data1/DeepSeek-R1-Distill-Llama-70B-bf16:/DeepSeek-R1-Distill-Llama-70B-bf16:rw \
    mindie:py311-ubuntu22.04-arm64 bash
```

```
docker exec -it mindie bash
cd /usr/local/Ascend/mindie/latest/mindie-service/bin
./mindieservice\_daemon
```

### 对话测试

进入llm_model路径

```
cd $ATB_SPEED_HOME_PATH
```

执行对话测试

```
torchrun --nproc_per_node 8 \
         --master_port 20037 \
         -m examples.run_pa \
         --model_path /DeepSeek-R1-Distill-Llama-70B-bf16 \
         --input_texts '你是什么模型' \
         --max_output_length 20 \
```

### 性能测试

进入ModelTest路径

```
cd $ATB_SPEED_HOME_PATH/tests/modeltest/
```

运行测试脚本
具体执行batch=1, 输入长度256, 输出长度256用例的2卡并行性能测试命令为：

```
bash run.sh pa_bf16 performance [[256,256]] 1 deepseekv2 /DeepSeek-R1-Distill-Llama-70B-bf16 8
```

## 服务化推理

- 打开配置文件

```
vim /usr/local/Ascend/mindie/latest/mindie-service/conf/config.json
```

- 更改配置文件

```
root@host-sysf6h:/# cat /usr/local/Ascend/mindie/latest/mindie-service/conf/config.json
{
    "Version" : "1.0.0",
    "LogConfig" :
    {
        "logLevel" : "Info",
        "logFileSize" : 20,
        "logFileNum" : 20,
        "logPath" : "logs/mindie-server.log"
    },

"ServerConfig" :
    {
        "ipAddress" : "172.28.119.4",
        "managementIpAddress" : "127.0.0.2",
        "port" : 1025,
        "managementPort" : 1026,
        "metricsPort" : 1027,
        "allowAllZeroIpListening" : false,
        "maxLinkNum" : 1000,
        "httpsEnabled" : false,
        "fullTextEnabled" : false,
        "tlsCaPath" : "security/ca/",
        "tlsCaFile" : ["ca.pem"],
        "tlsCert" : "security/certs/server.pem",
        "tlsPk" : "security/keys/server.key.pem",
        "tlsPkPwd" : "security/pass/key_pwd.txt",
        "tlsCrlPath" : "security/certs/",
        "tlsCrlFiles" : ["server_crl.pem"],
        "managementTlsCaFile" : ["management_ca.pem"],
        "managementTlsCert" : "security/certs/management/server.pem",
        "managementTlsPk" : "security/keys/management/server.key.pem",
        "managementTlsPkPwd" : "security/pass/management/key_pwd.txt",
        "managementTlsCrlPath" : "security/management/certs/",
        "managementTlsCrlFiles" : ["server_crl.pem"],
        "kmcKsfMaster" : "tools/pmt/master/ksfa",
        "kmcKsfStandby" : "tools/pmt/standby/ksfb",
        "inferMode" : "standard",
        "interCommTLSEnabled" : true,
        "interCommPort" : 1121,
        "interCommTlsCaPath" : "security/grpc/ca/",
        "interCommTlsCaFiles" : ["ca.pem"],
        "interCommTlsCert" : "security/grpc/certs/server.pem",
        "interCommPk" : "security/grpc/keys/server.key.pem",
        "interCommPkPwd" : "security/grpc/pass/key_pwd.txt",
        "interCommTlsCrlPath" : "security/grpc/certs/",
        "interCommTlsCrlFiles" : ["server_crl.pem"],
        "openAiSupport" : "vllm"
    },

"BackendConfig" : {
        "backendName" : "mindieservice_llm_engine",
        "modelInstanceNumber" : 1,
        "npuDeviceIds" : [[0,1,2,3,4,5,6,7]],
        "tokenizerProcessNumber" : 8,
        "multiNodesInferEnabled" : false,
        "multiNodesInferPort" : 1120,
        "interNodeTLSEnabled" : false,
        "interNodeTlsCaPath" : "security/grpc/ca/",
        "interNodeTlsCaFiles" : ["ca.pem"],
        "interNodeTlsCert" : "security/grpc/certs/server.pem",
        "interNodeTlsPk" : "security/grpc/keys/server.key.pem",
        "interNodeTlsPkPwd" : "security/grpc/pass/mindie_server_key_pwd.txt",
        "interNodeTlsCrlPath" : "security/grpc/certs/",
        "interNodeTlsCrlFiles" : ["server_crl.pem"],
        "interNodeKmcKsfMaster" : "tools/pmt/master/ksfa",
        "interNodeKmcKsfStandby" : "tools/pmt/standby/ksfb",
        "ModelDeployConfig" :
        {
            "maxSeqLen" : 131072,
            "maxInputTokenLen" : 131072,
            "truncation" : false,
            "ModelConfig" : [
                {
                    "modelInstanceType" : "Standard",
                    "modelName" : "llama",
                    "modelWeightPath" : "/DeepSeek-R1-Distill-Llama-70B-bf16",
                    "worldSize" : 8,
                    "cpuMemSize" : 10,
                    "npuMemSize" : -1,
                    "backendType" : "atb",
                    "trustRemoteCode" : false
                }
            ]
        },

"ScheduleConfig" :
        {
            "templateType" : "Standard",
            "templateName" : "Standard_LLM",
            "cacheBlockSize" : 128,

"maxPrefillBatchSize" : 50,
            "maxPrefillTokens" : 131072,
            "prefillTimeMsPerReq" : 150,
            "prefillPolicyType" : 0,

"decodeTimeMsPerReq" : 50,
            "decodePolicyType" : 0,

"maxBatchSize" : 200,
            "maxIterTimes" : 131072,
            "maxPreemptCount" : 0,
            "supportSelectBatch" : false,
            "maxQueueDelayMicroseconds" : 5000
        }
    }
}
```

- 拉起服务化

```
cd /usr/local/Ascend/mindie/latest/mindie-service/bin
./mindieservice_daemon
```

```
curl 172.28.119.4:1025/generate -d '{
"prompt": "是先有鸡还是先有蛋?",
"max_tokens": 128000,
"stream": false,
"do_sample":true,
"repetition_penalty": 1.00,
"temperature": 0.01,
"top_p": 0.001,
"top_k": 1,
"model": "llama"
}'
```

```
curl -H "Accept: application/json" -H "Content-type: application/json"  -X POST -d '{
    "model": "llama",
    "messages": [{
    "role": "user",
    "content": "是先有鸡还是先有蛋?"
    }],
    "stream": false
}' http://172.28.119.4:1025/v1/chat/completions
```

## 推理服务化docker

```
# 开一个screen窗口用来管理docker
screen -S docker
docker exec -it mindie bash
cd /usr/local/Ascend/mindie/latest/mindie-service/bin
./mindieservice\_daemon
```

使用oneapi管理key

```
docker run --name one-api -d --restart always -p 3000:3000 -e TZ=Asia/Shanghai -v /opt/oneapi/data:/data justsong/one-api
```

## 其他

> `max_split_size_mb:<value>，大于设定值的内存块在使用过程中不会进行切分，这有助于减少内存碎片。此选项主要应用于当模型由于oom(内存不足)而中断并显示存在大量非活动的拆分内存块场景的优化。<value>默认值为无限大，单位MB，即所有大小的内存块都可以拆分，最小设置值大于20MB。`

```
export PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:2048,garbage_collection_threshold:0.9
```

ref: [模型库-ModelZoo-昇腾社区](https://www.hiascend.com/software/modelzoo/models)

平台介绍

Atlas800训练服务器（型号：9000）

CPU: 4 × HUAWEI Kunpeng 920 5250 (2.6GHz, 48 cores)
NPU: 8 × Ascend 910B3 64GB
架构：Arm
系统: CtyunOS 2.0.1

安装驱动

https://www.hiascend.com/hardware/firmware-drivers/community?product=1&model=30&cann=8.0.0.beta1&driver=1.0.28.alpha

下载驱动，上传服务器，安装驱动

chmod +x Ascend-hdk-910b-npu-driver_24.1.rc3_linux-aarch64.run
./Ascend-hdk-910b-npu-driver_24.1.rc3_linux-aarch64.run --full --install-for-all

执行 npu-smi info，查看没有有信息输出，有代表驱动安装完成

(base) [root@host-sysf6h ~]# npu-smi info
+------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc3                 Version: 24.1.rc3                                             |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     910B3               | OK            | 92.2        37                0    / 0             |
| 0                         | 0000:C1:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 1     910B3               | OK            | 93.9        37                0    / 0             |
| 0                         | 0000:C2:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 2     910B3               | OK            | 88.1        37                0    / 0             |
| 0                         | 0000:81:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 3     910B3               | OK            | 92.3        37                0    / 0             |
| 0                         | 0000:82:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 4     910B3               | OK            | 91.5        42                0    / 0             |
| 0                         | 0000:01:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 5     910B3               | OK            | 94.5        41                0    / 0             |
| 0                         | 0000:02:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 6     910B3               | OK            | 97.2        41                0    / 0             |
| 0                         | 0000:41:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+
| 7     910B3               | OK            | 86.8        40                0    / 0             |
| 0                         | 0000:42:00.0  | 0           0    / 0          3370 / 65536         |
+===========================+===============+====================================================+

CANN Toolkit 和CANN 算子

wget wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.0.RC3.20/Atlas-A3-cann-kernels_8.0.RC3.20_linux-aarch64.run
wget https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%208.0.RC3.20/Ascend-cann-toolkit_8.0.RC3.20_linux-aarch64.run

安装 Toolkit，按提示操作：

sudo sed -i 's/user=true/user=false/' ~/.pip/pip.conf
sudo chmod +x Ascend-cann-toolkit_8.0.RC3.20_linux-aarch64.run && sudo ./Ascend-cann-toolkit_8.0.RC3.20_linux-aarch64.run --install --install-for-all

安装 Kernel，按提示操作：

sudo chmod +x Atlas-A3-cann-kernels_8.0.RC3.20_linux-aarch64.run && sudo ./Atlas-A3-cann-kernels_8.0.RC3.20_linux-aarch64.run --install --install-for-all

配置环境变量：

sudo echo "source /usr/local/Ascend/ascend-toolkit/set_env.sh" >> /etc/profile
source /usr/local/Ascend/ascend-toolkit/set_env.sh

下载docker，拉取MindIE容器镜像

下载安装docker

yum install -y docker
docker info # 查看是否安装成功

去华为申请MindIE容器镜像拉取链接

# 这里只是示例，镜像需要去华为申请
docker pull mindie:1.0.0-800I-A2-py311-openeuler24.03-lts

下载模型

这里可以通过各种方法下载DeepSeek R1 模型，我使用的huggingface-cli

huggingface-cli download --resume-download DeepSeek-R1-Distill-Llama-70B --local-dir /data2/DeepSeek-R1-Distill-Llama-70B

量化模型部署

# 先提前创建好量化后的模型存放位置,这里/data2/mindie/DeepSeek-R1是量化前的源模型路径，/data1/deepseek-R1-bf16是量化后的模型路径，具体请看命令
docker run -it -d --net=host --shm-size=1g \
    --privileged \
    --name mindie \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --device=/dev/devmm_svm \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
    -v /usr/local/sbin:/usr/local/sbin:ro \
    -v /data2/mindie/DeepSeek-R1-Distill-Llama-70B:/DeepSeek-R1-Distill-Llama-70B:rw \
    -v /data1/DeepSeek-R1-Distill-Llama-70B-bf16:/DeepSeek-R1-Distill-Llama-70B-bf16:rw \
    mindie:py311-ubuntu22.04-arm64 bash


docker exec -it mindie bash

量化权重生成

生成量化权重依赖msModelSlim工具，安装方式见此README
量化权重统一使用${ATB_SPEED_HOME_PATH}/examples/convert/model_slim/quantifier.py脚本生成，以下提供Llama模型量化权重生成快速启动命令
W8A8量化权重请使用以下指令生成
- 注意该量化方式仅支持在Atlas 800I A2服务器上运行

cd ~
git clone https://gitee.com/ascend/msit.git
cd msit/msmodelslim
bash install.sh



# 设置CANN包的环境变量
source /usr/local/Ascend/ascend-toolkit/set_env.sh
# 关闭虚拟内存
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False
# 进入atb-models目录
cd ${ATB_SPEED_HOME_PATH}
sed -i '167s/m3/m4/' examples/models/llama3/generate_quant_weight.sh
# DeepSeek-R1-Distill-Llama-70B量化 bf16，有回退层，antioutlier使用m4算法配置，使用min-max量化方式，校准数据集使用50条BoolQ数据，在NPU上进行运算
bash examples/models/llama3/generate_quant_weight.sh -src /DeepSeek-R1-Distill-Llama-70B -dst /DeepSeek-R1-Distill-Llama-70B-bf16 -type llama3.1_70b_instruct_bf16_w8a8

使用MindIE容器部署推理

docker run -it -d --net=host --shm-size=1g \
    --privileged \
    --name mindie \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --device=/dev/devmm_svm \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
    -v /usr/local/sbin:/usr/local/sbin:ro \
    -v /data1/DeepSeek-R1-Distill-Llama-70B-bf16:/DeepSeek-R1-Distill-Llama-70B-bf16:rw \
    mindie:py311-ubuntu22.04-arm64 bash

docker exec -it mindie bash
cd /usr/local/Ascend/mindie/latest/mindie-service/bin
./mindieservice\_daemon

对话测试

进入llm_model路径

cd $ATB_SPEED_HOME_PATH

执行对话测试

torchrun --nproc_per_node 8 \
         --master_port 20037 \
         -m examples.run_pa \
         --model_path /DeepSeek-R1-Distill-Llama-70B-bf16 \
         --input_texts '你是什么模型' \
         --max_output_length 20 \

性能测试

进入ModelTest路径

cd $ATB_SPEED_HOME_PATH/tests/modeltest/

运行测试脚本
具体执行batch=1, 输入长度256, 输出长度256用例的2卡并行性能测试命令为：

bash run.sh pa_bf16 performance [[256,256]] 1 deepseekv2 /DeepSeek-R1-Distill-Llama-70B-bf16 8

服务化推理

打开配置文件

vim /usr/local/Ascend/mindie/latest/mindie-service/conf/config.json

更改配置文件

root@host-sysf6h:/# cat /usr/local/Ascend/mindie/latest/mindie-service/conf/config.json
{
    "Version" : "1.0.0",
    "LogConfig" :
    {
        "logLevel" : "Info",
        "logFileSize" : 20,
        "logFileNum" : 20,
        "logPath" : "logs/mindie-server.log"
    },

    "ServerConfig" :
    {
        "ipAddress" : "172.28.119.4",
        "managementIpAddress" : "127.0.0.2",
        "port" : 1025,
        "managementPort" : 1026,
        "metricsPort" : 1027,
        "allowAllZeroIpListening" : false,
        "maxLinkNum" : 1000,
        "httpsEnabled" : false,
        "fullTextEnabled" : false,
        "tlsCaPath" : "security/ca/",
        "tlsCaFile" : ["ca.pem"],
        "tlsCert" : "security/certs/server.pem",
        "tlsPk" : "security/keys/server.key.pem",
        "tlsPkPwd" : "security/pass/key_pwd.txt",
        "tlsCrlPath" : "security/certs/",
        "tlsCrlFiles" : ["server_crl.pem"],
        "managementTlsCaFile" : ["management_ca.pem"],
        "managementTlsCert" : "security/certs/management/server.pem",
        "managementTlsPk" : "security/keys/management/server.key.pem",
        "managementTlsPkPwd" : "security/pass/management/key_pwd.txt",
        "managementTlsCrlPath" : "security/management/certs/",
        "managementTlsCrlFiles" : ["server_crl.pem"],
        "kmcKsfMaster" : "tools/pmt/master/ksfa",
        "kmcKsfStandby" : "tools/pmt/standby/ksfb",
        "inferMode" : "standard",
        "interCommTLSEnabled" : true,
        "interCommPort" : 1121,
        "interCommTlsCaPath" : "security/grpc/ca/",
        "interCommTlsCaFiles" : ["ca.pem"],
        "interCommTlsCert" : "security/grpc/certs/server.pem",
        "interCommPk" : "security/grpc/keys/server.key.pem",
        "interCommPkPwd" : "security/grpc/pass/key_pwd.txt",
        "interCommTlsCrlPath" : "security/grpc/certs/",
        "interCommTlsCrlFiles" : ["server_crl.pem"],
        "openAiSupport" : "vllm"
    },

    "BackendConfig" : {
        "backendName" : "mindieservice_llm_engine",
        "modelInstanceNumber" : 1,
        "npuDeviceIds" : [[0,1,2,3,4,5,6,7]],
        "tokenizerProcessNumber" : 8,
        "multiNodesInferEnabled" : false,
        "multiNodesInferPort" : 1120,
        "interNodeTLSEnabled" : false,
        "interNodeTlsCaPath" : "security/grpc/ca/",
        "interNodeTlsCaFiles" : ["ca.pem"],
        "interNodeTlsCert" : "security/grpc/certs/server.pem",
        "interNodeTlsPk" : "security/grpc/keys/server.key.pem",
        "interNodeTlsPkPwd" : "security/grpc/pass/mindie_server_key_pwd.txt",
        "interNodeTlsCrlPath" : "security/grpc/certs/",
        "interNodeTlsCrlFiles" : ["server_crl.pem"],
        "interNodeKmcKsfMaster" : "tools/pmt/master/ksfa",
        "interNodeKmcKsfStandby" : "tools/pmt/standby/ksfb",
        "ModelDeployConfig" :
        {
            "maxSeqLen" : 131072,
            "maxInputTokenLen" : 131072,
            "truncation" : false,
            "ModelConfig" : [
                {
                    "modelInstanceType" : "Standard",
                    "modelName" : "llama",
                    "modelWeightPath" : "/DeepSeek-R1-Distill-Llama-70B-bf16",
                    "worldSize" : 8,
                    "cpuMemSize" : 10,
                    "npuMemSize" : -1,
                    "backendType" : "atb",
                    "trustRemoteCode" : false
                }
            ]
        },

        "ScheduleConfig" :
        {
            "templateType" : "Standard",
            "templateName" : "Standard_LLM",
            "cacheBlockSize" : 128,

            "maxPrefillBatchSize" : 50,
            "maxPrefillTokens" : 131072,
            "prefillTimeMsPerReq" : 150,
            "prefillPolicyType" : 0,

            "decodeTimeMsPerReq" : 50,
            "decodePolicyType" : 0,

            "maxBatchSize" : 200,
            "maxIterTimes" : 131072,
            "maxPreemptCount" : 0,
            "supportSelectBatch" : false,
            "maxQueueDelayMicroseconds" : 5000
        }
    }
}

拉起服务化

cd /usr/local/Ascend/mindie/latest/mindie-service/bin
./mindieservice_daemon

curl 172.28.119.4:1025/generate -d '{
"prompt": "是先有鸡还是先有蛋?",
"max_tokens": 128000,
"stream": false,
"do_sample":true,
"repetition_penalty": 1.00,
"temperature": 0.01,
"top_p": 0.001,
"top_k": 1,
"model": "llama"
}'

curl -H "Accept: application/json" -H "Content-type: application/json"  -X POST -d '{
    "model": "llama",
    "messages": [{
    "role": "user",
    "content": "是先有鸡还是先有蛋?"
    }],
    "stream": false
}' http://172.28.119.4:1025/v1/chat/completions

推理服务化docker

docker run -it -d --net=host --shm-size=1g \
    --privileged \
    --name mindie \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --device=/dev/devmm_svm \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
    -v /usr/local/sbin:/usr/local/sbin:ro \
    -v /data1/DeepSeek-R1-Distill-Llama-70B-bf16:/DeepSeek-R1-Distill-Llama-70B-bf16:ro \
    -v /opt/oneapi/mindie/config.json:/usr/local/Ascend/mindie/latest/mindie-service/conf/config.json:rw \
    --workdir /usr/local/Ascend/mindie/latest/mindie-service/bin \
    mindie:py311-ubuntu22.04-arm64 bash

# 开一个screen窗口用来管理docker
screen -S docker
docker exec -it mindie bash
cd /usr/local/Ascend/mindie/latest/mindie-service/bin
./mindieservice\_daemon

使用oneapi管理key

docker run --name one-api -d --restart always -p 3000:3000 -e TZ=Asia/Shanghai -v /opt/oneapi/data:/data justsong/one-api

其他

max_split_size_mb:<value>，大于设定值的内存块在使用过程中不会进行切分，这有助于减少内存碎片。此选项主要应用于当模型由于oom(内存不足)而中断并显示存在大量非活动的拆分内存块场景的优化。<value>默认值为无限大，单位MB，即所有大小的内存块都可以拆分，最小设置值大于20MB。

export PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:2048,garbage_collection_threshold:0.9

ref: 模型库-ModelZoo-昇腾社区

最后修改：2025 年 02 月 11 日

感谢您的支持

3 条评论

啊啊啊
July 19th, 2025 at 08:57 am

RuntimeError: call aclnnIsFinite failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 3239121] 2025-07-19-08:55:59.546.125 Parse dynamic kernel config fail.
TraceBack (most recent call last): AclOpKernelInit failed opType IsFiniteAiCore ADD_TO_LAUNCHER_LIST_AICORE failed.
请问大佬这是为什么啊

回复
susuweiheng
February 16th, 2025 at 07:08 pm

您好想问一下那如果是训练服务器里面的300i duo的卡要不要安装驱动呢

回复
1. 小兔崽子
  February 21st, 2025 at 02:15 pm
  
  @susuweiheng
  
  如果用npu-smi info 没有显示，就要装驱动和固件
  
  回复

发表评论取消回复
使用cookie技术保留您的个人信息以便您下次快速评论，继续评论表示您已同意该条款

评论 *

私密评论

名称 *

🎲

邮箱 *

地址

Leo
这个思路很实用：直接把 MCP 插件跑在 Typecho 里面...
天天绿软Ttzip
感谢分享经验，谢谢站长！！已收藏
Ttzip天天软件
很棒的网站！@天天下载Ttzip
迈克
private-network 缺少了B类私网地址.172.1...
MD
太感謝了

华为昇腾910B3搭建DeepSeek-R1-Distill-Llama-70B

平台介绍

安装驱动

CANN Toolkit 和CANN 算子

下载docker，拉取MindIE容器镜像

下载模型

量化模型部署

量化权重生成

使用MindIE容器部署推理

对话测试

性能测试

服务化推理

推理服务化docker

其他

3 条评论

发表评论取消回复
使用cookie技术保留您的个人信息以便您下次快速评论，继续评论表示您已同意该条款

使用RouterOS，OSPF 和OpenWRT给国内外 IP 分流

RouterOS使用IP地址列表分流，分流给旁路由或者VPN

TrueNAS APP应用添加TrueCharts社区catalog目录

TrueNAS SCALE 常见错误排错（持续更新）

TrueNAS SCALE APP应用安装教程，自定义app安装

TrueNAS SCALE基础设置

TrueNAS SCALE 应用安装Alist，挂载共享各种网盘（百度云盘，阿里云盘，115等）

日历订阅- 支持苹果日历

TrueNAS SCALE 挂载其他文件系统硬盘

RouterOS wireguard与外网链接回家访问家里局域网

华为昇腾910B3搭建DeepSeek-R1-Distill-Llama-70B

平台介绍

安装驱动

CANN Toolkit 和CANN 算子

下载docker，拉取MindIE容器镜像

下载模型

量化模型部署

量化权重生成

使用MindIE容器部署推理

对话测试

性能测试

服务化推理

推理服务化docker

其他

3 条评论

发表评论 取消回复 使用cookie技术保留您的个人信息以便您下次快速评论，继续评论表示您已同意该条款

华为昇腾910B3搭建DeepSeek-R1-Distill-Llama-70B

发表评论取消回复
使用cookie技术保留您的个人信息以便您下次快速评论，继续评论表示您已同意该条款