Bert-vits2-v2.2新版本本地训练推理整合包(原神八重神子英文模型miko)

在这里插入图片描述

近日,Bert-vits2-v2.2如约更新,该新版本v2.2主要把Emotion 模型换用CLAP多模态模型,推理支持输入text prompt提示词和audio prompt提示语音来进行引导风格化合成,让推理音色更具情感特色,并且推出了新的预处理webuI,操作上更加亲民和接地气。

更多情报请参见Bert-vits2官网:

https://github.com/fishaudio/Bert-VITS2/releases/tag/v2.2

与此同时,基于FastApi的推理web界面项目也同步适配了Bert-vits2-v2.2版本,官网如下:

https://github.com/jiangyuxiaoxiao/Bert-VITS2-UI

本次我们基于此两个项目来克隆原神角色八重神子的英文语音模型miko。

Bert-vits2-v2.2新的底模和情感模型

首先克隆Bert-vits2-v2.2官方项目:

git clone https://github.com/fishaudio/Bert-VITS2/tree/v2.2

安装依赖:

pip3 install -r requirements.txt

这里注意是v2.2的tag分支,因为官方随时都在更新,主分支可能会存在bug。

进入项目的目录:

cd /Bert-VITS2

随后下载新的底模和情感模型,下载地址:

https://openi.pcl.ac.cn/Stardust_minus/Bert-VITS2/modelmanage/show_model

将新的情感模型clap-hatsat-fused放入到项目的emotional目录,结构如下:

E:\work\Bert-VITS2-v22\emotional>tree /f  
Folder PATH listing for volume myssd  
Volume serial number is 7CE3-15AE  
E:.  
├───clap-htsat-fused  
│       .gitattributes  
│       config.json  
│       merges.txt  
│       preprocessor_config.json  
│       pytorch_model.bin  
│       README.md  
│       special_tokens_map.json  
│       tokenizer.json  
│       tokenizer_config.json  
│       vocab.json  
│  
└───wav2vec2-large-robust-12-ft-emotion-msp-dim  .gitattributes  config.json  LICENSE  preprocessor_config.json  pytorch_model.bin  README.md  vocab.json

注意,wav2vec2-large-robust-12-ft-emotion-msp-dim是Bert-vits2-v2.1的情感模型,也需要保留,具体请移步:义无反顾马督工,Bert-vits2V210复刻马督工实践(Python3.10), 这里不再赘述。

至此,新模型就配置好了。

Bert-vits2-v2.2模型训练

首先下载训练集,以原神角色八重神子的英文配音为例子,数据集下载地址:

https://github.com/AI-Hobbyist/Genshin_Datasets

随后新建miko角色目录

mkdir miko

将语音标注文件以esd.list命名,放入miko目录。

同时将分片语音素材放入raw目录。

最后新建miko/configs/config.json配置文件:

{  "train": {  "log_interval": 50,  "eval_interval": 50,  "seed": 42,  "epochs": 1000,  "learning_rate": 0.0002,  "betas": [  0.8,  0.99  ],  "eps": 1e-09,  "batch_size": 6,  "fp16_run": false,  "lr_decay": 0.99995,  "segment_size": 16384,  "init_lr_ratio": 1,  "warmup_epochs": 0,  "c_mel": 45,  "c_kl": 1.0,  "skip_optimizer": false,  "freeze_ZH_bert": false,  "freeze_JP_bert": false,  "freeze_EN_bert": false  },  "data": {  "training_files": "data/miko/train.list",  "validation_files": "data/miko/val.list",  "max_wav_value": 32768.0,  "sampling_rate": 44100,  "filter_length": 2048,  "hop_length": 512,  "win_length": 2048,  "n_mel_channels": 128,  "mel_fmin": 0.0,  "mel_fmax": null,  "add_blank": true,  "n_speakers": 1,  "cleaned_text": true,  "spk2id": {  "miko": 0  }  },  "model": {  "use_spk_conditioned_encoder": true,  "use_noise_scaled_mas": true,  "use_mel_posterior_encoder": false,  "use_duration_discriminator": true,  "inter_channels": 192,  "hidden_channels": 192,  "filter_channels": 768,  "n_heads": 2,  "n_layers": 6,  "kernel_size": 3,  "p_dropout": 0.1,  "resblock": "1",  "resblock_kernel_sizes": [  3,  7,  11  ],  "resblock_dilation_sizes": [  [  1,  3,  5  ],  [  1,  3,  5  ],  [  1,  3,  5  ]  ],  "upsample_rates": [  8,  8,  2,  2,  2  ],  "upsample_initial_channel": 512,  "upsample_kernel_sizes": [  16,  16,  8,  2,  2  ],  "n_layers_q": 3,  "use_spectral_norm": false,  "gin_channels": 256  },  "version": "2.2"  
}

这里注意"version": “2.2”,即版本号为最新的v2.2。

其他参数根据当前的设备环境酌情调整即可。

随后启动预处理页面:

python3 webui_preprocess.py

访问http://127.0.0.1:7860/:

按照页面的步骤进行操作即可,简单且方便。

操作完之后,运行训练命令:

python3 train_ms.py

训练好的模型放在data/miko/models目录,结构如下:

E:\work\Bert-VITS2-v22\Data\miko\models>tree /f  
Folder PATH listing for volume myssd  
Volume serial number is 7CE3-15AE  
E:.  
│   DUR_0.pth  
│   DUR_100.pth  
│   DUR_150.pth  
│   DUR_50.pth  
│   D_0.pth  
│   D_100.pth  
│   D_150.pth  
│   D_50.pth  
│   events.out.tfevents.1702457087.ly.13044.0  
│   events.out.tfevents.1702458207.ly.12416.0  
│   githash  
│   G_0.pth  
│   G_100.pth  
│   G_150.pth  
│   G_50.pth  
│   train.log  
│  
└───eval  events.out.tfevents.1702457087.ly.13044.1  events.out.tfevents.1702458207.ly.12416.1

至此,训练环节结束。

Bert-vits2-v2.2模型推理

推理我们使用Bert-vits2-UI项目的页面,克隆web项目:

git clone https://github.com/jiangyuxiaoxiao/Bert-VITS2-UI

将Web项目放入Bert-vits2-v2.2的根目录中,目录结构如下:

E:\work\Bert-VITS2-v22_lilith\Web>tree /f  
Folder PATH listing for volume myssd  
Volume serial number is 7CE3-15AE  
E:.  
│   index.html  
│  
├───assets  
│       index-21bc6a28.css  
│       index-402c0217.js  
│  
└───img  helps1.png  helps2.png  Hiyori.ico

这里包含主页面、样式文件以及JS文件,基于Hiyori。

随后启动推理页面:

python3 server_fastapi.py

访问:http://127.0.0.1:5000/:

加载模型进行推理即可。

此外,还可以基于FastAPI的接口进行推理,换句话说,发送http请求即可获取推理音频,接口参数如下:

{  "openapi": "3.1.0",  "info": {  "title": "FastAPI",  "version": "0.1.0"  },  "paths": {  "/": {  "get": {  "summary": "Index",  "operationId": "index__get",  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  }  }  }  },  "/voice": {  "post": {  "summary": "Voice",  "description": "语音接口,若需要上传参考音频请仅使用post请求",  "operationId": "voice_voice_post",  "parameters": [  {  "name": "model_id",  "in": "query",  "required": true,  "schema": {  "type": "integer",  "description": "模型ID",  "title": "Model Id"  },  "description": "模型ID"  },  {  "name": "speaker_name",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "说话人名",  "title": "Speaker Name"  },  "description": "说话人名"  },  {  "name": "speaker_id",  "in": "query",  "required": false,  "schema": {  "type": "integer",  "description": "说话人id,与speaker_name二选一",  "title": "Speaker Id"  },  "description": "说话人id,与speaker_name二选一"  },  {  "name": "sdp_ratio",  "in": "query",  "required": false,  "schema": {  "type": "number",  "description": "SDP/DP混合比",  "default": 0.2,  "title": "Sdp Ratio"  },  "description": "SDP/DP混合比"  },  {  "name": "noise",  "in": "query",  "required": false,  "schema": {  "type": "number",  "description": "感情",  "default": 0.2,  "title": "Noise"  },  "description": "感情"  },  {  "name": "noisew",  "in": "query",  "required": false,  "schema": {  "type": "number",  "description": "音素长度",  "default": 0.9,  "title": "Noisew"  },  "description": "音素长度"  },  {  "name": "length",  "in": "query",  "required": false,  "schema": {  "type": "number",  "description": "语速",  "default": 1,  "title": "Length"  },  "description": "语速"  },  {  "name": "language",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "语言",  "title": "Language"  },  "description": "语言"  },  {  "name": "auto_translate",  "in": "query",  "required": false,  "schema": {  "type": "boolean",  "description": "自动翻译",  "default": false,  "title": "Auto Translate"  },  "description": "自动翻译"  },  {  "name": "auto_split",  "in": "query",  "required": false,  "schema": {  "type": "boolean",  "description": "自动切分",  "default": false,  "title": "Auto Split"  },  "description": "自动切分"  },  {  "name": "emotion",  "in": "query",  "required": false,  "schema": {  "anyOf": [  {  "type": "integer"  },  {  "type": "string"  },  {  "type": "null"  }  ],  "description": "emo",  "title": "Emotion"  },  "description": "emo"  }  ],  "requestBody": {  "required": true,  "content": {  "multipart/form-data": {  "schema": {  "$ref": "#/components/schemas/Body_voice_voice_post"  }  }  }  },  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  },  "get": {  "summary": "Voice",  "description": "语音接口",  "operationId": "voice_voice_get",  "parameters": [  {  "name": "text",  "in": "query",  "required": true,  "schema": {  "type": "string",  "description": "输入文字",  "title": "Text"  },  "description": "输入文字"  },  {  "name": "model_id",  "in": "query",  "required": true,  "schema": {  "type": "integer",  "description": "模型ID",  "title": "Model Id"  },  "description": "模型ID"  },  {  "name": "speaker_name",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "说话人名",  "title": "Speaker Name"  },  "description": "说话人名"  },  {  "name": "speaker_id",  "in": "query",  "required": false,  "schema": {  "type": "integer",  "description": "说话人id,与speaker_name二选一",  "title": "Speaker Id"  },  "description": "说话人id,与speaker_name二选一"  },  {  "name": "sdp_ratio",  "in": "query",  "required": false,  "schema": {  "type": "number",  "description": "SDP/DP混合比",  "default": 0.2,  "title": "Sdp Ratio"  },  "description": "SDP/DP混合比"  },  {  "name": "noise",  "in": "query",  "required": false,  "schema": {  "type": "number",  "description": "感情",  "default": 0.2,  "title": "Noise"  },  "description": "感情"  },  {  "name": "noisew",  "in": "query",  "required": false,  "schema": {  "type": "number",  "description": "音素长度",  "default": 0.9,  "title": "Noisew"  },  "description": "音素长度"  },  {  "name": "length",  "in": "query",  "required": false,  "schema": {  "type": "number",  "description": "语速",  "default": 1,  "title": "Length"  },  "description": "语速"  },  {  "name": "language",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "语言",  "title": "Language"  },  "description": "语言"  },  {  "name": "auto_translate",  "in": "query",  "required": false,  "schema": {  "type": "boolean",  "description": "自动翻译",  "default": false,  "title": "Auto Translate"  },  "description": "自动翻译"  },  {  "name": "auto_split",  "in": "query",  "required": false,  "schema": {  "type": "boolean",  "description": "自动切分",  "default": false,  "title": "Auto Split"  },  "description": "自动切分"  },  {  "name": "emotion",  "in": "query",  "required": false,  "schema": {  "anyOf": [  {  "type": "integer"  },  {  "type": "string"  },  {  "type": "null"  }  ],  "description": "emo",  "title": "Emotion"  },  "description": "emo"  }  ],  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  }  },  "/models/info": {  "get": {  "summary": "Get Loaded Models Info",  "description": "获取已加载模型信息",  "operationId": "get_loaded_models_info_models_info_get",  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  }  }  }  },  "/models/delete": {  "get": {  "summary": "Delete Model",  "description": "删除指定模型",  "operationId": "delete_model_models_delete_get",  "parameters": [  {  "name": "model_id",  "in": "query",  "required": true,  "schema": {  "type": "integer",  "description": "删除模型id",  "title": "Model Id"  },  "description": "删除模型id"  }  ],  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  }  },  "/models/add": {  "get": {  "summary": "Add Model",  "description": "添加指定模型:允许重复添加相同路径模型,且不重复占用内存",  "operationId": "add_model_models_add_get",  "parameters": [  {  "name": "model_path",  "in": "query",  "required": true,  "schema": {  "type": "string",  "description": "添加模型路径",  "title": "Model Path"  },  "description": "添加模型路径"  },  {  "name": "config_path",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "添加模型配置文件路径,不填则使用./config.json或../config.json",  "title": "Config Path"  },  "description": "添加模型配置文件路径,不填则使用./config.json或../config.json"  },  {  "name": "device",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "推理使用设备",  "default": "cuda",  "title": "Device"  },  "description": "推理使用设备"  },  {  "name": "language",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "模型默认语言",  "default": "ZH",  "title": "Language"  },  "description": "模型默认语言"  }  ],  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  }  },  "/models/get_unloaded": {  "get": {  "summary": "Get Unloaded Models Info",  "description": "获取未加载模型",  "operationId": "get_unloaded_models_info_models_get_unloaded_get",  "parameters": [  {  "name": "root_dir",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "搜索根目录",  "default": "Data",  "title": "Root Dir"  },  "description": "搜索根目录"  }  ],  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  }  },  "/models/get_local": {  "get": {  "summary": "Get Local Models Info",  "description": "获取全部本地模型",  "operationId": "get_local_models_info_models_get_local_get",  "parameters": [  {  "name": "root_dir",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "搜索根目录",  "default": "Data",  "title": "Root Dir"  },  "description": "搜索根目录"  }  ],  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  }  },  "/status": {  "get": {  "summary": "Get Status",  "description": "获取电脑运行状态",  "operationId": "get_status_status_get",  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  }  }  }  },  "/tools/translate": {  "get": {  "summary": "Translate",  "description": "翻译",  "operationId": "translate_tools_translate_get",  "parameters": [  {  "name": "texts",  "in": "query",  "required": true,  "schema": {  "type": "string",  "description": "待翻译文本",  "title": "Texts"  },  "description": "待翻译文本"  },  {  "name": "to_language",  "in": "query",  "required": true,  "schema": {  "type": "string",  "description": "翻译目标语言",  "title": "To Language"  },  "description": "翻译目标语言"  }  ],  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  }  },  "/tools/random_example": {  "get": {  "summary": "Random Example",  "description": "获取一个随机音频+文本,用于对比,音频会从本地目录随机选择。",  "operationId": "random_example_tools_random_example_get",  "parameters": [  {  "name": "language",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "指定语言,未指定则随机返回",  "title": "Language"  },  "description": "指定语言,未指定则随机返回"  },  {  "name": "root_dir",  "in": "query",  "required": false,  "schema": {  "type": "string",  "description": "搜索根目录",  "default": "Data",  "title": "Root Dir"  },  "description": "搜索根目录"  }  ],  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  }  },  "/tools/get_audio": {  "get": {  "summary": "Get Audio",  "operationId": "get_audio_tools_get_audio_get",  "parameters": [  {  "name": "path",  "in": "query",  "required": true,  "schema": {  "type": "string",  "description": "本地音频路径",  "title": "Path"  },  "description": "本地音频路径"  }  ],  "responses": {  "200": {  "description": "Successful Response",  "content": {  "application/json": {  "schema": {}  }  }  },  "422": {  "description": "Validation Error",  "content": {  "application/json": {  "schema": {  "$ref": "#/components/schemas/HTTPValidationError"  }  }  }  }  }  }  }  },  "components": {  "schemas": {  "Body_voice_voice_post": {  "properties": {  "text": {  "type": "string",  "title": "Text"  },  "reference_audio": {  "type": "string",  "format": "binary",  "title": "Reference Audio"  }  },  "type": "object",  "required": [  "text"  ],  "title": "Body_voice_voice_post"  },  "HTTPValidationError": {  "properties": {  "detail": {  "items": {  "$ref": "#/components/schemas/ValidationError"  },  "type": "array",  "title": "Detail"  }  },  "type": "object",  "title": "HTTPValidationError"  },  "ValidationError": {  "properties": {  "loc": {  "items": {  "anyOf": [  {  "type": "string"  },  {  "type": "integer"  }  ]  },  "type": "array",  "title": "Location"  },  "msg": {  "type": "string",  "title": "Message"  },  "type": {  "type": "string",  "title": "Error Type"  }  },  "type": "object",  "required": [  "loc",  "msg",  "type"  ],  "title": "ValidationError"  }  }  }  
}

最后奉上Bert-vits2-v2.2本地训练推理整合包:

https://pan.baidu.com/s/1OVX9seRwZR6bZ-xsE_nRLg?pwd=v3uc

与众乡亲同飨。

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.hqwc.cn/news/288327.html

如若内容造成侵权/违法违规/事实不符,请联系编程知识网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

SpringIOC之BeanExpressionContextAccessor

博主介绍:✌全网粉丝5W,全栈开发工程师,从事多年软件开发,在大厂呆过。持有软件中级、六级等证书。可提供微服务项目搭建与毕业项目实战,博主也曾写过优秀论文,查重率极低,在这方面有丰富的经验…

Ansible(一)

Ansible: 远程操作主机功能: 自动化运维(playbook剧本YAML) 是基于Python开发的配置管理应用部署攻具,在自动化运维当中,现在是异军突起 Ansible能批量配置,部署,管理上千台主机&#xff0c…

kafka offset sasl加密连接

kafka-tool(offset) 进行SCRAM连接,直接上图 填写jaas的认证(账密 引用包)

pytest--allure报告中添加用例详情

前言 前面介绍了如何生成allure的报告,看着allure的页面非常好看,但是感觉少了一些内容,allure还可以增加一些用例详情内容,这样让我们的报告看着更加绚丽。 allure增加用例详情 我们可以在报告测试套件中增加用例详情内容。 …

【简洁易懂】为什么绝对收敛的级数一定条件收敛?【考研向】

针对某高热帖子真的想说,什么时候讨论知识的戾气不那么重了,什么时候能把知识深入浅出的讲明白了,什么时候我们才能有希望吧。 图源:https://www.youtube.com/watch?v3ShPTvJUFrU

一文解决idea导入源码控制台爆红问题

文章目录 唠嗑部分背景说明idea查看maven配置 言归正传安装mavenidea配置maven 结语及资料获取 唠嗑部分 背景说明 很多新手伙伴们在导入项目源码时,都会遇到大片依赖爆红,项目跑不起来,小白也是把自己电脑重新配置了一番,复现了…

短视频时代,又恰逢双旦来临之际,普通人又该如何立足?

我是电商珠珠 在电商发展迅速的同时,短视频也在同步发展。国内的短视频比较热门的有抖音、视频号、快手这几个。 抖音在19年的时候发展起了自己的电商行业-抖音小店,并顺势掀起了直播电商的热潮。 直播电商在短视频中很火,所以很多人都选择…

MySQL——表的增删查改

目录 一.Create(创建) 1.单行数据 全列插入 2.多行数据 指定列插入 3.插入否则更新 4. 替换 二.Retrieve(读取) 1. select 列 查询 2.where 条件 3.结果排序 4.筛选分页结果 三.Update (修改)…

《PySpark大数据分析实战》-15.云服务模式Databricks介绍创建集群

📋 博主简介 💖 作者简介:大家好,我是wux_labs。😜 热衷于各种主流技术,热爱数据科学、机器学习、云计算、人工智能。 通过了TiDB数据库专员(PCTA)、TiDB数据库专家(PCTP…

矩阵式键盘实现的电子密码锁

#include<reg51.h> //包含51单片机寄存器定义的头文件 sbit P14P1^4; //将P14位定义为P1.4引脚 sbit P15P1^5; //将P15位定义为P1.5引脚 sbit P16P1^6; //将P16位定义为P1.6引脚 sbit P17P1^7; //将P17位定义为P1.7引脚 sbit soundP3^7; //将so…

英码科技受邀参加2023计算产业生态大会,分享智慧轨道交通创新解决方案

12月13-14日&#xff0c;“凝心聚力&#xff0c;共赢计算新时代”——2023计算产业生态大会在北京香格里拉饭店成功举办。英码科技受邀参加行业数字化分论坛活动&#xff0c;市场总监李甘来先生现场发表了题为《AI哨兵&#xff0c;为铁路安全运营站好第一道岗》的精彩主题演讲&…

ROS学习笔记(七)---参数服务器

ROS学习笔记文章目录 01. ROS学习笔记(一)—Linux安装VScode 02. ROS学习笔记(二)—使用 VScode 开发 ROS 的Python程序&#xff08;简例&#xff09; 03. ROS学习笔记(三)—好用的终端Terminator 04. ROS学习笔记(四)—使用 VScode 启动launch文件运行多个节点 05. ROS学习笔…