WhisperGPT-SoVITS语音识别到音色克隆的全链路实战一、引言语音 AI 正在经历爆发式增长。OpenAI 开源的 Whisper 实现了接近人类的语音识别能力而 GPT-SoVITS 等音色克隆技术让复制一个人的声音成为现实。本文将打通语音全链路Whisper语音识别→微调→流式ASR→GPT-SoVITS音色克隆→端到端语音对话系统。二、Whisper 语音识别2.1 模型选择模型参数显存速度中文识别率tiny39M1GB10x85%base74M1GB7x90%small244M2GB4x94%medium769M5GB2x96%large-v31550M10GB1x97%2.2 基础使用importwhisper modelwhisper.load_model(large-v3)# 1. 语音转文字resultmodel.transcribe(meeting.mp3,languagezh,tasktranscribe,# 或 translate翻译为英文verboseTrue,# 高级参数temperature0.0,beam_size5,best_of5,fp16True,word_timestampsTrue,# 词级别时间戳condition_on_previous_textTrue,initial_prompt这是一场技术会议讨论关于AI和机器学习。)print(f识别结果:{result[text]})print(f语言:{result[language]})# 词级别时间戳forsegmentinresult[segments]:print(f[{segment[start]:.1f}s -{segment[end]:.1f}s]{segment[text]})2.3 中文微调importtorchfromtransformersimport(WhisperForConditionalGeneration,WhisperProcessor,Seq2SeqTrainingArguments,Seq2SeqTrainer)fromdatasetsimportload_dataset,Audio# 微调 Whisper 适应领域术语deffine_tune_whisper(custom_dataset_path,output_dir./whisper-finetuned):model_nameopenai/whisper-large-v3modelWhisperForConditionalGeneration.from_pretrained(model_name)processorWhisperProcessor.from_pretrained(model_name)# 加载自定义数据集datasetload_dataset(json,data_filescustom_dataset_path)defpreprocess(batch):# 加载音频audiobatch[audio]# 提取特征featuresprocessor(audio[array],sampling_rate16000,return_tensorspt).input_features# 编码文本labelsprocessor.tokenizer(batch[text]).input_idsreturn{input_features:features,labels:labels}datasetdataset.map(preprocess)training_argsSeq2SeqTrainingArguments(output_diroutput_dir,per_device_train_batch_size8,gradient_accumulation_steps2,learning_rate1e-5,warmup_steps500,max_steps4000,fp16True,logging_steps50,save_steps500,evaluation_strategysteps,eval_steps500,predict_with_generateTrue,generation_max_length225,)trainerSeq2SeqTrainer(modelmodel,argstraining_args,train_datasetdataset[train],eval_datasetdataset[test],tokenizerprocessor.feature_extractor,)trainer.train()model.save_pretrained(output_dir)processor.save_pretrained(output_dir)2.4 实时流式 ASRimportnumpyasnpimportsounddeviceassdimportqueueimportthreadingimportwhisperclassStreamingASR:实时流式语音识别def__init__(self,model_sizemedium):self.modelwhisper.load_model(model_size)self.audio_queuequeue.Queue()self.sample_rate16000self.is_recordingFalseself.buffernp.array([],dtypenp.float32)defaudio_callback(self,indata,frames,time,status):麦克风回调self.audio_queue.put(indata.copy())defstart_recording(self):开始录制self.is_recordingTrueself.streamsd.InputStream(samplerateself.sample_rate,channels1,callbackself.audio_callback,blocksize1600# 100ms 块)self.stream.start()# 后台转写线程self.transcribe_threadthreading.Thread(targetself._transcribe_loop)self.transcribe_thread.start()def_transcribe_loop(self):持续转写whileself.is_recording:ifnotself.audio_queue.empty():chunkself.audio_queue.get()self.buffernp.concatenate([self.buffer,chunk.flatten()])# 每 3 秒转写一次iflen(self.buffer)3*self.sample_rate:resultself.model.transcribe(self.buffer,languagezh,tasktranscribe)print(f实时转写:{result[text]})# 保留最后 1 秒作为上下文context_lenself.sample_rate self.bufferself.buffer[-context_len:]defstop_recording(self):停止录制self.is_recordingFalseself.stream.stop()self.transcribe_thread.join()asrStreamingASR()asr.start_recording()# ... 说话中 ...asr.stop_recording()三、GPT-SoVITS 语音合成与克隆3.1 环境搭建gitclone https://github.com/RVC-Boss/GPT-SoVITS.gitcdGPT-SoVITS conda create-nsovitspython3.9-yconda activate sovits pipinstall-rrequirements.txt pipinstalltorch torchaudio --index-url https://download.pytorch.org/whl/cu1183.2 音色克隆流程1. 准备参考音频3-10秒清晰录音 2. 音频预处理降噪、音量归一化 3. GPT阶段文本→语义Token 4. SoVITS阶段语义Token参考音频→波形importtorchimporttorchaudioimportnumpyasnpfromTTS_infer_pack.text_segmentationimportsplit_sentencesclassVoiceClone:def__init__(self,gpt_path,sovits_path):self.gpt_modeltorch.load(gpt_path)self.sovits_modeltorch.load(sovits_path)self.sample_rate32000defpreprocess_audio(self,audio_path):音频预处理waveform,srtorchaudio.load(audio_path)# 重采样到32kHzifsr!32000:resamplertorchaudio.transforms.Resample(sr,32000)waveformresampler(waveform)# 去噪waveformtorchaudio.functional.gain(waveform,1.0)returnwaveformdefextract_timbre(self,ref_audio_path,ref_text):从参考音频提取音色特征ref_wavself.preprocess_audio(ref_audio_path)# 音色编码withtorch.no_grad():timbreget_timbre_encoder()(ref_wav)returntimbredefsynthesize(self,text,timbre,ref_audioNone):合成语音# 1. GPT阶段文本→语义Tokensemantic_tokensget_generate_model()(text,timbre,top_k5,top_p1.0,temperature1.0)# 2. SoVITS阶段Token音色→波形ifref_audioisnotNone:audioget_vits_model()(semantic_tokens,ref_audio)else:audioget_vits_model()(semantic_tokens,timbre)returnaudiodeftext_to_speech(self,text,ref_audio_path,output_path):完整TTS流程# 分句sentencessplit_sentences(text)# 提取音色ref_text参考音频对应的文本timbreself.extract_timbre(ref_audio_path,ref_text)ref_wavself.preprocess_audio(ref_audio_path)# 逐句合成audio_segments[]forsentenceinsentences:audioself.synthesize(sentence,timbre,ref_wav)audio_segments.append(audio)# 拼接所有片段full_audiotorch.cat(audio_segments,dim-1)torchaudio.save(output_path,full_audio,self.sample_rate)returnfull_audio四、传统TTS合集4.1 Coqui TTS快速上手importtorchfromTTS.apiimportTTS# 获取可用模型ttsTTS(model_nametts_models/zh-CN/baker/tacotron2-DDC-GST)tts.tts_to_file(text你好欢迎使用语音合成技术。,file_pathoutput.wav)# 多说话人模型tts_multispeakerTTS(tts_models/multilingual/multi-dataset/xtts_v2)tts_multispeaker.tts_to_file(textHello, this is a voice clone demo.,speaker_wavreference.wav,languageen,file_pathcloned_output.wav)4.2 ChatTTS对话风格importChatTTSfromIPython.displayimportAudio chatChatTTS.Chat()chat.load_models()# 带情绪控制的合成texts[哇这个消息太令人惊喜了[laughter],嗯让我想想这个问题怎么回答...,请注意前方路段有事故请减速慢行。]wavschat.infer(texts,use_decoderTrue,params_infer_code{spk_emb:None,# 随机音色temperature:0.3,top_P:0.7,top_K:20,})fori,wavinenumerate(wavs):ChatTTS.tools.save_wav(wav,fchattts_{i}.wav,24000)4.3 Fish Audio最简方案fromfish_audio_sdkimportSession,TTSRequest sessionSession(your-api-key)withopen(output.mp3,wb)asf:forchunkinsession.tts(TTSRequest(text用最自然的语气说出这句话。,reference_idspeaker-id)):f.write(chunk)五、语音到语音对话系统importasyncioimportnumpyasnpclassVoiceChat:语音对话系统ASR → LLM → TTSdef__init__(self):self.asrwhisper.load_model(medium)self.llmChatOpenAI(modelgpt-4)self.ttsTTS(tts_models/multilingual/multi-dataset/xtts_v2)self.speaker_audiomy_voice.wavasyncdefprocess(self,audio_input:np.ndarray)-np.ndarray:# 1. 语音识别resultself.asr.transcribe(audio_input,languagezh)user_textresult[text]print(f 用户:{user_text})# 2. LLM 生成回复responseawaitself.llm.achat(user_text)print(f 助手:{response})# 3. 语音合成wavself.tts.tts(textresponse,speaker_wavself.speaker_audio,languagezh)returnwavasyncdefrealtime_chat(self):实时语音对话streampyaudio.PyAudio().open(formatpyaudio.paInt16,channels1,rate16000,inputTrue,frames_per_buffer1024)whileTrue:# 语音活动检测 (VAD)frames[]silent_count0whilesilent_count30:# 1.5秒静默后停止datastream.read(1024)audio_chunknp.frombuffer(data,dtypenp.int16)# 简单能量检测energynp.sqrt(np.mean(audio_chunk.astype(np.float32)**2))ifenergy500:frames.append(audio_chunk)silent_count0else:ifframes:silent_count1ifframes:audionp.concatenate(frames).astype(np.float32)/32768.0response_audioawaitself.process(audio)# 播放回复sd.play(response_audio,24000)sd.wait()六、主流模型对比特性WhisperGPT-SoVITSChatTTSXTTS v2语音识别⭐⭐⭐⭐⭐❌❌❌音色克隆❌⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐⭐情绪控制❌⭐⭐⭐⭐⭐⭐⭐⭐⭐多语言99中英日中英17语言实时推理✅✅✅✅开源✅✅✅✅七、总结语音AI全链路的核心组件Whisper— 最强开源ASRlarge-v3识别率达97%GPT-SoVITS— 3秒音频即可克隆音色创造性最强ChatTTS— 对话风格合成情绪自然XTTS v2— 多语言支持最好适合国际化场景组合使用可实现完整的语音对话系统Whisper 听 → LLM 想 → GPT-SoVITS 说。