Standalone Faster-Whisper-XXL features

[Purfview]

Includes all Standalone Faster-Whisper features +the additional ones mentioned below.
Includes all needed libs.

Vocal extraction model:

--ff_mdx_kim2: Preprocess audio with MDX23 Kim vocal v2 model (thanks to Kimberley Jensen). [Better than HT Demucs v4 FT]

Alternative VAD (Voice activity detection) methods:

--vad_alt_method

Alternative choices:
silero_v3 - Generally less accurate than v4, but doesn't have some quirks of v4.
silero_v4 - Same model as used originally, except that it runs original Silero's code instead of adapted one.
pyannote_v3 - The best accuracy.
pyannote_onnx_v3 - Similar performance to silero.
webrtc - Low accuracy, outdated VAD. Takes only 'vad_min_speech_duration_ms' & 'vad_speech_pad_ms'.
auditok - Actually it's not VAD, it's AAD - Audio Activity Detection.

Latest CTranslate2:

Up to ~26% faster on CPU with the int8 quantizations.
Flash attention support, that's CUDA, but the benchmarks shows no effect on the performance.

[despairTK]

I really like the new parameter --vad_alt_method. Among them, silero_v3/silero_v4/pyannote_onnx_v3 are much better than the original VAD.

For example, there will be some gaps in the original VAD, and for example, sentences starting with "So" will often have a delayed start of the timeline. These issues are resolved in the silero_v3/silero_v4/pyannote_onnx_v3 parameters.

Finally, let me ask, which of the three parameters of silero_v3/silero_v4/pyannote_onnx_v3 has the best test results? Or what are their characteristics?

[Purfview]

First why I've added alternatives, because Silero v4 has problems like false-positive detection on nearly silent parts and other imperfections:
snakers4/silero-vad#396
snakers4/silero-vad#369

...sentences starting with "So" will often have a delayed start of the timeline

Did you evaluate that looking at a transcription?

I really like the new parameter --vad_alt_method. Among them, silero_v3/silero_v4/pyannote_v3 are much better than the original VAD.

Actually the original VAD runs exactly same model as "silero_v4". 😌
"silero_v4" runs original code from silero's repo, when the original VAD runs the same model with adapted code.

Finally, let me ask, which of the three parameters of silero_v3/silero_v4/pyannote_onnx_v3 has the best test results?

Probably neither of them are best, I not tested them much yet.
Did one quick test with "pyannote_onnx_v3", for my disappointment it was less accurate than original, it missed few segments.

[despairTK]

Well...after replying to you, I transcribed dozens of audios, ranging from 10 minutes to 2 hours, and the audio language is English. After comparing the transcription results, I think silero_v3 is the best choice.

[Purfview]

After comparing the transcription results,

That's not how VAD evaluation works, you are looking at whisper's randomness not at VAD's accuracy.

To evaluate VAD you need to look only at VAD's timestamps, you can get them with --vad_dump, then those can be loaded in SE and looked on the waveform.

And VAD params needs to be adjusted so you can see accuracy better, like vad_speech_pad_ms=0 and vad_min_silence_duration_ms=0.
And the same threshold doesn't convert to a same responsiveness on the different methods.

[tristan-mcinnis]

any hope of doing something similar for mac in the future?

[Purfview]

Probably there won't be macOS builds for this.

[dgoryeo]

I guess no chance of a python package either?

[Purfview]

I guess no chance of a python package either?

Yeap, it's out of the scope of this repo.

[tristan-mcinnis]

Sorry, saw a comment here earlier about 'why post' this here but now its gone -- I was trying to keep it in the thread of this 'XXL' version. For example, if there can be an XXL version why not as if there will be a '-MAC' version in the future... who knows - just asking :)

Separately, for others who may find this thread and dismayed at no mac support there is a lot of work being done with 'whisperkit' which also does great work. https://github.com/argmaxinc/WhisperKit

The future of high-quality transcriptions is very bright.

[aehlke]

WhisperKit has no plans to add VAD though. I plan to do it using a flutter port I found of silero

[Sonnenfleck]

A little annoyance with --ff_mdx_kim2 in r192.3.3 [XXL], which otherwise works great (well, after I realized I had to change my scripts because it's now faster-whisper-xxl.exe instead of whisper-faster-xxl.exe 😉 ):

I'm running Faster-Whisper-XXL in a Nextcloud folder (with a cronjob checking if new audio files have been synchronized, then running faster-whisper-xxl). So far, this worked fine, but in r192.3.3 with MDX filtering enabled, it seems first the *_mdx.wav file is created and then it's moved to a temp folder (?). This move fails because Nextcloud already tries to sync the mdx file, and this leads to whisper-faster-xxl just quitting with an error that the *_mdx.wav file is already in use.

I now set the Nextcloud rules to just ignore *_mdx.wav files, but would it be possible to create them in a temp folder from the start?

[Purfview]

it seems first the *_mdx.wav file is created and then it's moved to a temp folder (?)

It's intermediate file, first it's written to the same folder as the original input file then it's deleted. No "movements" of it.
If Nextcloud app tries to do anything with it then of course deletion fails.

would it be possible to create them in a temp folder

Should be possible.

[Sonnenfleck]

I'm confused: If the mdx file is deleted before transcription starts, then what file is whisper running on?
At least as I observe it, the problem here is: As soon as mdx filtering is finished, Nextcloud grabs the _mdx.wav to sync it, but before it finished syncing (which would be just a few seconds later), faster-whisper tries to delete the _mdx.wav but can't, because it's in use. Shouldn't the intermediate file be deleted after transcription? Or does VAD create another intermediate file (if yes, then it's not visible in the same folder)?

[Purfview]

I'm confused: If the mdx file is deleted before transcription starts, then what file is whisper running on?

The intermediate file is loaded into variables further and it's not needed in the physical form anymore.
Yesterday I've made a quick patch to use a temp folder but not tested it yet, I'll try to release it today.

Or does VAD create another intermediate file (if yes, then it's not visible in the same folder)?

VAD's "intermediate" is created only in memory.

[Sonnenfleck]

Ah, I get it, thanks for clarifying and for implementing the patch. :)

[Purfview]

Implemented in r192.3.4

[Herzfrequenz21]

Do I need to use some kind of tag to make the recognition against a little noise or soft music better?

[dgoryeo]

You're right, I thought the memory error was at the point of vocal isolation.

[Herzfrequenz21]

This has nothing to do with the audio length.

How can I make the memory be used from the CPU as before?
This line now specifies CUDA:
Standalone Faster-Whisper-XXL r192.1.1.1 running on: CUDA

[Purfview]

Use --device cpu

[1001ruchka]

This has nothing to do with the audio length.

How can I make the memory be used from the CPU as before? This line now specifies CUDA: Standalone Faster-Whisper-XXL r192.1.1.1 running on: CUDA

Try using CUDA with int8:
--compute_type=int8_float16 or --compute_type=int8
With these parameters I managed to run large_v2 on 4GB of video memory.

for Russian:

-prompt default

faster-whisper-xxl.exe --language ru --model "large-v2" --compute_type=int8_float16 --sentence -prompt default --beep_off --print_progress --vad_alt_method pyannote_v3 --ff_mdx_kim2 --mdx_device cpu "video.mp4"

or

-prompt None

faster-whisper-xxl.exe --language ru --model "large-v2" --compute_type=int8_float16 --sentence -prompt None --beep_off --print_progress --vad_alt_method pyannote_v3 --ff_mdx_kim2 --mdx_device cpu "video.mp4"

or

-prompt auto

faster-whisper-xxl.exe --language ru --model "large-v2" --compute_type=int8_float16 --sentence -prompt auto --beep_off --print_progress --vad_alt_method pyannote_v3 --ff_mdx_kim2 --mdx_device cpu "video.mp4"

[Herzfrequenz21]

Use --device cpu

It's taking me a long time. I've never been able to wait.
I haven't tried the option suggested below yet.

[koebbe14]

Such a great tool. Especially for those who aren't very saavy in Python or command line! Thanks for creating!

Is it possible to perform speaker diarization with this standalone version?

[koebbe14]

I would throw in $50

[dgoryeo]

You can count on $20 from me as well.

[Purfview]

A new thread would be better place for this -> #281
Make a donation and post there, or I'll update there with the amount donated.

...posting PayPal addresses or other personal information might not be so clever...

If you don't trust me with your email "address" then how you can trust me running the program?! 😉

[thbaero]

I'll do the rest of it to make the 100£ ;)
Thanks for the others to chip in.

Let's continue on the other thread. #281

[thbaero]

@dgoryeo - only 10£ will be needed in the end

[MahmoudAshraf97]

Hey @Purfview , I was wondering if you have (or willing to run) any benchmarks that compare pyannote_v3 with silero_v5
Thanks

[rodnvs]

Hi @Purfview, I did a test with the --ff_mdx_kim2 feature and it took a long time to complete, about 45min for a 10min video. Is the voice extraction feature processed using the GPU, or CPU?

[1001ruchka]

If you have an NVIDIA graphics card with CUDA support and you didn't specify --mdx_device cpu - CUDA should be used. On my NVIDIA 3060 12GB, processing ff_mdx_kim2 for a 90 minute video takes no more than 3-5 minutes. Please note ff_mdx_kim2 is quite demanding on video memory, my video memory usage is over 8GB during ff_mdx_kim2.

[rodnvs]

Yep, I forgot about it, my bad.
Here are the results from a new test using the GPU, on a RTX 3050 (laptop):
The UVR5 still very fast compared to the built in extractor. Any thoughts?

>> Transcribe only
Command: faster-whisper-xxl.exe test_video.mp4 --language es --task transcribe --model medium --compute_type auto --device cuda --standard --print_progress --vad_alt_method pyannote_v3 --vad_min_silence_duration_ms 2000
Start Time: 2024-07-31 11:40:32.919825
End Time: 2024-07-31 11:42:43.851822
Duration: 2m10s
Result: Successful
-------------------------------------------------
>> Voice extraction + Transcribe
Command: faster-whisper-xxl.exe test_video.mp4 --language es --task transcribe --model medium --compute_type auto --device cuda --standard --print_progress --vad_alt_method pyannote_v3 --vad_min_silence_duration_ms 2000 --ff_mdx_kim2 --mdx_device cuda
Start Time: 2024-07-31 11:45:37.965749
End Time: 2024-07-31 12:01:47.287552
Duration: 16m09s
Result: Successful
-------------------------------------------------
>> UVR5 Python script voice extraction only (https://github.com/nomadkaraoke/python-audio-separator)
Command: audio-separator test_video.mp4 --model_filename Kim_Vocal_2.onnx --output_format wav --single_stem Vocals
Start Time: 2024-07-31 14:29:06.907680
End Time: 2024-07-31 14:30:15.179309
Duration: 1m08s
Result: Successful

[Purfview]

about 45min for a 10min video

That's too slow for GPU. Try --mdx_device cpu in your benchmark test.
EDIT: I think that's too slow even for CPU.

RTX 3050 (laptop)

How much VRAM?

Is the voice extraction feature processed using the GPU, or CPU?

It can use both.

[Xylemm]

Is there a series of parameters that work best to capture very short audio clips?

My clips with just "Yes" or "Let's go" produce a blank transcription. I've adjusted --vad_min_speech_duration_ms and others, but nothing catches these short clips.

[dgoryeo]

Which vad method do you use? I believe Auditok would perform best for your scenario.
In addition, if your clips are super short (say less than 15 sec), I think you should disable VAD.

[Xylemm]

I fixed this with --vad_filter=false

Now it perfectly transcribes every file down to about 1 or 2 seconds long.

Thank you!

[JustAndreww]

Is there any way to make auto dialogs to work?
Quite often I see following lines:

<Time1>: Hello.
<Time2>: Hi.

instead of

<Time1>:- Hello.</br>
- Hi.

Thanks!

[Ad4mts]

Since this faster Whisper model has been modified from the original version, could you please upload the source code so the community can contribute and add new features or im i missing something? Thanks!

[Purfview]

or im i missing something?

I think so, this is not the repo to modify anything, go directly to faster-whisper repo, fork it and modify/add new features there whatever you like,