Whisper API Server: Setup Options and the Text Step

The short answer

Whisper is an open-source speech recognition model that OpenAI published. On its own it is a model file and some inference code. A server puts an HTTP interface in front of it, so your app can send an audio file and get text back instead of calling Python functions directly.

Most servers copy the shape of the OpenAI transcription endpoint, POST /v1/audio/transcriptions, which takes a multipart upload with the audio file and a model name. That shape is why the same client library can talk to a hosted service, a container on your own machine, or a small wrapper someone else wrote.

• Input: an audio file, usually mp3, wav, m4a or similar.
• Output: plain text, or JSON with segments and timestamps, depending on the server.
• Not included: summaries, translation into polished prose, extraction of action items. Those need a language model.

Three ways to get a Whisper server

1. A hosted endpoint

OpenAI offers a hosted transcription endpoint, and other providers sell Whisper-based transcription as well. You pay per use, you do not manage hardware, and you send your audio to someone else's infrastructure. Billing models differ between providers, and speech services commonly charge by audio duration, so check their current pricing before you commit to a volume.

2. A self-hosted wrapper

Several open-source projects wrap Whisper or reimplementations of it in a web server that mimics the OpenAI transcription route. You run it in Docker on a machine with a GPU, or on CPU if you accept slower processing. This gives you control over where audio goes. It also makes you responsible for model downloads, updates, queueing and restarts.

3. A local binary

The whisper.cpp project is a C and C++ port of the model, and its repository includes an example HTTP server. It runs on ordinary laptops and small servers, which makes it a good choice for a prototype or for a single user who transcribes a few recordings a day. For batch jobs of hundreds of hours, a GPU-backed service is usually more practical.

Pick based on three questions: where may the audio travel, how many hours do you process per month, and who will keep the service running. The text step described below works the same way with any of the three.

Where a text LLM API fits after transcription

A raw Whisper transcript is a wall of text. It has few paragraph breaks, speaker names are missing unless you add diarization separately, and filler words stay in. Whisper also makes mistakes with product names, jargon and proper nouns. Almost every real project adds a second stage that works on the text.

Typical second-stage jobs:

• Fix punctuation, split into paragraphs, remove false starts.
• Summarize a call or lecture into a few bullets.
• Pull out structured data such as decisions, names and dates as JSON.
• Rewrite spoken material into an article, a script or show notes.
• Translate or localize the cleaned text.

That is where this service fits. It is an OpenAI-compatible text API with one model, uncensored, a 100,000-token context window and streaming. A two-hour interview transcript often fits in a single request, so you can skip the chunking logic that smaller context windows force on you. For longer material see the quickstart docs for request details.

The model is tuned to answer without content refusals for lawful adult use. That matters for transcription pipelines because source audio is unpredictable: interviews, podcasts, fiction readings and support calls contain strong language and difficult subjects. A model that refuses to clean up a transcript because the speaker swore wastes a paid transcription run. The one hard limit is that nothing involving sexual content with minors is accepted.

A working two-stage pipeline

The OpenAI Python SDK lets you create two clients with different base URLs. One points at your Whisper server, the other at this API. Set the environment variables for your own transcription service; the second client uses the base URL for this site.

Copyimport os
from openai import OpenAI

# Stage 1: any Whisper-compatible transcription server you run or rent.
stt = OpenAI(base_url=os.environ["STT_BASE_URL"], api_key=os.environ["STT_KEY"])

# Stage 2: the text model.
llm = OpenAI(base_url="https://api.whisperapis.com/v1", api_key=os.environ["LLM_KEY"])

with open("call.mp3", "rb") as f:
    transcript = stt.audio.transcriptions.create(
        model=os.environ["STT_MODEL"],  # the model name your server expects
        file=f,
    )

resp = llm.chat.completions.create(
    model="uncensored",
    messages=[
        {"role": "system", "content": "Clean up this transcript: fix punctuation, keep every statement, no commentary."},
        {"role": "user", "content": transcript.text},
    ],
    temperature=0.2,
)
print(resp.choices[0].message.content)

Two details are easy to get wrong. First, the transcription model name is whatever your server expects, which is why it comes from an environment variable here. Second, the key for the text API is separate from any key you hold for the speech service; each one authenticates only its own host.

If you prefer a shell, the text stage is a plain chat completion request:

Copycurl https://api.whisperapis.com/v1/chat/completions \
  -H "Authorization: Bearer $LLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "uncensored",
    "messages": [
      {"role": "user", "content": "Summarize this transcript in five bullet points: ..."}
    ]
  }'

Authentication is a Bearer key in the Authorization header, and the endpoints are POST /v1/chat/completions and GET /v1/models. You can get a key from the API key page; new accounts can claim a one-click $0.50 trial credit there, valid for 7 days and without a card.

Things that break in practice

Large audio files

Many Whisper servers cap upload size, and long recordings can time out in the middle of a request. Split audio into pieces of ten to thirty minutes with a tool such as ffmpeg, transcribe each piece, then join the text. Because the text API accepts up to 100,000 tokens, you can concatenate the pieces and send them as one prompt. The request body limit on this API is 8 MB, which is far more than any transcript needs.

Rate and concurrency limits

A full key allows 8 concurrent requests and 300 requests per minute; the trial key allows 2 and 60. If you run a queue with many workers, cap it at the concurrency of your key. Going over returns HTTP 429, with the code rate_limited for requests per minute or concurrency for too many at once. A streamed request keeps its slot until the stream finishes, so a stalled client can block a slot longer than you expect. Back off and retry; if the response carries a Retry-After: 5 header, wait that long.

Transcription errors that the LLM cannot see

If Whisper heard the wrong word, the text model only sees the wrong word. It may even smooth over the mistake and make it look intentional. Put a glossary of names and terms in the system prompt, and tell the model to mark uncertain passages instead of guessing. For critical material, keep the original transcript next to the cleaned one.

Hallucinated filler

Whisper sometimes produces invented text during silence or music. Trim silence before sending, and ask the second stage to flag sentences that read as out of place. A low temperature, such as 0.2, keeps the cleanup close to the source.

What this API does not do

It does not transcribe audio. There is no audio input, no speech output, no image or video generation, no embeddings and no fine-tuning. It is a hosted text model behind a chat completions endpoint, run by an independent provider, and it is not the official API of OpenAI or any other model vendor. If your project needs both halves, you need a speech server for the first and this service for the second.

Prompts are not used for training. That is separate from what your speech provider does with the audio, so read that provider's terms on its own.

For cost planning, the text side is billed per token at $0.15 per million input tokens and $0.50 per million output tokens, from prepaid credit that never expires. The pricing page lists top-up sizes and bonuses. Failed or refused requests are not charged, which keeps retry loops from costing you money.

A sensible way to start

1. Get a Whisper-compatible server running, hosted or local, and confirm it returns text for a one-minute file.
2. Create an account, claim the trial credit, and run the curl command above with a short transcript.
3. Wire the two clients together with the Python script, using your own system prompt for the cleanup style you want.
4. Add a queue only after the single-file flow works, and size it to your key's concurrency.

Most of the effort goes into the prompt and the audio handling, not into the glue code. Start with one recording you know well, compare the output with what you heard, and adjust the instructions before you scale up.

More guides

• Whisper API Pricing: What the Text Step Adds
• LLM API Meeting Notes: Transcript to Summary in Python
• Midjourney API: Troubleshooting & Alternatives