← Ajit Pandey · Blog

Voice AI Call Analytics on a VoIP or PBX System

A business that runs a phone system builds up conversations it may never look at again. A customer explains a problem, a technician promises a callback, a client mentions a competitor, and all of it vanishes into a recording nobody has time to play.

Voice AI call analytics is an attempt to change that: turn calls into text, score them, and make them searchable. This post explains how that works on top of a VoIP or PBX system, what limits the quality of the results, and where it is wise to be careful. It is written for people who run a phone system or a service desk, not for people who want a product demo.

What this looks like in practice

The voice AI work described on the home page is built on 20+ years of telecom and VoIP experience. It describes voice assistants and call analytics that sit on top of VoIP and PBX systems, where calls can be transcribed, scored for sentiment and turned into call intelligence that teams can search and act on, without replacing the phone system they already have.

That last phrase matters. Call analytics is not a new phone system. It is a layer that reads what your existing system already produces.

Two different jobs: assistants and analytics

"Voice AI" covers two quite different things:

For most organisations, analytics is the lower-risk place to start. Nothing the AI produces goes back to the caller, and a person can compare any transcript to the original audio. If you later decide to add a live assistant, you will already know how well the speech recognition performs on your own calls.

The pipeline, step by step

A typical call analytics setup has four steps.

1. Record the call. The phone system has to save the audio. PBX software such as Asterisk gives a useful example of what this involves. The Asterisk documentation for its MixMonitor application describes it as recording a call and mixing the audio during the recording. The same page contains a security warning that applies to any recording setup: never use untrusted strings, such as caller ID, in the application's parameters, because of command injection risks. Caller ID arrives from outside your system, so a recording pipeline that pastes it into a command is a door left open. Treat every value that comes from outside your system as untrusted.

2. Transcribe. A speech-to-text service converts the audio into text. This is the step most sensitive to audio quality, covered below.

3. Analyse. Software reads the transcript and adds structure: a short summary, a topic, a sentiment score, perhaps flags for words or phrases you care about.

4. Make it searchable. The transcripts and labels go into a place where people can search them: "show me calls from this client that mention a recurring fault," or "find the call where we agreed to a callback."

What limits transcription quality

Audio quality sets a limit on any transcript. The documentation for Google Cloud's speech-to-text service gives several concrete points, and they apply to any speech recognition system:

What does this mean for you? Before you buy or build anything, find out how your PBX records calls: at what sample rate, in which format, with how much compression. Those settings set a ceiling on how good your transcripts can be. It is worth testing a handful of your real calls, including the noisy ones, before you commit to a design.

What sentiment scores can and cannot tell you

Sentiment analysis sounds precise, but it is a rough instrument. Google Cloud's natural language documentation describes the output as two numbers. The score ranges from -1.0 (negative) to 1.0 (positive) and corresponds to the overall emotional leaning of the text. The magnitude indicates the overall strength of emotion, both positive and negative.

The same page includes a caveat that is easy to miss. A neutral score near 0.0 may mean a low-emotion document, or it may mean mixed emotions; a truly neutral document will have a low magnitude, while a mixed one has a higher magnitude. It also recommends that you define a threshold that works for you and adjust it after testing.

Two practical lessons follow:

  1. Use sentiment to sort, not to judge. A score can help a manager decide which calls to listen to first. It should not be used on its own to rate a technician or label a customer.
  2. Test the threshold on your own calls. Pull twenty calls, listen to them, compare what you hear with the scores, and set the cut-off from that evidence. Do not copy someone else's number.

What you can usefully ask of your call data

Once transcripts exist and are searchable, the questions get practical. These are examples of the kind of thing a team could search for, not claims about what any system will find:

Before you start: people and permissions

Recording and transcribing conversations raises questions that are not technical. Before recording anything, find out what the law and your client contracts require wherever your callers are located, decide how callers will be told, and decide who may access transcripts and for how long. This post cannot answer those questions for you, and they should be settled before the first call is recorded.

Also decide where audio and text will be stored and processed. If a third-party speech service receives your recordings, read its data-handling terms and check that they fit what your clients have agreed to.

A sensible way to start

  1. Pick one use. For example, searchable transcripts for one team, or a daily list of calls to review.
  2. Run it on a small sample first. Compare the transcript with the audio and keep a record of where it fails.
  3. Keep a person in the loop. Scores sort the work; people make decisions.
  4. Write down what it will not be used for. For example, not for ranking staff on sentiment scores alone.
  5. Review after a few weeks. Look at the failures first, and expand only what has earned trust.

The same approach applies to automation inside a service desk, which we cover in What an AI Service Desk Can (and Can't) Do for an MSP.

Talk it through

If you are thinking about call analytics or a voice assistant on top of an existing VoIP or PBX system, the home page describes this work and links to a way to schedule a consultation. Start with one narrow use and your own test calls.

Sources

All pages were read on 17 October 2026.