Making audio editing feel like editing a document

The goal was one sentence. Editing audio should feel like editing a document.

That sentence is easy to say and brutal to ship, because audio resists the comparison. Text is a data structure that happens to be readable. Audio is samples, in buffers, behind decoders, next to a playback graph that was never designed to change while somebody is listening to it.

A single cut looks harmless. Cutting three seconds out of the middle of a 40 minute interview touches four things at once: the waveform on screen, the audio coming out of the speakers, the word level timestamps underneath the transcript, and the undo history. Get the order wrong and the app is correct on the server and visibly broken in front of the user.

The editor in its first weeks. Transcript on the left, four labelled tracks under the ruler, playhead running through them.

I built the editor for Koolio.ai. The work turned out to be two problems, and I underestimated the second one.

The first was speed. Every operation had to land the instant it was triggered. Not fast enough to be tolerable. Instant, in the way a text editor is instant when you hit backspace. If cutting audio takes a few hundred milliseconds to appear, people stop trusting the tool and start hesitating before every edit.

The second was how it feels while it does that. This part is easy to write off as polish, and that would be a mistake. The design team at ownpath built the entire system for it, from the logo and the colours to the way a segment settles into place after you drag it. Fast is the floor, not the finish.

So we made one decision early that shaped everything after it. The editor would run in the browser. The backend would not sit in the path of an edit.

That bought us the feeling we were after. The price came due elsewhere, in how the audio gets stored, how much of the screen has to redraw, and in memory. This is what that cost, and what it was worth.

The decision that shaped everything

Every operation runs on the client. Cut, split, paste, move, resize, delete, rename, nine handlers for them, and not one of them calls the server to do its job. The call happens afterwards, in a queue that runs one operation at a time and waits for the reply before starting the next. Nobody editing ever waits for that queue. They already saw the change.

Cutting audio at network speed does not feel like editing. It feels like filling in a form and hoping the answer comes back right.

The honest part is what this pushed onto us. Multi track audio stitching, the playback graph, the sync between waveform and transcript, and heavy things like stripping every filler word out of an episode. Any of those could have been a backend endpoint that returns new audio and a new transcript, and life would have been simpler. We built them on the client because the alternative was a spinner in the middle of an edit.

The one place we hand the work over is export. Encoding a finished file properly is not an interactive problem, so that runs through WASM, and it is the only part of the editor that does.

An edit lands in the interface first. The server records it afterwards, in order.

How an edit actually works

All the audio lives on the client in one manager. It holds buffers in a map, keyed by position:

speakers-0:103.01:131.84

That is speaker track zero, from 103.01 seconds to 131.84 seconds. The key is not an id and it is not a filename. It is where the audio sits on the timeline, which means the key expires every time you drag the edge of a segment.

The key of a segment is its position, so dragging an edge rewrites it.

Every edit runs through an OfflineAudioContext, never through the graph that is making sound. The file says it in five words: no playback, no shared state. Practically it means you can cut a segment while the episode is playing and the audio does not flinch. One split returns three buffers in a single call: the part before, the part you pulled out, the part after.

Slicing is lazy and cached. Drag a segment edge twice and the second drag reuses the first cut instead of decoding the same range again.

Playback is one mixed buffer for the entire timeline, built on demand, sample by sample, with track gain multiplied by segment gain. That is the only reason multi track playback works at all in a browser. You are listening to a mix you built, not to four audio elements agreeing to start at the same millisecond.

Optimistic updates are written down, not implied. Renaming a speaker updates the UI through a slice reducer before the server has heard anything about it.

The price shows up after every operation. The editor remixes every segment of every track, encodes the result as a WAV in the browser, and writes it into IndexedDB so a reload does not lose your work. Ten minutes of stereo audio is around 101 MB and 26.5 million sample frames, and that runs on the main thread.

Naming audio after its position also has a tax. The client and the server each keep their own id for the same segment, and every cut or split or paste changes the client side one. That is why a helper exists to map one id to the other, and why a bug can report a successful operation while moving the wrong segment.

Two views, one model

The best idea in the project was not an optimisation. It was showing the transcript and the waveform together, synced, both editable, either one collapsible when you only want to look at one of them.

Before: two copies of the project, kept in step by an API call. After: one object that both panels render from.

The first version kept two copies of project state and kept them together with API calls. Switching between the transcript and the timeline fired an operation, then refetched the project metadata. I wrote down what was wrong with that at the time, and the sentence holds up: the endpoint resent the whole project on every switch, which defeats the purpose of making the UI update quickly.

The result was a waveform that lagged behind the transcript. It was noticeable on ordinary projects. Then we worked on it and it got better on smaller ones. Then we took it apart properly and it got fixed everywhere.

What fixed it was not a faster sync. It was deleting one of the two copies. Both panels now render from a single object, and the transcript is generated on demand from it rather than stored next to it. The comments recording the decision are still in the code: removed, now generated on demand from metadataResponse.

You cannot sync two things that cannot disagree. That is the whole lesson, and it took two months to learn.

The design half is what turned it from correct into pleasant. A single expand control sits between the two views. Collapsed, you get the transcript with a compact waveform strip under it. Expanded, you get the full multi track timeline. Two sets of geometry, one model underneath, and animations so the switch never feels like a page load. A speaker is the same colour in the transcript and on the timeline, which quietly does more for the feeling of one surface than any transition does.

The collapsed view: the transcript with a compact waveform strip under it, which is the state the editor opens in.

The fight for speed

None of this arrived at once.

Most of a month went into it. First it was rendering. Then the waveform itself became the problem, loading slowly and fighting playback, so a day went on tuning WaveSurfer. Then loader time, then an IndexedDB fix, then a bug where the app simply kept loading, then cut not refreshing. Nine consecutive days, and not one of them was a feature.

The fix that mattered most was a redraw that had to be narrowed three times. First the whole editor was re-rendering after an operation. Then one track. Then, in my own words from the end of that sprint, one segment. The reason it kept re-rendering was mundane and is still commented in the file: waveform data sat in the dependency list of the handlers, so every operation rebuilt every handler and redrew everything downstream.

The retainer chain from the heap snapshot: request objects, the project payload, the segments, and the decoded audio they kept alive.

Then the memory. Decoded audio held in IndexedDB grew until a heap snapshot showed 7.45 GB retained across 5,992 live ArrayBuffers. It took three or four days to find, and the chain was embarrassingly ordinary once it was visible. The request objects that read the audio were holding it alive long after the editor was done with it. Store a reference to the audio, decode it when you need it, and the leak disappears.

The ruler deserves its own paragraph, because it is the part of an audio editor nobody thinks about until it is wrong. One ruler has to work for a 30 second clip and a two hour podcast. The label interval is 0.5 seconds at the base, raised to 1, 2.5, 5 or 10 seconds as you zoom out, so that marks stay about 3 pixels apart instead of turning into a grey smear. Labels stay in MM:SS at every zoom. On a one hour file that works out at roughly 0.2 pixels per second, so the furthest zoom shows about six minutes of audio. We had to rebuild it around the size of the actual file instead of assuming everyone was editing something three minutes long.

Instant never arrived in one commit. It arrived by shrinking the blast radius of a single edit, three separate times.

Delight is a design project

The motion in the editor had to be invented, because nothing off the shelf could do it.

We used WaveSurfer for the waveforms, and it does that one job well. What it does not give you is an editable timeline where a block of audio can be picked up from one speaker's track, carried across the timeline, and dropped into another speaker's track. Dragging across tracks means the audio changes owner. The segment has to be rebuilt against a different track and snapped to the grid the eye is expecting, then reported to the server as a move with a destination rather than a delete plus a paste. Nobody had an animation library for that, so I wrote it.

The part that made it feel good is smaller than the code behind it. While you drag, the segment shrinks and stretches in real time. You are not holding a ghost that will only reveal its length after you let go. You are holding the answer, and you can feel it before you commit.

That is one example of the thing I want to say plainly. Speed is the floor, and the design team at ownpath built everything above it, from nothing. The whole system for Koolio was made for Koolio, starting with the new logo and the colour palette, and it is the reason the editor looks like a product rather than a demo.

Some of my favourite decisions in it are quiet ones. A speaker keeps the same colour in the transcript and on the timeline, so the two panels read as one surface. Controls appear near the cursor instead of in a fixed toolbar, so your hand stays where the work is. Four destructive operations, stripping filler words, closing long gaps, levelling, noise reduction, all sit behind one calm word: Magic Studio.

Four tracks, four colours. A speaker keeps the same colour in the transcript and on the timeline, which is what makes the two panels read as one surface.

And the last sprint was not features. My own fix list from the end of it opens with UI changes that came out of a usability test: the expand icon, the music and SFX modals pre-filling, and the speaker colours matching the waveform. Real people touched it, and what came back was a list about feel.

Building against an API that did not exist

The first metadata API I ever saw was a mock. Audio only, no annotations, and I built the complete set of frontend components on it. The real one landed weeks later, with annotations and a shape that had nothing to do with the mock. Every component built on the mock had to be refactored.

The operation payloads went through three rewrites. The one worth keeping is the third. We ran on that third shape for a week, and the results were strange. Every response came back 200. The operations were still wrong. A green status code tells you the request arrived. It does not tell you the thing you asked for happened, and a week of that is a good way to stop trusting your own test suite.

Export and publish were built with nothing to integrate against, because the APIs were still being released. Share and collaborate, and version history, sat blocked for longer: restore history, preview version and public access did not exist yet. So we built the whole flow on the frontend and left it waiting.

The one API I argued with was the sync. It was not a sync API. It was the metadata endpoint called again, so switching from the transcript to the waveform sent the entire project back, including the transcript the user was already looking at. I wrote down why that was wrong at the time: if you are working in the transcript and you open the waveform, the server does not need to send the transcript again, and the other way around. It made life easier for the backend team and it defeated the purpose of making the UI update quickly.

Same argument twice more, about work that had no business being a round trip. Pasting needed a separate call to hand a file to a server side clipboard. Adding music or SFX needed its own upload, even though the file was already sitting on the backend. In both cases I asked the same question: why can I not send the name and the details in the operations call I am already making?

What held up, and what I would change

What held up: we opened projects up to two hours long with a lot of speakers, specifically to test loading times and usability, and it held. That is the number I am proudest of, because it is the one that could have gone badly. The fear with a client side editor is that it works beautifully on a three minute test file and dies on an hour long interview with a hundred speakers in it.

What I would change: the visualizer grew to 9,011 lines before operations were pulled out into their own handlers and transcript work moved into its own utilities. A file that size is not a decision anyone makes. It is a decision you stop making.

The migration I would have done next is in my own notes from the end of that sprint: move playback off stitched audio buffers and onto Web Audio nodes, to remove the delay between playback events. I had written down that it would need a substantial reconfiguration of the editor. It was item two of four, and it is the one that would have mattered most.

Two small things I would rather admit than hide. Pointer based paste left a slight gap before the pasted segment. Split occasionally loaded the same audio twice. Neither one sank the product, and both are still in my head.

What I took from it

I spent more time deciding what things were than making them fast. What is a segment, and where does its identity live. Are the transcript and the waveform two views or one object. Every answer I got right made the next hundred lines obvious. Every answer I got wrong cost me a week of optimising something that was never slow.

That is the part I have carried furthest. When something is sluggish, my first instinct is no longer to profile it. It is to ask what the thing actually is, because a wrong model shows up as a performance problem long before it shows up as a bug.

The sync argument was the same lesson in different clothes. A sync API that returns the whole project is not a sync, and the fix was never a faster round trip, it was deciding who owns the state. Most performance work turns out to be ownership work wearing a disguise.