Keeping Handwriting as Handwriting

Some thoughts come out better with a pen. A sketch of a room, an arrow between two ideas, a word underlined three times: typing flattens all of that. So when I built Write by Hand for WOYM on iPhone and iPad, I set one rule. The page stays handwriting. It isn’t converted to typed text or redrawn, and it looks the same in the app, on the entry’s card and on the web.

Keeping handwriting as handwriting still has to fit a modern journal. A page has to sync between devices and work offline. It has to show up when you search for a word you wrote, count towards your word count, and look right in dark mode. This post walks through how I did each of those in a Flutter app, and the bugs I hit on the way.

Write by Hand is coming to WOYM on iPhone and iPad in the next App Store update, and the pages you write will show in your journal on woym.app too.

Putting PencilKit inside Flutter

A good pen is mostly things you don’t notice: low latency, pressure and tilt, palm rejection, an eraser that knows what a stroke is. Apple’s PencilKit does all of it, and people already know its tool picker from Notes. Rebuilding that in Flutter would have been slower and worse, so the canvas is native and Flutter hosts it as a platform view.

UiKitView(
  viewType: 'woym/handwriting_canvas',
  creationParams: {
    'drawing': page == null ? null : HandwritingFile.drawingOf(page),
    'dark': theme.brightness == Brightness.dark,
    'background': theme.scaffoldBackgroundColor.toARGB32(),
  },
  creationParamsCodec: const StandardMessageCodec(),
  onPlatformViewCreated: _onCreated,
  // Every touch goes straight to the page, so strokes start at once.
  gestureRecognizers: {
    Factory<OneSequenceGestureRecognizer>(EagerGestureRecognizer.new),
  },
)

The EagerGestureRecognizer matters more than it looks. By default Flutter waits to see whether a touch is a tap or a scroll before handing it to a platform view. For a pen, that wait shows up as a missing start to every stroke. Giving the canvas every touch straight away makes it feel like Notes.

Each canvas gets its own method channel, woym/handwriting_canvas_<id>. Flutter calls export, undo, redo and close; the canvas calls back with changed whenever the drawing changes, so the Flutter app bar knows whether there is ink to save or to warn about discarding.

One page size for every screen

A page written on a 13-inch iPad has to look the same on an iPhone. So strokes live on a page that is always 768 points wide, and the canvas is zoomed to fit whatever width it has. The zoom is locked (minimum and maximum are the same), so the page scrolls down but never pinches. There are 500 points of room below the last stroke to keep writing.

My first version set that zoom inside the canvas’s own layoutSubviews. It looked fine until you wrote on the right half of the page: nothing appeared there. Zooming a PKCanvasView from inside its own layout pass left part of the page unrendered. The fix was a plain container view that owns the layout and fits the canvas from outside:

private func fitPage() {
  canvas.frame = container.bounds
  let width = canvas.bounds.width
  guard width > 0, width != fittedWidth else { return }
  fittedWidth = width
  let scale = width / Self.pageWidth   // pageWidth = 768
  canvas.minimumZoomScale = scale
  canvas.maximumZoomScale = scale
  canvas.zoomScale = scale
  updateContentSize()
}

Two smaller details. On an iPhone the tool picker sits over the bottom of the page, so I watch toolPickerFramesObscuredDidChange and add that height to the canvas’s bottom inset, keeping your writing scrollable above it. And when the screen closes, the app hides the tool picker first; otherwise it can linger over the entry for a moment.

One file, two jobs

A handwritten page has to do two things. It has to display anywhere: in the app, on a card, in a browser, in an export. And it has to stay editable, which means keeping PencilKit’s strokes, not just pixels. The obvious design is two files per page. I wanted one, because then a page uploads, syncs, caches and counts against storage exactly like a photo, and every part of the media pipeline already knew how to do that.

PNG makes this possible. A PNG is a list of chunks, and the format lets you add your own. The case of each letter in a chunk’s name is a flag, so my chunk, pkDw, says three things to any program that reads it:

• p, lowercase: ancillary. Viewers that don’t know it skip it and show the image.
• k, lowercase: private, not part of the PNG standard.
• w, lowercase: safe to copy, even if an editor changes the image.

The PencilKit drawing’s own data goes inside that chunk, just before the file’s end marker, with the CRC-32 checksum PNG requires. Browsers, image viewers and the app’s image cache all see an ordinary PNG. When you open the page again, the app reads the chunk back and hands the strokes to the canvas.

/// [png] with [drawing] added just before its end chunk.
static Uint8List embed(Uint8List png, Uint8List drawing) {
  final end = _chunkOffset(png, 'IEND');
  if (end == null) throw const FormatException('Not a PNG');
  final out = BytesBuilder(copy: false)
    ..add(Uint8List.sublistView(png, 0, end))
    ..add(_chunk(drawing))           // length, 'pkDw', data, CRC-32
    ..add(Uint8List.sublistView(png, end));
  return out.toBytes();
}

The picture itself is exported in light mode, whatever the app is showing: dark ink on a transparent background, at 2× scale, as tall as the writing plus a small margin. Storing one known look keeps every other screen simple, and leaves dark mode to be handled when the page is shown, which turned out to be the most interesting problem of all.

Syncing a page through an immutable media pipeline

WOYM is offline first. Everything you write goes into a local database on the device, and a sync engine pushes and pulls changes when there is a connection. Handwritten pages ride the same road as photos, with a new media type, handwriting, and one new column for the words read from the page.

Media on the server is immutable: an uploaded file never changes under the same id. That is a good rule for photos and an awkward one for a page you can keep writing on. So editing a page replaces it. The new version gets a new id but keeps the original creation time, so it stays in the same place among the entry’s pages, and the old one is removed in the same sync.

Future<void> replaceMedia(EntryMedia old, PlatformFile file,
    {String? handwritingText}) async {
  await addMedia(old.entryId, file, old.mediaType,
      createdAt: old.createdAt, handwritingText: handwritingText);
  await removeMedia(old.id);
}

The page’s words are the one exception to immutability. They are often read after the page has already been uploaded, sometimes by another device, so the server lets a media row’s text be set once, while it is still empty, and never overwritten.

That opened two small races, both fixed with a rule each:

• Pulling never overwrites a local change. If the device has just read a page’s words and hasn’t pushed them yet, an older copy of the row from the server is skipped.
• Pushing only settles what it sent. A row is marked clean after a push only if its text is still what was pushed. Words that arrived mid-push go up next time.

Reading the words, without replacing them

Handwriting you can’t search is a notebook in a drawer. So once a page is saved, a vision model reads it, and the words are stored next to the page. You never see that text in place of your writing. It is there so that searching your journal for a word finds the page you wrote it on, and so that your entry’s word count and your journal stats include what you wrote by hand.

WOYM could already turn a photo of a paper journal into typed text, so pages reuse that endpoint with a prompt tuned for a single handwritten page. One catch: the page is stored as ink on a transparent background, and a model may well see transparent as black. Before sending it, the app draws the page onto white:

static Future<Uint8List> _onWhite(Uint8List png) async {
  final codec = await ui.instantiateImageCodec(png, targetWidth: 1200);
  final ink = (await codec.getNextFrame()).image;
  final recorder = ui.PictureRecorder();
  ui.Canvas(recorder)
    ..drawColor(const ui.Color(0xFFFFFFFF), ui.BlendMode.src)
    ..drawImage(ink, ui.Offset.zero, ui.Paint());
  final page = await recorder.endRecording().toImage(ink.width, ink.height);
  return (await page.toByteData(format: ui.ImageByteFormat.png))!
      .buffer.asUint8List();
}

Reading runs right after a page is saved, and again at the end of every sync. The second run catches pages written offline and pages that arrived from another device. If a page can’t be read on the current plan, the app stores an empty result, so it doesn’t keep asking.

The red ink that turned pink

In dark mode, PencilKit shows black ink as white, so your writing stays readable on a dark page. The stored PNG is always the light version, so I had to do the same when showing a page. My first attempt was a colour matrix that inverts luminance, which is the usual way to turn dark-on-light into light-on-dark. Black became white, as hoped.

Then the first person to test on an iPad wrote in red, and the entry showed it pink. A luminance inversion moves every channel up by the same amount, enough to make the colour as light as it was dark. Red has low luminance, so it got lifted a long way, and red plus a lot of grey is pink.

So what does PencilKit actually do? I drew strokes in a range of colours and sampled the pixels PencilKit rendered in light and in dark appearance. The answer was clean: it flips each colour’s lightness in HSL terms (L becomes 1 − L) and keeps its hue and saturation. A fully saturated red or blue has a lightness of exactly one half, so it doesn’t move at all. Black becomes white, navy becomes a light blue, and mid grey stays close to where it was.

That sounds like a colour space round trip per pixel, but it reduces to one line. Keeping hue and saturation keeps the chroma (brightest channel minus darkest), so every channel moves by the same amount, and that amount is 255 − brightest − darkest:

function flipLightness(rgba: Uint8ClampedArray) {
  for (let i = 0; i < rgba.length; i += 4) {
    if (rgba[i + 3] === 0) continue // clear stays clear
    const r = rgba[i], g = rgba[i + 1], b = rgba[i + 2]
    const shift = 255 - Math.max(r, g, b) - Math.min(r, g, b)
    rgba[i] = r + shift
    rgba[i + 1] = g + shift
    rgba[i + 2] = b + shift
  }
}

The catch is that it is not a matrix: the shift depends on each pixel’s own brightest and darkest channel, so the GPU colour filters both Flutter and CSS offer can’t express it. In the app, a page is decoded to raw RGBA, flipped on a background isolate so scrolling stays smooth, and turned back into an image. The last dozen pages are cached, so scrolling back up doesn’t redo the work. On the web, the same function runs over a canvas’s pixels. Reading a canvas’s pixels needs the image to be loaded with CORS, which the storage already allowed.

I kept the tests honest with real numbers: black goes to white, red (255, 38, 0) and bright blue stay put, grey 146 goes to 109, and navy (1, 25, 147) goes to (108, 132, 254).

Premium, with one free page

Write by Hand is part of WOYM Premium. Everyone gets one page free to try it, counted on the server when the page is first read. Pages you already have always stay yours to open, read and search, whatever plan you are on.

How I tested it

• Unit tests write a page with strokes inside, read the strokes back, and check the ink colours above.
• A Flutter driver script opens the canvas on a simulator with a saved drawing, exports it, and screenshots the entry and its card in light and dark mode, so layout changes show up as pictures, not bug reports.
• And the most useful test of all: someone writing in red on a real iPad.

What I learned

• Use the platform where it is best. Flutter for the app, PencilKit for the pen. The boundary between them is one Swift file and one Flutter screen.
• File formats have room for you. A private PNG chunk turned two files into one and saved me from teaching the sync engine a new kind of attachment.
• Measure the thing you are imitating. “Invert it for dark mode” sounded obvious and was wrong. Sampling PencilKit’s own output gave me an exact rule.

Next, I want handwriting to live between paragraphs, so a sketch can sit right under the sentence it belongs to. If you journal by hand, I would love to hear what you want from it: get in touch.