Kyle Howells Blog

Loading Oversized 360 Panoramas with Metal

2026-01-17T00:51:57.109793+00:00

Figure 1. The decision tree our renderer follows whenever a panorama exceeds Metal's single-texture limit.

Summary — Metal caps 2D textures at 16,384 px per side on iOS (32K on recent Macs). To display 20K panoramas you need a fallback ladder: try a single texture upload, then progressively tile the image (vertical strips → grids) before punting to a CPU conversion. This post walks through each rung with simple code, diagrams, and the tradeoffs we measured.

Why the cap exists

The texture dimension limit keeps GPUs from having to allocate unbounded surfaces. Anything exceeding the device's limit must be split into smaller pieces or sampled on the CPU. Here's an example of how to query the limit based on the Metal GPU family:

let device = MTLCreateSystemDefaultDevice()!

// Query the actual texture size limit from Metal
func maxTextureSize(for device: MTLDevice) -> Int {
    // Apple10 (M5, A19+) supports 32K textures
    if device.supportsFamily(.apple10) {
        return 32768
    }

    // Apple3+ (A9 through M4) support 16K textures
    if device.supportsFamily(.apple3) {
        return 16384
    }

    // Older devices (Apple2) have 8K limit
    return 8192
}

let maxSize = maxTextureSize(for: device)
if max(cgImage.width, cgImage.height) <= maxSize {
    // ok to upload as a single texture
}

On iOS devices (A11+), the limit is 16,384 pixels per dimension. Recent Macs with M5 chips support 32K. Our test panoramas were 11K×5.5K (which fits on all devices) and 19,937×9,969 (which requires tiling on iOS).

The Metal feature set is listed by Apple here (in a PDF??).

Metal-Feature-Set-Tables.pdf > "GPU implementation limits by family" > page 7 of 15 > "Resources" > "Maximum 2D texture width and height".

GPU Family	Max 2D Texture
Metal3	16,384 px
Metal4	16,384 px
Apple2	8,192 px
Apple3	16,384 px
Apple4	16,384 px
Apple5	16,384 px
Apple6	16,384 px
Apple7	16,384 px
Apple8	16,384 px
Apple9	16,384 px
Apple10	32,768 px
Mac2	16,384 px

And these are the devices (or SoCs at least) which correspond to each GPU Family.

Source: Metal-Feature-Set-Tables.pdf

Strategy 1 — Single GPU texture

+──────────────+    MTLTextureDescriptor.texture2D
|  11K × 5.5K  |───▶ .texture2D, pixelFormat:.bgra8Unorm
+──────────────+    usage: .shaderRead

Upload via MTKTextureLoader:

let options: [MTKTextureLoader.Option: Any] = [
    .SRGB: false,
    .textureUsage: NSNumber(value: MTLTextureUsage.shaderRead.rawValue)
]
let texture = try loader.newTexture(cgImage: cgImage, options: options)

Fastest path. Only limitation: fails if the image exceeds the device's texture limit. That's when we escalate.

Strategy 2 — Vertical strip tiling

Original: 20K × 10K
Split into 2 strips → each 10K × 10K
┌──────────┬──────────┐
│          │          │
│  Tile 0  │  Tile 1  │  (UV origin 0.0, 0.5)
│          │          │
└──────────┴──────────┘

Key idea: cut along longitude, keep full latitude. Each tile knows its (uvOrigin, uvSize) within the source panorama.

let tilesX = Int(ceil(Double(width) / Double(maxTextureSize)))

for tileX in 0..<tilesX {
    let start = tileX * maxTextureSize
    let tileWidth = min(maxTextureSize, width - start)
    let rect = CGRect(
        x: start,
        y: 0,
        width: tileWidth,
        height: height  // Full height for vertical strips
    )
    let cropped = cgImage.cropping(to: rect)!
    let texture = try loader.newTexture(cgImage: cropped, options: options)
  
    tiles.append(PanoramaTileTexture(
        texture: texture,
        uvOrigin: [Float(start)/Float(width), 0],
        uvSize: [Float(tileWidth)/Float(width), 1]
    ))
}

During the compute pass we bind both textures and choose based on uv.x:

float wrappedU = uv.x - floor(uv.x);
if (wrappedU < tile1Start) { sample tile0 } else { sample tile1 }

This cut the 20K panorama from 78s CPU to 0.93s GPU on our benchmarks. Pros: low tile count, simple metadata. Cons: still limited if the image height also exceeds the texture limit.

Slicing the image on the CPU

The tiling happens in Swift before we touch Metal. We use CGImage.cropping(to:) to extract each strip, then upload each strip as a separate texture:

// Figure out how many tiles we need
let tileCount = Int(ceil(Double(imageWidth) / Double(maxTextureSize)))

for i in 0..<tileCount {
    // Calculate this tile's horizontal slice
    let startX = i * tileWidth
    let actualWidth = min(tileWidth, imageWidth - startX)

    // Crop the strip from the source image
    let cropRect = CGRect(x: startX, y: 0, width: actualWidth, height: imageHeight)
    let tileImage = cgImage.cropping(to: cropRect)!

    // Upload to Metal (each tile is now under the size limit)
    let texture = textureLoader.newTexture(cgImage: tileImage, options: options)
    tiles.append(texture)
}

The key is that each tile stores its UV origin so the shader knows where it belongs in the full panorama. Tile 0 starts at U=0, tile 1 starts at U=0.5 (for 2 tiles), etc.

Strategy 3 — Grid tiling

20K × 20K (hypothetical)
Split into 4 tiles (2×2 grid)
┌───────┬───────┐
│  0,0  │  1,0  │
├───────┼───────┤
│  0,1  │  1,1  │
└───────┴───────┘

We extend the loader to tile along both axes:

let tilesX = Int(ceil(Double(width) / Double(maxTextureSize)))
let tilesY = Int(ceil(Double(height) / Double(maxTextureSize)))

The shader becomes a small loop over metadata:

// Tile metadata structure
struct PanoramaTile {
    float2 uvOrigin;   // Where this tile starts in UV space (e.g., [0.5, 0])
    float2 uvSize;     // Size of this tile in UV space (e.g., [0.5, 1.0])
    uint textureIndex; // Which texture slot this tile occupies
};

// In the compute kernel:
float2 wrappedUV = float2(uv.x - floor(uv.x), clamp(uv.y, 0.0, 1.0));

for (uint i = 0; i < uniforms.tileCount; ++i) {
    PanoramaTile tile = tiles[i];

    // Check if this UV falls within this tile's region
    bool inTileX = wrappedUV.x >= tile.uvOrigin.x &&
                   wrappedUV.x < (tile.uvOrigin.x + tile.uvSize.x);
    
    bool inTileY = wrappedUV.y >= tile.uvOrigin.y &&
                   wrappedUV.y < (tile.uvOrigin.y + tile.uvSize.y);

    if (inTileX && inTileY) {
        // Remap global UV to local tile UV
        float2 localUV = (wrappedUV - tile.uvOrigin) / tile.uvSize;
        return tileTextures[tile.textureIndex].sample(sampler, localUV);
    }
}

ASCII of the texture bindings (we cap at 16):

Metal slot layout
[0] tile0  [1] tile1  [2] tile2  [3] tile3  ...  [15] tile15
[16] output cube texture

Grid tiling was ~10% faster than vertical strips in our tests (0.84s vs 0.93s for the 20K panorama) because each tile is smaller, which helps texture cache locality near the poles.

The tiled compute shader

The key challenge in the shader is figuring out which tile to sample for any given UV coordinate, then remapping that global UV to the tile's local UV space.

Here's the core logic, which runs identically to the single-texture version except for the tile selection:

kernel void equirectangularToCubeTiled(
    texture2d<float> tile0 [[texture(0)]],
    texture2d<float> tile1 [[texture(1)]],
    // ... up to 4 tiles
    texturecube<float, access::write> cubeOutput [[texture(4)]],
    constant Uniforms &uniforms [[buffer(0)]],
    uint3 gid [[thread_position_in_grid]])
{
    // Same as before: get direction, convert to panorama UV
    float3 direction = cubeFaceDirection(gid.z, pixelUV);
    float2 globalUV = directionToUV(direction);

    // NEW: Figure out which tile this UV falls into
    float wrappedU = globalUV.x - floor(globalUV.x);  // Wrap at seam
    uint tileIndex = uint(wrappedU * float(uniforms.tileCount));
    tileIndex = min(tileIndex, uniforms.tileCount - 1);  // Clamp to valid range

    // NEW: Remap global U → local tile U
    // If we have 2 tiles: tile 0 covers U=[0, 0.5), tile 1 covers U=[0.5, 1)
    // A global U of 0.75 becomes local U of 0.5 in tile 1
    float tileWidth = 1.0 / float(uniforms.tileCount);
    float tileStartU = float(tileIndex) * tileWidth;
    float localU = (wrappedU - tileStartU) / tileWidth;
    float2 localUV = float2(localU, globalUV.y);

    // Sample from the correct tile (switch is required in Metal)
    float4 color;
    switch (tileIndex) {
        case 0: color = tile0.sample(sampler, localUV); break;
        case 1: color = tile1.sample(sampler, localUV); break;
        // ...
    }

    cubeOutput.write(color, uint2(gid.xy), gid.z);
}

The UV remapping math is the heart of tiling. Think of it like this: if you have 2 tiles, tile 0 "owns" the left half (U=0 to 0.5) and tile 1 owns the right half (U=0.5 to 1.0). When the shader wants to sample at global U=0.75, it needs to:

Determine that 0.75 falls in tile 1 (the right half)
Remap 0.75 to local coordinates: (0.75 - 0.5) / 0.5 = 0.5
Sample tile 1 at local U=0.5

Figure 2. The UV remapping process: find which tile contains the sample point, then convert global UV to local tile UV.

Strategy 4 — CPU fallback

Sometimes the GPU path still fails: more than 16 tiles needed, out-of-memory errors, or you need bit-exact deterministic output for offline export. The CPU fallback mirrors the GPU shader logic exactly—it's just ~100× slower because we're doing millions of texture samples on the CPU instead of GPU cores.

Why you need bilinear filtering

The GPU's texture sampler does bilinear filtering automatically. On CPU, we have to implement it ourselves. Without filtering, you'd see blocky pixels when the panorama resolution doesn't match the cube face resolution exactly.

Bilinear filtering samples four neighboring pixels and blends them based on where the sample point falls:

func sample(u: Float, v: Float) -> SIMD4<Float> {
    // Convert UV to pixel coordinates (may land between pixels)
    let fx = u * Float(width - 1)
    let fy = v * Float(height - 1)

    // Get the four corner pixel coordinates
    let x0 = Int(floor(fx))
    let x1 = min(x0 + 1, width - 1)
    let y0 = Int(floor(fy))
    let y1 = min(y0 + 1, height - 1)

    // How far between pixels? (0.0 = exactly on x0, 1.0 = exactly on x1)
    let tx = fx - Float(x0)
    let ty = fy - Float(y0)

    // Read the four corners and blend
    let c00 = readPixel(x0, y0)
    let c10 = readPixel(x1, y0)
    let c01 = readPixel(x0, y1)
    let c11 = readPixel(x1, y1)

    // Interpolate horizontally, then vertically
    let top = mix(c00, c10, tx)
    let bottom = mix(c01, c11, tx)
    return mix(top, bottom, ty)
}

The conversion loop

The CPU conversion uses the same math as the shader. The key to acceptable performance is parallelizing aggressively:

// Process all 6 faces in parallel
DispatchQueue.concurrentPerform(iterations: 6) { face in
    // Within each face, process rows in parallel
    DispatchQueue.concurrentPerform(iterations: faceSize) { y in
        for x in 0..<faceSize {
            // Convert pixel → cube face UV → direction → panorama UV
            let cubeUV = SIMD2<Float>(
                (Float(x) + 0.5) / Float(faceSize) * 2.0 - 1.0,
                (Float(y) + 0.5) / Float(faceSize) * 2.0 - 1.0
            )
            let direction = cubeFaceDirection(face, cubeUV)
            let panoUV = directionToUV(direction)

            // Sample and write (same as GPU, just 100× slower per pixel)
            let color = imageSampler.sample(u: panoUV.x, v: panoUV.y)
            writePixel(face, x, y, color)
        }
    }
}

The nested DispatchQueue.concurrentPerform gives us parallelism at two levels: faces and rows within each face. On an 8-core Mac, this brings a 20K panorama conversion from ~5 minutes (single-threaded) to ~78 seconds—still 85× slower than the GPU path, but workable as a fallback.

Pros: works for any size, produces identical results to the GPU path, no Metal texture limits to worry about.

Cons: 38–78s runtime on modern Macs for 11–20K inputs, ~1.5 GB RAM for the decoded image plus output buffers.

Putting it together — automatic ladder

switch strategy {
   case .automatic:
      if trySingleTexture(cgImage) { break }
      if tryTiling(.verticalStrips) { break }
      if tryTiling(.grid) { break }
  
      runCPUFallback(cgImage)
   
   case .forceVerticalTiles:
      _ = tryTiling(.verticalStrips) || runCPUFallback(cgImage)
      // ... etc
}

ASCII flow:

[Upload single texture?]
     | yes
   render
     | no
[Vertical tiles succeed?]
     | yes → render
     | no
[Grid tiles succeed?]
     | yes → render
     | no
[CPU fallback]

Benchmarks

All tests performed on Apple Silicon with Release builds. The results are total time from CGImage input to usable MTLTexture cube map output:

Panorama	CPU Fallback	GPU Vertical Tiles	GPU Grid Tiles	Speedup
11K × 5.5K	38.2s	0.34s	0.31s	~110×
19.9K × 10K	78.2s	0.93s	0.84s	~85×

Figure 3. Performance comparison showing the dramatic speedup from GPU tiling—up to 110× faster than CPU fallback.

The dramatic speedup comes from three factors:

Massive parallelism — GPU shader cores process millions of pixels simultaneously
Hardware texture sampling — Built-in bilinear filtering and optimized memory access patterns
No CPU↔memory bottleneck — Data stays on GPU after upload

Additional observations:

Vertical tiles keep resource usage low (only 2 textures for 20K×10K) and are perfect for most 2:1 panoramas
Grid tiles cost more texture slots (up to 16) but squeeze out ~10% extra performance and handle non-standard aspect ratios
CPU fallback remains as the safety net for extremely large assets or offline export scenarios requiring bit-exact determinism

Memory comparison

The GPU tiled path also uses less peak memory than the CPU fallback:

Figure 4. Memory breakdown for a 20K panorama. The GPU path can release tiles after upload, reducing peak usage.

Visual aids

Panorama view            Cube face view
┌───────────────────┐    ┌────────┐
│ longitudinal lines│    │  sky   │
│ ■ camera forward  │    │  ▲     │
└───────────────────┘    └────────┘

Figure 5. Visualising how the same panorama is sliced for each upload strategy before being converted into cube faces.

Left: single texture (fits). Middle: vertical strips (2 tiles). Right: grid tiles (4 tiles). The rendered cube map looks the same—the difference is all under the hood.

Takeaways

Keep work on the GPU. Hardware texture units and compute pipelines devour these conversions in milliseconds once the data is in VRAM.
Tile gradually. Longitudinal splits cover most panoramas; only fall back to grids when height also exceeds the cap.
Leave a CPU escape hatch. There will always be some "mega" image, offline export requirement, or simulator configuration that needs it.

Next time I'll cover how we stream these tiles incrementally so users see a preview almost instantly, even before the full-resolution cube finishes baking.

Rendering 360° Photos: From Equirectangular Pixels to a View

2026-01-16T22:55:02.189949+00:00

Figure 1. Annotated equirectangular source showing meridians, parallels, and the forward camera marker.

Figure 2. Left: the source panorama with a 90° frustum overlay at the center. Right: the corresponding region rendered as viewed through the virtual camera.

TL;DR — Treat every pixel in an equirectangular panorama as a latitude/longitude sample on the unit sphere. Convert those angles into a direction vector, rotate it by your virtual camera pose, then sample the texture at the corresponding UV. The math is identical whether you're rendering in Metal, SceneKit, Unity, or WebGL.

1. The coordinate system

An equirectangular image is essentially a world map projection applied to a sphere. It encodes (longitude, latitude) over (u, v) texture space:

Longitude spans [-π, +π] horizontally (left edge = -180°, right edge = +180°)
Latitude spans [+π/2, -π/2] vertically (top = north pole, bottom = south pole)

I keep the math straight with this ASCII map:

      u=0        u=0.5       u=1.0
       ↓           ↓           ↓
+90° ┌─────────────────────────┐ v=0   North pole
     │                         │
  0° │  -180°       0°    +180°│ v=0.5  Equator
     │                         │
-90° └─────────────────────────┘ v=1   South pole

The key insight: every pixel in the texture corresponds to a direction in 3D space.

Figure 3. The spherical coordinate system. Every direction on this sphere maps to a unique UV coordinate in the panorama texture.

UV to Direction

Given normalized texture coordinates (u, v) in [0, 1]², first convert to angles:

let longitude = (u - 0.5) * 2.0 * Float.pi  // Range: [-π, +π]
let latitude = (0.5 - v) * Float.pi          // Range: [+π/2, -π/2]

Then convert those spherical coordinates to a Cartesian direction vector:

let direction = SIMD3<Float>(
    cos(latitude) * cos(longitude),  // x: left/right
    sin(latitude),                   // y: up/down
    cos(latitude) * sin(longitude)   // z: forward/back
)

This follows the standard convention where:

+X points right (longitude = 0°)
+Y points up (latitude = +90°, north pole)
+Z points forward (longitude = +90°)

Direction to UV (the inverse)

For rendering, we need the reverse: given a 3D direction, find the texture UV:

func directionToUV(_ direction: SIMD3<Float>) -> SIMD2<Float> {
    let longitude = atan2(direction.z, direction.x)        // Range: [-π, +π]
    let latitude = asin(clamp(direction.y, -1.0, 1.0))     // Range: [-π/2, +π/2]

    let u = (longitude / (2.0 * Float.pi)) + 0.5           // Range: [0, 1]
    let v = 0.5 - (latitude / Float.pi)                    // Range: [0, 1]

    return SIMD2<Float>(u - floor(u), clamp(v, 0.0, 1.0))  // Wrap u, clamp v
}

The u - floor(u) handles wrapping at the ±180° seam, while v gets clamped because the poles are singularities.

Figure 4. Left: a sample point in UV texture space. Right: the corresponding direction on the sphere.

2. Camera rotation & projection

The camera's orientation is represented as a rotation matrix (or quaternion). To find which direction in the panorama corresponds to a given screen pixel, we work backwards: start from the screen, unproject to a view ray, then rotate into world space.

Screen to view ray

For a pixel at normalized device coordinates (ndcX, ndcY) where (-1, -1) is bottom-left and (1, 1) is top-right:

// Convert screen pixel to NDC
let ndcX = (2.0 * pixelX / viewportWidth) - 1.0
let ndcY = 1.0 - (2.0 * pixelY / viewportHeight)  // Flip Y

// Create view ray (camera looking down -Z axis)
let viewDir = normalize(SIMD3<Float>(
    ndcX * tan(fovX / 2.0),
    ndcY * tan(fovY / 2.0),
    -1.0
))

Rotate to world space

Multiply the view direction by the camera's rotation matrix to get the world-space direction:

let worldDir = (cameraRotationMatrix * SIMD4<Float>(viewDir, 0.0)).xyz

The result is a unit vector pointing from the viewer into the panorama. Pass this to directionToUV() to sample the texture.

The reverse (world to screen)

If you need to project a world direction back to screen coordinates:

func worldToScreen(_ worldDir: SIMD3<Float>,
                   cameraMatrix: simd_float4x4,
                   fov: SIMD2<Float>,
                   viewport: SIMD2<Float>) -> SIMD2<Float>?
{
    // Transform to view space (inverse rotation)
    let invMatrix = cameraMatrix.inverse
    let viewDir = (invMatrix * SIMD4<Float>(worldDir, 0.0)).xyz

    // Behind the camera? Can't project.
    guard viewDir.z < 0 else { return nil }

    // Perspective divide
    let ndcX = viewDir.x / (-viewDir.z * tan(fov.x / 2.0))
    let ndcY = viewDir.y / (-viewDir.z * tan(fov.y / 2.0))

    // NDC to pixels
    let screenX = (ndcX + 1.0) * 0.5 * viewport.x
    let screenY = (1.0 - ndcY) * 0.5 * viewport.y

    return SIMD2<Float>(screenX, screenY)
}

3. Practical shader

Here's a fragment shader I use inside a Metal MTKView to render a full equirectangular image without cubemaps:

fragment float4 equirectangularFragment(VertexOut in [[stage_in]],
                                        constant CameraUniforms &uniforms [[buffer(0)]],
                                        texture2d<float> pano [[texture(0)]],
                                        sampler panoSampler [[sampler(0)]])
{
    float3 viewDir = normalize(float3(
        in.ndc.x * uniforms.tanHalfFov.x,
        in.ndc.y * uniforms.tanHalfFov.y,
        -1.0
    ));
    float3 worldDir = normalize((uniforms.rotationMatrix * float4(viewDir, 0)).xyz);
    float lon = atan2(worldDir.z, worldDir.x);
    float lat = asin(clamp(worldDir.y, -1.0, 1.0));
    float2 uv = float2(
        (lon / (2.0 * M_PI_F)) + 0.5,
        0.5 - (lat / M_PI_F)
    );
    return pano.sample(panoSampler, fract(uv));
}

The CPU just updates uniforms.rotationMatrix from a quaternion every frame.

4. Visualising the math

I like to keep scratch graphics around to sanity-check the orientation math:

 Viewer     Cube Face (forward cam)
    •          ┌──────────────┐
   /|\         │      ↑ y     │
  / | \        │← -x  +z  +x →│
 /  |  \       │      ↓ y     │
/   |   \      └──────────────┘

Draw the equator and longitudes directly on the source JPEG with Photoshop or Preview annotations. When the shader runs you should see those lines land where you expect.
Render a tiny minimap (equirectangular) alongside the 3D view. Whenever you drag the camera, highlight the direction vector on the minimap so it's obvious where you're looking.

Figure 5. The same camera position rendered at different fields of view. Wider FOV shows more of the panorama but with more distortion at the edges.

5. Sampling only what you see

A 20K×10K panorama is ~758 MB once decoded. You rarely need the entire thing at native resolution. Three approaches:

Approach 1: Direct equirectangular sampling

Load the whole image into a texture2d<float> and let the fragment shader sample directly using directionToUV(). This is the simplest approach, but hits Metal's 16K texture limit on iOS devices (32K on recent Macs).

Approach 2: Pre-bake to cube map (recommended)

Run a compute shader to convert the equirectangular panorama into six cube faces at load time. Rendering then becomes a simple texturecube lookup per pixel:

// In the fragment shader - just one line!
float4 color = cubeTexture.sample(sampler, worldDir);

The cube map conversion uses the same directionToUV() math, but runs once up front rather than every frame. For each pixel in each cube face, we:

Calculate the world-space direction for that pixel
Convert to UV coordinates in the source panorama
Sample and write to the cube face

ASCII view of the six cube faces (unfolded):

          ┌─────┐
          │ +Y  │  (up)
    ┌─────┼─────┼─────┬─────┐
    │ -X  │ +Z  │ +X  │ -Z  │
    └─────┼─────┼─────┴─────┘
          │ -Y  │  (down)
          └─────┘

Figure 6. The six cube faces generated from the panorama, arranged in standard cross layout. The +Z (forward) face is highlighted.

This approach saves battery during panning since we're sampling a cube map (which GPUs are highly optimized for) instead of recomputing spherical coordinates per pixel.

Approach 3: Tiled GPU conversion

For panoramas exceeding 16K, we can't upload a single texture. The solution is to split the source image into tiles that each fit within the limit, then sample from multiple textures in the compute shader. See my follow-up post Taming Oversized Panoramas on Metal for the full implementation—we measured ~85× speedup over CPU fallback.

6. Bringing it together in Swift

Now let's connect all these concepts into a working renderer. The architecture has three main pieces:

A full-screen quad — We draw two triangles that cover the entire screen
A fragment shader — For each pixel, it calculates which direction we're looking and samples the panorama
A compute shader — Converts the equirectangular image to a cube map once at load time

6.1 The rendering approach

The trick is that we don't render a 3D sphere. Instead, we render a flat full-screen quad and do all the spherical math in the fragment shader. For each pixel on screen, we:

Calculate its normalized device coordinates (NDC) from -1 to +1
Use the FOV to convert that to a view-space ray direction
Rotate that ray by the camera orientation to get a world direction
Convert that direction to panorama UV coordinates
Sample the texture

This is why the vertex shader is trivially simple—it just passes through positions:

vertex VertexOut equirectangularVertex(uint vertexID [[vertex_id]],
                                       constant float2 *positions [[buffer(0)]]) {
    VertexOut out;
    out.position = float4(positions[vertexID], 0.0, 1.0);
    out.ndc = positions[vertexID];  // Pass NDC to fragment shader
    return out;
}

6.2 The fragment shader

The fragment shader does the heavy lifting. It needs two pieces of data from the CPU: the camera's rotation matrix and the tangent of half the field of view (pre-computed to avoid trig in the shader).

struct CameraUniforms {
    float4x4 rotationMatrix;  // Camera orientation as 4x4 matrix
    float2 tanHalfFov;        // tan(fov/2) for x and y, pre-multiplied by aspect ratio
    float2 padding;           // Metal requires 16-byte alignment
};

The shader itself chains together the math from earlier sections:

fragment float4 cubeFragment(VertexOut in [[stage_in]],
                             constant CameraUniforms &uniforms [[buffer(0)]],
                             texturecube<float> cubeTexture [[texture(0)]],
                             sampler textureSampler [[sampler(0)]]) {
    // Step 1: NDC → view ray
    // The tanHalfFov scales the NDC so the edges of the screen
    // correspond to the edges of our field of view
    float3 viewDir = normalize(float3(
        in.ndc.x * uniforms.tanHalfFov.x,
        in.ndc.y * uniforms.tanHalfFov.y,
        -1.0  // Camera looks down -Z
    ));

    // Step 2: Rotate view ray → world direction
    float3 worldDir = normalize((uniforms.rotationMatrix * float4(viewDir, 0)).xyz);

    // Step 3: Sample the cube map
    // GPU hardware handles the direction → face + UV conversion for us!
    return cubeTexture.sample(textureSampler, worldDir);
}

Notice we're sampling a texturecube here, not the equirectangular texture directly. Cube maps are what GPUs are optimized for—the hardware handles converting a 3D direction into the correct face and UV coordinates. That's why we pre-convert the panorama.

6.3 The cube map conversion

The conversion runs once when loading an image. For each pixel on each of the six cube faces, we need to figure out what direction that pixel represents, then sample the equirectangular source at the corresponding UV.

The key insight is that gid.z (the third dimension of our compute grid) represents which cube face we're on:

kernel void equirectangularToCube(
    texture2d<float, access::sample> source [[texture(0)]],
    texturecube<float, access::write> cube [[texture(1)]],
    constant CubeConversionUniforms &uniforms [[buffer(0)]],
    uint3 gid [[thread_position_in_grid]])  // x, y = pixel; z = face index
{
    // Bounds check (compute shaders can overshoot)
    if (gid.x >= uniforms.faceSize || gid.y >= uniforms.faceSize || gid.z >= 6) {
        return;
    }

    // Convert pixel (0..faceSize) to UV (-1..+1) on the cube face
    float2 uv = ((float2(gid.xy) + 0.5) / float(uniforms.faceSize)) * 2.0 - 1.0;

    // Get the 3D direction this pixel represents
    float3 direction = cubeFaceDirection(gid.z, uv);

    // Convert direction → panorama UV, then sample
    float2 panoUV = directionToUV(direction);
    float4 color = source.sample(sampler(filter::linear, address::repeat), panoUV);

    // Write to the cube face
    cube.write(color, uint2(gid.xy), gid.z);
}

The cubeFaceDirection function maps each face to its axis. The pattern is: one component is ±1 (the face's axis), and the other two come from the UV coordinates:

float3 cubeFaceDirection(uint face, float2 uv) {
    switch (face) {
        case 0: return normalize(float3( 1, -uv.y, -uv.x));  // +X face
        case 1: return normalize(float3(-1, -uv.y,  uv.x));  // -X face
        case 2: return normalize(float3(uv.x,  1,  uv.y));   // +Y face (up)
        case 3: return normalize(float3(uv.x, -1, -uv.y));   // -Y face (down)
        case 4: return normalize(float3(uv.x, -uv.y,  1));   // +Z face (front)
        default: return normalize(float3(-uv.x, -uv.y, -1)); // -Z face (back)
    }
}

6.4 Swift: Dispatching the compute shader

On the Swift side, we dispatch one thread per output pixel. The grid is (faceSize × faceSize × 6) to cover all six faces:

// Create the output cube texture
let desc = MTLTextureDescriptor.textureCubeDescriptor(
    pixelFormat: .bgra8Unorm,
    size: faceSize,
    mipmapped: false
)
desc.usage = [.shaderRead, .shaderWrite]
let cubeTexture = device.makeTexture(descriptor: desc)!

// Dispatch the compute shader
let encoder = commandBuffer.makeComputeCommandEncoder()!
encoder.setComputePipelineState(computePipeline)
encoder.setTexture(sourceTexture, index: 0)
encoder.setTexture(cubeTexture, index: 1)
encoder.setBytes(&uniforms, length: MemoryLayout<CubeConversionUniforms>.stride, index: 0)

// One thread per pixel, across all 6 faces
encoder.dispatchThreads(
    MTLSize(width: faceSize, height: faceSize, depth: 6),
    threadsPerThreadgroup: MTLSize(width: 16, height: 16, depth: 1)
)
encoder.endEncoding()
commandBuffer.commit()
commandBuffer.waitUntilCompleted()

6.5 Swift: The render loop

Each frame, we update the camera orientation and render the full-screen quad:

func render(to drawable: MTLTexture) {
    // Convert quaternion orientation to matrix for the shader
    var uniforms = CameraUniforms(
        rotationMatrix: simd_float4x4(orientation),
        tanHalfFov: SIMD2<Float>(
            tan(fovRadians / 2) * aspectRatio,  // X includes aspect ratio
            tan(fovRadians / 2)                  // Y is just the FOV
        )
    )

    let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: passDesc)!
    encoder.setRenderPipelineState(renderPipeline)
    encoder.setVertexBuffer(quadVertexBuffer, offset: 0, index: 0)
    encoder.setFragmentBytes(&uniforms, length: MemoryLayout<CameraUniforms>.stride, index: 0)
    encoder.setFragmentTexture(cubeTexture, index: 0)
    encoder.setFragmentSamplerState(samplerState, index: 0)
    encoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 6)
    encoder.endEncoding()
}

The quadVertexBuffer contains six vertices forming two triangles that cover clip space from (-1,-1) to (1,1).

6.6 Putting it in a view

Here's how the pieces connect in a minimal Metal-based panorama viewer:

final class PanoramaView: MTKView {

    private var renderer: CubePanoramaRenderer?

    // Camera state
    private var yaw: Float = 0       // Horizontal rotation (radians)
    private var pitch: Float = 0     // Vertical rotation (radians)

    override init(frame: CGRect, device: MTLDevice?) {
        super.init(frame: frame, device: device ?? MTLCreateSystemDefaultDevice())

        self.renderer = CubePanoramaRenderer()
        self.renderer?.fieldOfViewDegrees = 90

        // Enable continuous redraw for smooth interaction
        self.isPaused = false
        self.enableSetNeedsDisplay = false
    }

    override func draw(_ rect: CGRect) {
        guard let drawable = currentDrawable,
              let renderer = renderer
        else { return }

        // Build orientation quaternion from yaw/pitch
        let yawQuat = simd_quatf(angle: yaw, axis: SIMD3<Float>(0, 1, 0))
        let pitchQuat = simd_quatf(angle: pitch, axis: SIMD3<Float>(1, 0, 0))
        renderer.orientation = yawQuat * pitchQuat

        _ = renderer.render(to: drawable.texture, clearColor: clearColor)
        drawable.present()
    }

    func setImage(_ image: UIImage) {
        renderer?.setTexture(from: image)
    }

    // Called by gesture recognizer
    func pan(deltaX: Float, deltaY: Float) {
        let sensitivity: Float = 0.005
        yaw -= deltaX * sensitivity
        pitch = max(-.pi/2, min(.pi/2, pitch - deltaY * sensitivity))
    }

}

The key integration points:

Initialization — Create the renderer once; it holds the Metal pipeline state and any converted cube textures.
Loading images — setTexture(from:) handles the equirectangular → cube conversion automatically (GPU path for images within the device's texture limit, tiled path for larger).
Orientation — Convert user input (pan gestures, device motion) into yaw/pitch angles, then build a quaternion. The renderer multiplies this into the view matrix.
Render loop — Each frame, update orientation and call render(to:). The fragment shader samples the panorama (or cube map) using the math from sections 1–3.

For snapshots or thumbnails, use makeSnapshot(width:height:) which renders to an offscreen texture and returns a CGImage.

The math here is the same whether you target Metal, SceneKit, Unity, or WebGL. If you can turn (u, v) into (lon, lat) and then into a view ray, you can show a convincing 360° photo on any screen.

7. Further reading

Taming Oversized Panoramas on Metal — my follow-up post on GPU tiling for images exceeding the 16K texture limit

DateFormatter's New Year's Bug - YYYY vs yyyy

2026-01-11T03:54:19.018058+00:00

Every year around New Year's Eve, a wave of date-related bugs show up in iOS apps. The reason is a single character mistake in DateFormatter: using YYYY instead of yyyy.

The Bug

DateFormatter has two different year specifiers:

yyyy - Calendar year (what you almost always want)
YYYY - ISO week-based year (rarely what you want)

The ISO week-based year can differ from the calendar year during the first and last few days of the year. When a week spans two calendar years, the ISO week-based year uses the year that contains the majority of that week.

When This Bites You

On December 30th, 2024 (a Monday), the ISO week-based year is already 2025 because that Monday starts ISO week 1 of 2025. But the calendar year is still 2024. This year it showed up Monday 29th December 2025, the first day of the week it becomes 2026.

import Foundation

let formatter = DateFormatter()
let dec30_2024 = Calendar.current.date(from: DateComponents(year: 2024, month: 12, day: 30))!

// Wrong - uses ISO week-based year
formatter.dateFormat = "YYYY-MM-dd"
print(formatter.string(from: dec30_2024))
// Output: 2025-12-30

// Correct - uses calendar year
formatter.dateFormat = "yyyy-MM-dd"
print(formatter.string(from: dec30_2024))
// Output: 2024-12-30

Here's a table showing how dates around the 2019/2020 boundary behave differently:

Date	Weekday	yyyy (Calendar)	YYYY (ISO Week)	ISO Week
2019-12-29	Sunday	2019-12-29	2019-12-29	52
2019-12-30	Monday	2019-12-30	2020-12-30	01
2019-12-31	Tuesday	2019-12-31	2020-12-31	01
2020-01-01	Wednesday	2020-01-01	2020-01-01	01

Notice how December 30th and 31st, 2019 show as "2020" when using YYYY because they fall in ISO week 1 of 2020.

Real-World Bug Example

Imagine you're generating log filenames in your app:

let formatter = DateFormatter()
let dec31_2019 = Calendar.current.date(from: DateComponents(year: 2019, month: 12, day: 31))!

// Bug: This creates a file with the wrong year!
formatter.dateFormat = "''YYYY-MM-dd'.txt'"
print(formatter.string(from: dec31_2019))
// Output: 2020-12-31.txt

// Correct: Uses the actual calendar year
formatter.dateFormat = "''yyyy-MM-dd'.txt'"
print(formatter.string(from: dec31_2019))
// Output: 2019-12-31.txt

Your 31st December 2019 log file ends up named 2020-12-31.txt.

When to Use Each

Use yyyy for:

User-facing dates
Filenames and paths
Database records
API date strings
Anything where you want the actual calendar year

Use YYYY only when:

Displaying ISO 8601 week dates (e.g., "2020-W01-3")
Specifically working with ISO week numbering
Always paired with the week number format specifier (ww)

The Rule

If you're not explicitly working with ISO week numbers, you want lowercase yyyy. The uppercase YYYY variant exists for a specific use case (week-based date systems), and using it for general date formatting is almost always a bug.

This is one of those bugs that's invisible 360+ days of the year, then strikes during New Year's when everyone's on vacation.

Porting an HTML5 Parser to Swift

2025-12-18T05:00:59.926593+00:00

Porting an HTML5 Parser to Swift

Emil Stenström spent months building justhtml, a Python HTML5 parser that achieves 100% compliance with the html5lib test suite. He wrote about the process - it involved starting from scratch multiple times, pivoting strategies when things weren't working, and iterating with AI coding agents until every edge case was handled.

Simon Willison then ported it to JavaScript in 4.5 hours. He had GPT-5.2 inspect the Python codebase, create a specification, and then just told it "do the rest, commit and push often" while he decorated for Christmas.

Looking around after reading those posts, I saw on the Github page:

JustHTML	✅ 100%	✅ Yes	⚡ Fast
BeautifulSoup	🔴 4%	✅ Yes	🐢 Slow

I have some projects which use BeautifulSoup and know how slow it can be. So I switched some of my python projects from BeautifulSoup to Emil Stenström's new justhtml library and loved it.

After that I started wishing I had a similar "known good" stable pure swift HTML parsing library for use in my Swift projects (the main language I actually write and use day to day).
Then I realised, I couple probably just have Claude make one. The same way Emil Stenström made the library originally. The tests!

So I built swift-justhtml, a port of the justhtml API to Swift which also passes 100% of the html5lib-tests like the original python version.

The Agentic Feedback Loop

LLMs are amazing at writing mostly correct code. But only mostly. Coding Agents like Claude Code and Codex are so powerful because they allow for instant feedback and iteration. The models can try to compile or run the code they just wrote, see the error and fix it. Repeatedly if needed.
So any programming task you can give them which fits that loop, fast iteration, quick feedback on any errors soon after they are made, guidance if the error has actually been fixed or not, and tools they can run to inspect, debug and test in different ways are perfect.

This project, with a fully spec'd out set of tests via the html5lib test suite, is perfect.
Both the input and output are text, something LLMs understand.
The task is to transform from one to another, via a CLI tool the coding agent can run itself to check its work.
And detailed feedback, including on regressions or things it has broken by accident while making another change, via the fully spec'd test suite.

So I downloaded the tests, onto my home dev server, created an empty swift library/package directory structure and setup the CLI tools needed. Then also downloaded the python and js implementations alongside the tests and told Claude Code with Opus 4.5 (which I still find produces the most reliable code, even if Codex GPT 5 is smarter at debugging errors) to look at the public API interface the python and js versions offer, and then to create a rough spec for a similar Swift API. This was to try to allow the model room to adjust the API to Swift's type system and idioms by not being too strict it be the same public facing API.

I then told it to setup the swift library tests to load and run the html5lib tests, and report how many pass/fail, and to implement the HTML parser, run the tests. To then work on fixing the failing tests, and re-run the tests. And to repeat that until it had 100% of the tests passing.

It took two days of intensive work with Claude working away in the background iterating on the tests, sometimes trying to cheat (we'll get back to that), and then after a lot more work on performance tuning than I thought would be needed! Here's how it actually went.

Racing to 100%

The initial commit was a straightforward smoke test of a valid HTML string to test and verify the DOM tree works <p>Hello</p>. This was because I remember'ed Emil's blog post mentioned he started with this before having it dive into edge cases and handling invalid input and that sounded very wise, so I made sure to start from a known point too.

Then came the core implementation. HTML5 parsing is deceptively complex. The specification defines 67 distinct tokenizer states, 23 tree builder insertion modes, and algorithms with names like "adoption agency" and "Noah's Ark clause."

This is the Noah's Ark clause. But with three per family instead of two.

The first real implementation got to 53% test compliance - just over half the html5lib tests passing.

From there, it was a steady but slow climb. Each commit tackled another piece of the spec:

Foreign content handling (SVG and MathML have their own rules)
Foster parenting (table elements need special handling)
The full adoption agency algorithm (for misnested formatting elements)
Template element handling
Entity parsing with partial matches

After a few hours of this, we were at 97.3% compliance. Then progress slowed dramatically.

At 99.6% (1763/1770 tests), the model hit a wall. The remaining 7 tests were genuinely hard edge cases - complex template/table/SVG interactions, multiply-nested table foster parenting, and a new HTML feature called <selectedcontent> that I'd never heard of before. The model choose instead to documented them as "remaining edge cases" and moved on to adding features like CSS selectors and streaming parsing, just skipping implementing these entirely.

When I noticed, it was actually quite difficult to make the agent go back and include them, it really wanted to just mark them as skipped and consider that a pass.

Eventually I stopped the agent, had a second fresh agent clear away and delete all the skipping code, and then restarted the first agent telling it to resume working through the failing tests. Which now included the ones it tried to skip.

The fixes turned out to be surprisingly small. The select mode marker handling needed a single check. The nested table problem needed explicit handling in two insertion modes. The final commit that achieved 100% compliance added just 34 lines to the tree builder:

// Select mode marker handling (webkit02.dat test 49):
// Insert marker when entering select mode to prevent reconstruction
// of formatting elements from outside select
case .select:
    self.activeFormattingElements.append(.marker)

Eventually: 1,770/1,770 tree construction tests passing.

Performance Reality Check

Having achieved 100% test coverage I thought I was now done (minus setting up some documentation and API examples).

I ran the benchmarks comparing Swift, JavaScript, and Python implementations parsing the same HTML files as I had all 3 downloaded to my computer already:

Implementation	Parse Time
Swift	308ms
Python	417ms
JavaScript	108ms

Swift was only 1.4x faster than Python. JavaScript was 2.9x faster than Swift.

This was not the result I expected from a compiled language. I knew python was generally slow, but also expected js to be slow. So expected the Javascript might be around the same or a bit faster.

I wasn't expecting the js version to by 4x faster than the python version, and for the Swift version to be barely faster than the Python.

If you remember one of the lines from Swift's introduction (which has been shown to be not true in practice over an over again since then) was "python like code, with speed faster than C".

However, I then remembered, Swift's string are famously "spec correct" and slow. And HTML parsing is mostly string processing.

The Fuzzer Finds a Bug

Before investigating performance, I wanted to check my library was reliable and stable, before worrying about making it fast. So I also ran the test files for the crash scenarios Emil's fuzzer had found crashes in his implementation against my implementation.

Fortunately, none of those crashed my library, but I also might just have been lucky. So I had Claude Code write and run ran a swift fuzzer of our own - generating millions of random and malformed HTML documents to test parser robustness.

After running those hundred's of thousands of fuzzer tests, the fuzzer found 1 crash. I had told the coding agent not to fix any fuzzer crashes it found, but to investigate and create a test file which reproducible the crash reliably.
Once we had that test file setup I had it fix the issue and resume fuzzing the library.

Input: <table></table><li><table></table>
Context: select fragment
Result: SIGSEGV (segmentation fault)

The parser was hitting infinite recursion. When parsing table-related tags inside a select fragment context, popUntil("select") had no effect (select wasn't on the stack, just the context), and resetInsertionMode() would restore select mode, causing an infinite loop when reprocessing the tag.

Interestingly, simpler variants didn't crash:

<table></table> in select - fine
<table></table><li> in select - fine
<li><table></table> in select - fine
<table></table><li><table></table> in select - crash

It was the specific sequence that triggered infinite recursion. The fix was checking if select was context-only and, if so, clearing the context and switching directly to inBody mode.

This is why fuzzing is useful, especially for testing input handling - the html5lib tests don't cover every possible fragment parsing scenario.

The Performance Hunt

With the crash fixed, all the html5lib test passing, none of the other test files in the other repos causing crashes or failures, and my own fuzzer failing to find any issues, I turned back to performance. The 2.9x gap with JavaScript was troubling me.

I now had a reliable, stable swift implementation of the html5 spec with a justhtml style API to parse and process it. So I didn't want to break that, or make the code an unrecognisable mess in pursuit of performance.

So I created a turbo branch and started profiling. The initial breakdown showed roughly 50/50 time split between tokenizer and tree builder. Both needed work to speed this up.

Swift Strings to raw UTF-8 bytes

Unfortunately Swift strings are slow, and if you want to go faster one of the most straightforward decisions to avoid them.

So I gave the agent some benchmarking tools, sample code and guidance on how to research what was causing the time and told the agent to carefully profile, experiment, benchmark and then re-run the tests and fix any regressions caused. If it found a faster solution which still passed all the test to commit it and then start again, profiling to find the now slowest part and experiment with ways to optimise that.

The first optimization was switching from Swift's String.Index to raw UTF-8 bytes. Swift's string handling is Unicode-correct but expensive - String.Index advancement is O(n) because it has to account for grapheme clusters.

// Before: String.Index iteration
for ch in html { ... }  // O(n) per access

// After: Byte-level access
let bytes = ContiguousArray<UInt8>(html.utf8)
for i in 0..<bytes.count {
    let byte = bytes[i]  // O(1) access
}

Converting to ContiguousArray<UInt8> gave immediate gains:

302ms → 261ms (14% faster)

Batch text insertion

Next was batch text insertion. The tree builder was creating a new text node for every character token from the tokenizer. So instead now it builds up a buffer of raw bytes coming in, until it gets to the character it needs to switch on, then at that point it converts the buffer its been building up into a string object. Instead of repeatedly turning each char into a string and appending the string with the pending buffer string each time. So we only pay the object creation and conversion cost once.

Coalescing consecutive characters before insertion:

261ms → 182ms (30% faster)

Avoiding Memory Allocation

Then I directed it to look into memory allocation and ways to reduce how many objects we are creating in the loops, the agent found inline array literals in hot paths:

// This creates a temporary array on EVERY tag
if ["td", "th", "tr"].contains(name) { ... }

Moving these to module-level Set<String> constants eliminated thousands of allocations per document:

182ms → 172ms (6% faster)

Batch scanning for tag names

The biggest single win came from batch scanning for tag names. Instead of building names character-by-character with string concatenation, scan ahead to find the delimiter and extract the whole chunk at once, again avoiding paying the string creation and modification cost more than once:

159ms → 118ms (26% faster)

Removing the other inline array literals

The final optimizations involved hunting down every remaining inline array literal in the tree builder. 7 more were found in processStartTagInBody that were creating arrays on every single tag. For a "span" tag (common), the code was:

Create and search a 10-element array (head tags) - no match
Create and search a 26-element array (block tags) - no match
Create and search an 11-element array (table tags) - no match
Keep searching...

Converting these to static Sets:

118ms → 98ms (17% faster)

Final Result

One more pass with buffer reuse instead reassigning the object to a new empty [:] to clear it brought the final improvement to:

302ms → 97ms - a 3.1x speedup.

The Swift implementation now matched or just barely beat the JavaScript. But it took ~20 optimization commits, detailed profiling, and completely throwing out Swift's string APIs in favor of raw byte manipulation. The JavaScript version does none of this - it just uses simple, readable character-by-character processing with array.push().

More on that in a follow-up post about V8's performance.

Merging and Polish

With the turbo branch proving out, I merged it back to main. The rest of work was setting up documentation and examples:

DocC documentation with GitHub Pages deployment
Example CLI tools (htmltool, html2md, extractlinks, fetchpage)
A Swift Playground for interactive experimentation
Memory usage benchmarking of the 3 languages implementations.
README improvements

The Final Result

8,953 tests passing:

Tree construction: 1,831
Tokenizer: 6,810
Serializer: 230
Encoding: 82

Performance: 97ms to parse 2.5MB of HTML across 5 Wikipedia articles. 4x faster than Python, matching JavaScript.

Zero dependencies. Pure Swift using only Foundation (and even that lightly now String isn't being used much).

Fuzz tested. Millions of malformed documents without crashes.

How Do Other Swift Libraries Compare?

After finishing, I was curious how existing Swift HTML libraries fared against the same test suite. So downloaded and benchmarked the ones which supported linux. The results were:

Library	Pass Rate	Notes
swift-justhtml	100% (1831/1831)	None
Kanna	94.4% (1542/1633)	Uses libxml2 (HTML 4.01 parser)
SwiftSoup	87.9% (1436/1633)	Infinite loop on 197 tests
LilHTML	47.4% (775/1634)	Crashes on 52% of tests

Kanna and LilHTML both use libxml2 under the hood. libxml2 is a C library that implements HTML 4.01 parsing, not the WHATWG HTML5 specification that browsers actually now use. It's fast (native C code) but won't handle modern HTML correctly.

SwiftSoup is a port of Java's Jsoup library. It hit infinite loops on all 197 tests in tests16.dat - edge cases involving script tags.

LilHTML was the most surprising. It wraps libxml2 but crashes on over half the test inputs due to unhandled NULL returns. Not sure why the big difference to Kanna (it's possible I'm doing something wrong with the test for 1 of them. I didn't look into this that closely).

What I Learned

HTML5 parsing is harder than it looks. It looks like simple XML parsing, but that went out the window when XHTML was abandoned, now the specification is thousands of pages. And the edge cases interact in surprising ways.

Test suites are invaluable. The html5lib tests gave concrete pass/fail feedback for every change. Without them, LLM coding agents would not be able to tackle a task of this size!

Given tools and feedback LLM agents are now incredible powerful!: The key is packaging your task in a format which lets the llm check how it is going, and get feedback on what it did right or wrong so it can iterate it's way to a solution.

Fuzzing finds real bugs. The select fragment crash would never have been caught by the standard test suite alone.

Swift's performance isn't automatic. Being a compiled language doesn't guarantee speed unfortunately. Understanding memory allocation, string handling, and avoiding unnecessary work matters a lot if you want fast Swift code. Otherwise Swift can quite easily end up as slow to run as it is to compile.

V8 is WAY faster than I thought! Even the super optimised version of swift-justhtml with minimal String use, raw byte processing, no inline arrays or anything else which could cause allocations still only draws level to the default straightforward js implementation running via V8 in node.

AI coding agents are transformative for this kind of work. To reiterate: The tight feedback loop - run tests, see failures, propose fix, repeat - is exactly what they're good at. I spent my time on architecture decisions and code review rather than typing out implementations.

Which is basically the same conclusion Emil reached.

How to Build Text Based Image Search

2025-10-27T06:21:40.012243+00:00

Searching for images is an image classification task. There are 2 types of image classification:

Normal Image classification.
Classify an image into a preset list of categories.
Zero-shot Image classification.
Classify an image into classes previously unseen during training of a model.

Normal Image classification

These models are trained with a list of categories and predicts which of these categories the image falls into.

from transformers import pipeline
# Create the pipeline with the `google/vit-base-patch16-224` model
clf = pipeline("image-classification", "google/vit-base-patch16-224")
# Get the categories
results = clf("temp/lake-smaller.jpg")
print(results)

{'label': 'lakeside, lakeshore', 'score': 0.9137738943099976}
{'label': 'seashore, coast, seacoast, sea-coast', 'score': 0.017617039382457733}
{'label': 'promontory, headland, head, foreland', 'score': 0.016670119017362595}
{'label': 'valley, vale', 'score': 0.007515666540712118}
{'label': 'dam, dike, dyke', 'score': 0.004389212466776371}

You can then save these categories (at least the ones scoring above 0.1) in a database and then use these to search and for images matching the search term.

However, the above approach will match "lake" but not "water", or "river".

The google/vit-base-patch16-224 is a small 86.6M params (346 MB) size model which predicts one of the 1000 ImageNet classes.

There is also the larger google/vit-large-patch16-224 model (1.22 GB) which is more accurate, but larger and slower.

The name means:

vit: Vision Transformer (ViT).
base: the size of the model.
patch16: Images are presented to the model as a sequence of fixed-size patches (resolution 16x16), which are linearly embedded.
224: All images are resized to a resolution of 224x224 pixels before being fed to the model.

Zero-shot Image classification

In order to get arbitrary matching and score the similarity between the search term and the image, regardless of what the search term is you need a zero-shot image classification model.

These are embedding models which have a shared embedding space where the embeddings of text representations of an image, and the image data itself are trained to end up with similar coordinates in the embedding space.

This means when you calculate the distance between the search terms embedding, and the embeddings of several images you get how similar each one is to the search term.

This also means "ice" will be similar to photos of "snow" even though the words are different.

2 good models for this are:

OpenAI CLIP model.
Google SigLIP 2

CLIP large is a 0.4B (1.71GB) model. The smaller openai/clip-vit-base-patch32 is only 605MB and is the more popular version. Both as fairly small, quick models.
However, the models were released 2021, and since then newer models have been trained which surpass them in benchmarks.

SigLIP 2 is a family of models released by Google in February 2025, which claims to outperform their previous counterparts at all model scales in core capabilities.
The models come in various sizes: ViT-B (86M), L (303M), So400m (400M), and g (1B).
There are also fixed size versions which resize and crop images to a fixed square resolution, or flexible resolution and aspect ratio versions, which divide the image into the fixed size tiles, but divide up images of different sizes and resolutions.

FG-CLIP 2 is a new model released in October which claims to further improve image classification results.

It comes in base (1.54GB), large (3.59GB), and so400m (4.62GB) versions.

Simple Zero-shot Classification Example

If you just want to classify an image into several categories which you provide the huggingface transformers pipeline.

from transformers import pipeline
classifier = pipeline("zero-shot-image-classification", model="openai/clip-vit-large-patch14-336")
candidate_labels = [
    "2 cats",
    "a plane",
    "a remote",
    "a bear",
    "a ocean",
    "a water",
    "a river",
    "a lake",
    "a pond",
    "rain",
    "a mountain",
    "a car",
]
image = load_image("temp/lake-smaller.jpg")
results = image_classifier(image, candidate_labels)
print(results)

{'score': 0.9330312609672546, 'label': 'a lake'}
{'score': 0.02649594098329544, 'label': 'a water'}
{'score': 0.023041551932692528, 'label': 'a river'}
{'score': 0.01358820404857397, 'label': 'a pond'}
{'score': 0.0017832380253821611, 'label': 'a ocean'}
{'score': 0.0006893535610288382, 'label': 'a mountain'}
{'score': 0.0004389677778817713, 'label': 'a phone'}
{'score': 0.00038122545811347663, 'label': 'a remote'}
{'score': 0.0002814349136315286, 'label': 'a plane'}
{'score': 0.00019272200006525964, 'label': 'rain'}
{'score': 3.4059521567542106e-05, 'label': 'a car'}
{'score': 3.0467408578260802e-05, 'label': 'a bear'}
{'score': 1.143596091424115e-05, 'label': '2 cats'}

Using the google/siglip2-so400m-patch16-naflex model we get roughly the same results, though the scores are all much lower (the order is still good though).

from transformers import pipeline
image_classifier = pipeline(model="google/siglip2-so400m-patch16-naflex", task="zero-shot-image-classification")
outputs = image_classifier(image, candidate_labels)
print(outputs)

{'score': 0.1465340554714203, 'label': 'a lake'}
{'score': 0.0038345942739397287, 'label': 'a water'}
{'score': 0.001178617007099092, 'label': 'a river'}
{'score': 0.00041009532287716866, 'label': 'a ocean'}
{'score': 0.00018598142196424305, 'label': 'a mountain'}
{'score': 8.42182053020224e-05, 'label': 'a pond'}
{'score': 3.1798244890524074e-05, 'label': 'rain'}
{'score': 1.0240605661238078e-05, 'label': 'a plane'}
{'score': 1.7888405636767857e-06, 'label': '2 cats'}
{'score': 1.6506612610101001e-06, 'label': 'a bear'}
{'score': 5.040944870415842e-07, 'label': 'a phone'}
{'score': 4.662872470362345e-07, 'label': 'a car'}
{'score': 3.0268274997524713e-08, 'label': 'a remote'}

Text to Image Search

However, if we want to build up a database of image embeddings and do image text and image based searches against those embeddings we need to get the embeddings from these models.

Example Images

Generate Image Embeddings

import torch
from PIL import Image
from transformers import CLIPProcessor, CLIPModel
import numpy as np
import torch.nn.functional as F

def generate_image_embedding(image_path):
    """
    Generate an L2-normalized image embedding.

    Args:
        image_path: Path to the image file

    Returns:
        Tuple of (filename, embedding vector as numpy array)
    """
    device = get_device()
    model, processor = get_clip_model()

    # Load and convert image to RGB
    image = Image.open(image_path).convert("RGB")

    # Generate embedding
    with torch.no_grad():
        inputs = processor(images=image, return_tensors="pt").to(device)
        features = model.get_image_features(**inputs)
        # L2 normalize for proper cosine similarity
        features = F.normalize(features, p=2, dim=-1)
        embedding = features.cpu().numpy()[0]

    # Extract just the filename from the path
    filename = image_path.split("/")[-1]

    return filename, embedding

Generate Text Images.

def generate_text_embedding(query_text):
    """
    Generate an L2-normalized text embedding.

    Args:
        query_text: Text query string

    Returns:
        Embedding vector as numpy array
    """
    device = get_device()
    model, processor = get_clip_model()

    with torch.no_grad():
        inputs = processor(text=[query_text], return_tensors="pt", padding=True).to(device)
        features = model.get_text_features(**inputs)
        # L2 normalize
        features = F.normalize(features, p=2, dim=-1)
        embedding = features.cpu().numpy()[0]

    return embedding

Load the image embedding

def load_images():
    """
    Load all images and generate their embeddings.

    Returns:
        List of tuples: [(filename, embedding), ...]
    """
    print("Loading and embedding images...")
    embeddings = []

    for image_path in IMAGE_PATHS:
        print(f"  Processing {image_path.split('/')[-1]}...")
        filename, embedding = generate_image_embedding(image_path)
        embeddings.append((filename, embedding))

    print(f"Generated embeddings for {len(embeddings)} images\n")
    return embeddings

Compare image embedding.

def search(query_text, image_embeddings):
    """
    Search for images matching the text query.

    Args:
        query_text: Text description to search for
        image_embeddings: List of (filename, embedding) tuples

    Returns:
        List of (filename, similarity_score) tuples sorted by similarity (best first)
    """
    print(f"Searching for: '{query_text}'")

    # Generate text embedding
    text_embedding = generate_text_embedding(query_text)

    # Compare with all image embeddings
    results = []
    for filename, image_embedding in image_embeddings:
        similarity = cosine_similarity(text_embedding, image_embedding)
        results.append((filename, similarity))

    # Sort by similarity (highest first)
    results.sort(key=lambda x: x[1], reverse=True)

    print("\nResults:")
    print("-" * 50)
    for filename, similarity in results:
        print(f"{filename:30s} | Similarity: {similarity:.4f}")
    print("-" * 50)
    print()

    return results

Example compare the images.

image_embeddings = load_images()
search("a beautiful lake with mountains", image_embeddings)

Results

Searching for: 'a beautiful lake with mountains'

Results:
----------------------------------------------------
lake-smaller.jpg               | Similarity: 0.2865
beech-tree.jpg                 | Similarity: 0.1330
cats.jpg                       | Similarity: 0.1164
bmw-i7.jpg                     | Similarity: 0.1073
----------------------------------------------------

Searching for: 'luxury car'

Results:
----------------------------------------------------
bmw-i7.jpg                     | Similarity: 0.2513
beech-tree.jpg                 | Similarity: 0.1920
cats.jpg                       | Similarity: 0.1818
lake-smaller.jpg               | Similarity: 0.1685
----------------------------------------------------

Searching for: 'tree in nature'

Results:
----------------------------------------------------
beech-tree.jpg                 | Similarity: 0.2992
lake-smaller.jpg               | Similarity: 0.2080
cats.jpg                       | Similarity: 0.1543
bmw-i7.jpg                     | Similarity: 0.1377
----------------------------------------------------

Searching for: 'water and nature'

Results:
----------------------------------------------------
lake-smaller.jpg               | Similarity: 0.2672
beech-tree.jpg                 | Similarity: 0.2209
cats.jpg                       | Similarity: 0.1669
bmw-i7.jpg                     | Similarity: 0.1512
----------------------------------------------------

Search By Image

You can also do an image search by generating an image embedding and using that for the comparisons instead of using the text embedding.

def search(image_path, image_embeddings):
    print(f"Searching for: '{image_path}'")

    # Generate text embedding
    _, search_image_embed = generate_image_embedding(image_path)

    # Compare with all image embeddings
    results = []
    for filename, image_embedding in image_embeddings:
        similarity = cosine_similarity(search_image_embed, image_embedding)
        results.append((filename, similarity))

    # Sort by similarity (highest first)
    results.sort(key=lambda x: x[1], reverse=True)

    print("Results:")
    print("-" * 50)
    for filename, similarity in results:
        print(f"{filename:30s} | Similarity: {similarity:.4f}")
    print("-" * 50)
    print()

    return results

As the image embeddings and text embeddings are designed to end up in a similar location in the multi-dimensional embedding space either can be used for lookups and get good results.

References

Image Classification References

Zero-Shot Image Classification References

huggingface docs: Zero-Shot Image Classification

CLIP References

SigLIP 2 References

FG-CLIP 2 References

Talk: Claude Code & The Evolution of Agentic Coding

2025-07-16T13:38:15.478599+00:00

Boris Cherny's talk on how Claude Code represents a new paradigm in AI-powered development tools.

Boris Cherny, creator of Claude Code at Anthropic, presents a compelling vision of how programming is evolving in the age of AI. He traces the evolution from punch cards to natural language programming, explaining why Claude Code takes an intentionally simple, unopinionated approach to AI-powered development.

The talk covers Claude Code's core features—terminal integration, IDE support, GitHub integration, and SDK—while sharing practical tips for effective use, including code Q&A, teaching Claude your tools, and leveraging Plan Mode for complex tasks.

YouTube Video (17m)

Overview

Boris Cherny, creator of Claude Code at Anthropic, presents a compelling vision of how programming is evolving in the age of AI. In this talk from the AI Startup School in San Francisco, he explores the tension between exponentially improving AI models and the products we build to harness them. Claude Code's intentionally simple, unopinionated approach represents a fundamental shift in how developers interact with AI—providing direct access to powerful models without unnecessary abstraction layers.

Talk Chapters

0:00 The model is moving really fast, on exponential. The product is struggling to keep up.
1:25 Programming evolution: from punch cards to natural language
5:45 Ed, the first text editor - still available on Unix systems 50 years later
8:22 GitHub Copilot and Devin: the leap to AI-powered coding
10:30 Claude Code's philosophy: intentionally simple and general
12:15 Terminal, IDE, GitHub integration, and SDK features
14:00 Tips for effective use: code Q&A, teaching tools, and workflows
15:30 New Plan Mode feature and the power of context
16:40 Multi-Claude workflows and the future

Talk Summary

The Exponential Growth Challenge

"The model is moving really fast. It's on exponential. It's getting better at coding very, very quickly," Boris begins. "And the product is kind of struggling to keep up. We're trying to figure out what product to build that's good enough for a model like this."

This tension between rapidly advancing AI capabilities and the products we build to harness them is at the heart of Claude Code's philosophy. Rather than trying to predict the perfect interface for AI-assisted coding, Claude Code takes a deliberately minimal approach—providing the "bare minimum" to let developers experience and experiment with the model directly.

Getting Started with Claude Code

For those new to Claude Code, getting started is straightforward:

Head to claude.ai/code to install it
Run the installation from NPM
As of the talk, Claude Code supports both Claude Pro and Claude Max plans

"Just try it out. Tell us what you think," Boris encourages, emphasizing the experimental nature of the tool.

The Evolution of Programming Languages

To understand where we're heading, Boris takes us on a journey through programming history. "Programming is changing, and it's changing faster," he notes.

The evolution began in the 1930s and 40s with physical switchboards—there was no such thing as software. By the 1950s, punch cards became the norm. Boris shares a personal connection: "My grandpa, actually, in the Soviet Union, he was one of the first programmers in the Soviet Union. And my mom would tell me stories about... he would bring these big stacks of punch cards home from work and she would like draw all over them with crayons."

Programming evolved from:

Hardware (switchboards)
Physical media (punch cards)
Software with assembly language
Higher-level languages (COBOL, C)
The explosion of language families in the early 90s (Haskell, JavaScript, Java, Python)

"I think nowadays, if you kind of squint, all the languages sort of look the same," Boris observes. "Like when I write TypeScript, it kind of feels like writing Rust, and that kind of feels like writing Swift, and that kind of feels like writing Go. The abstractions have started to converge a bit."

The Evolution of Programming UX

From Punch Cards to Text Editors

The IPMO29 was "like the MacBook of the time for programming punch cards." This physical machine represented the state-of-the-art in programming interfaces before the advent of text editors.

Then came Ed, the first text editor, invented by Ken Thompson at Bell Labs. "If you open your MacBook, you can actually still type Ed. This is still distributed on Unix as part of Unix systems," Boris notes with amazement. "And this is crazy, because this thing was invented like 50 years ago."

Ed was incredibly primitive by today's standards:

No cursor
No scroll back
No fancy commands
No type ahead
Built for teletype machines that literally printed on paper

The GUI Revolution

In 1980, SmallTalk 80 represented "one of the first, I think the first graphical interface for programming." Remarkably, it had features we still struggle with today: "For anyone that's tried to set up live reload with React or Redux or any of this stuff, this thing had live reload in 1980, and it worked."

Visual Basic in 1991 "was the first code editor that introduced a graphical paradigm to the mainstream." While tools like SmallTalk existed, Visual Basic brought GUI-based programming to the masses.

Modern AI-Powered Evolution

GitHub Copilot marked "a big jump forward with single-line type ahead and then multi-line type ahead," using AI to suggest code completions.

Devin introduced the next major abstraction: "to program, you don't have to write code, you can write natural language, and that becomes code." This concept, which developers had been trying to achieve for decades, finally became practical with advanced AI models.

Claude Code's Philosophy

"With all this in mind, Claude Code's approach is a little different," Boris explains. "It's to start with a terminal and to give you as low-level access to the model as possible in a way that you can still be productive."

The key principles:

Unopinionated: No flashy UI or unnecessary scaffolding
Simple and General: Shows off the model's capabilities without getting in the way
Model-First: Lets developers experience the raw power of the AI model

"We actually just don't know. Like, we don't know what the right UX is. So we're starting simple," Boris admits candidly.

The guiding principle? "The more general model always wins, and the model increases in capability exponentially."

Claude Code Features

Terminal Product

The core Claude Code experience is deliberately minimal: "You can install Claude Code, and then you just run claude in any terminal." It's truly unopinionated—working in iTerm2, WSL, over SSH and TMUX sessions, VS Code terminal, Cursor terminal, anywhere.

IDE Integration

When running Claude Code in an IDE, it provides enhanced features:

Beautiful diff displays within the IDE
Diagnostic information ingestion
Minimal polish compared to specialized tools like Cursor or Windsurf

"This is to let you experience the model in a low-level, raw way," Boris emphasizes.

GitHub Integration

A few weeks before the talk, Anthropic announced GitHub integration. The setup is simple:

Open Claude
Run the slash command to install the GitHub app
Pick your repository
Claude runs on your compute—your data stays local

SDK for Custom Integrations

For maximum flexibility, Claude Code offers an SDK: "You can build it however you want. People have built all sorts of UIs, all sorts of awesome integrations."

Boris shares a personal use case: "I'll take my GitHub logs, or my sorry, my GCP logs. I'll pipe it into claude -p, because it's a Unix utility, so you can pipe in, you can pipe out, and then I'll JQ the result."

Tips for Using Claude Code Effectively

Boris shares practical advice for getting the most out of Claude Code:

1. Start with Code-Based Q&A

"At Anthropic, we teach Claude Code to every engineer on day one, and it shortened onboarding times from like two or three weeks to like two days."
Engineers can simply ask Claude questions about the codebase.

2. Teach Claude to Use Your Tools

"With every IDE, there's sort of like a plug-in ecosystem... For this new kind of coding tool, it can just use all your tools."
The process is:

Give it bash tools or MCP tools
Run --help on a CLI tool
Put the information in claude.md
Claude now knows how to use the tool

3. Effective Workflows

Traditional coding tools focused on writing code, but AI-powered tools do much more:

Exploration and Planning: Have Claude explore and make a plan before writing code. Use extended thinking when there's substantial context.

Test-Driven Development (TDD): "I know I try to use TDD. It's like, it's pretty hard to use in practice. But I think now with coding tools, it actually works really well."
The workflow:

Tell Claude to write tests first
Explicitly state the tests won't pass yet
Commit the tests
Then write the code and commit

Provide Iteration Targets: "If Claude has a target to iterate against, it can do much better."
This could be:

Unit tests
Integration tests
Screenshots from iOS simulator or Puppeteer
Even camera feedback from 3D printers!

New Features: Plan Mode

"Today we launched plan mode in Claude Code," Boris announces. By hitting Shift+Tab, Claude switches to Plan Mode where it:

Creates a plan for your request
Waits for approval before executing
Allows you to review and modify the approach

The Power of Context

"Give Claude more context," Boris emphasizes. Several methods make this easy:

claude.md files: Place in your home folder or project
Slash commands: Put markdown files in .claude/slash-commands folders
Memory function: Type # to ask Claude to memorize something

"You can tell this is still pretty rough. This is our first version, but it's the first version that works," Boris acknowledges, inviting feedback on the UX.

The Future: Multi-Claude Workflows

During the Q&A, an interesting question arose about managing multiple Claude instances. Power users are already "multi-Clauding"—running several terminal tabs with different Claude instances working in parallel on different parts of a codebase or different work trees.

"If you do want to coordinate, the best way is just ask them to write to a markdown file, and that's it," Boris suggests for inter-Claude communication.

Conclusion

Claude Code represents a fundamental shift in how we think about AI-assisted programming. Rather than trying to build the perfect IDE or the ultimate coding assistant, it provides direct, unopinionated access to increasingly powerful AI models. This approach acknowledges a crucial reality: in a world where AI capabilities are growing exponentially, the most valuable tool might be the simplest one—one that gets out of the way and lets developers experiment, learn, and discover new ways of working with AI.

As Boris concludes, the evolution of programming—from punch cards to natural language—is accelerating. Claude Code positions itself not as the final answer, but as a flexible foundation for whatever comes next. In the race between exponentially improving models and the products built to harness them, sometimes the winning move is to build less, not more.

The message is clear: the future of coding isn't just about better tools—it's about fundamentally rethinking the relationship between human creativity and AI capability. Claude Code is an invitation to join that exploration.

Notes

Claude Code shortened onboarding times at Anthropic from 2-3 weeks to just 2 days
The tool is intentionally "unopinionated" - it works in any terminal environment
Power users are already "multi-Clauding" - running multiple instances in parallel
Plan Mode (Shift+Tab) was launched the day of this talk
Claude Code can be taught to use any CLI tool by showing it the --help output

YouTube Description

A ten thousand foot view of the coding space, the UX of coding, and the Claude Code team's approach.

About Boris Chemy Created Claude Code. Member of Technical Staff @Anthropic. Prev: Principal Engineer @Meta, Architect @Coatue. Author, OReilly's Programming TypeScript.

Chapters on YouTube

0:00 - Introduction & the exponential model challenge
0:49 - Getting started with Claude Code
1:25 - Evolution of programming languages
3:45 - Evolution of programming UX
4:30 - IPMO29: The MacBook of punch cards
5:00 - Ed: The first text editor
6:20 - SmallTalk 80 & GUI revolution
7:10 - Visual Basic brings GUI mainstream
8:00 - GitHub Copilot & AI evolution
8:45 - Devin & natural language programming
10:00 - Claude Code's philosophy
11:30 - Terminal product demo
12:15 - IDE integration
12:45 - GitHub integration
13:15 - SDK for custom integrations
14:00 - Tips: Code-based Q&A
14:30 - Teaching Claude your tools
15:00 - Effective workflows & TDD
15:45 - Plan Mode launch
16:15 - The power of context
16:40 - Q&A: Multi-Claude workflows

References:

Talk: Agentic Coding - The Future of Software Development with Agents

2025-07-04T02:10:28.919128+00:00

Video: https://www.youtube.com/watch?v=nfOVgz_omlU
Title: Agentic Coding - The Future of Software Development with Agents
Speaker: Armin Ronacher (Flask creator, former Sentry engineer)
Duration: 37 minutes
Topic: Real-time AI agent collaboration in software development

Overview

Armin Ronacher, the creator of Flask, has been one of the people diving headfirst into LLM agentic programming recently and has put together a talk going over some of the main insights he's learnt from working with these tools over the last few months.

The agents are now good enough to collaborate. Running independently for hours on tasks, changing how we approach software development.

One key takeaway, which I've also found recently in my own experience is: Claude Code.
Claude Code is finally over the threshold of being good enough, and reliable enough, to fundamentally change how you use it.

The talk goes over how to make use of these new tools, where they are strongest, and where they are still lacking. As well as some tips on how to use them effectively.

Talk Chapters

00:00 - Introduction & Speaker Background
02:42 - What Is Agentic Coding?
05:16 - Why Now? The Perfect Storm
07:37 - Claude Code vs. Cursor
13:37 - Practical Usage Beyond Programming
15:47 - Programming Language Recommendations
18:06 - Development Environment Optimization
20:50 - MCP (Model Context Protocol) Discussion
23:58 - Context Management & Sub-agents
26:29 - Advanced Development Patterns
32:05 - Creative Use Cases & Examples
36:28 - Future Vision & Conclusion

Talk Summary

Introduction & The Addiction Factor (0:00-2:42)

Armin admits he's fallen down a rabbit hole of AI experimentation since May. (The talk is recorded at midnight, emphasizing just how consuming this technology has become)

What Is Agentic Coding? (2:42-5:04)

Agentic coding represents a fundamental shift from autocomplete tools like Cursor or GitHub Copilot to true real-time collaboration between humans and AI agents. These agents actively participate in the entire coding process, can run for hours on complex tasks through sub-agents, and manage context effectively.

Why Now? (5:04-7:35)

Anthropic's Sonnet 4 and Opus 4 models have been specifically trained on tool usage
Claude Code is a step above everything else at the moment.
The subscription pricing model ($100-200) makes these tools far more cost-effective than paying for API tokens.

Claude Code vs. Cursor (7:35-13:34)

Claude Code supports longer sessions, and is more ready to read files and use tools. Unlike Cursor's editor-focused approach, Claude Code is a terminal interface, so supports, SSH access, composition and remote access easier. The terminal-based approach enables better experimentation and even nesting agents within each other.

Practical Usage Philosophy (13:34-15:39)

Armin uses Claude Code far beyond programming - for CI setup, machine configuration, creating tools on the spot, and even web browsing. The trust factor is crucial: Claude Code feels safer than Cursor, with fewer accidental deletions and better protective measures.

Programming Language Recommendations (15:39-18:06)

Simple project structures, well-known languages and frameworks work best with agents. Go, PHP, and "basic Python" (simple scripts with minimal dependencies) are ideal because they have less ecosystem churn, clear patterns, and rely heavily on standard libraries. Using long, descriptive function names to reduce AI confusion and prevent code duplication across the codebase.

Development Environment Optimization (18:06-20:50)

The quality of your development environment is the primary factor determining agent success.
Agents will misuse tools, so those tools must handle misuse gracefully with clear error messages.
Speed matters - agents will abandon slow tools and stop using them entirely.
Agents want simple log files and good exceptions, not complex telemetry setups.
Providing ways for agents to create scratch tools and throwaway code is essential for experimentation.

MCP (Model Context Protocol) Reality Check (20:50-23:58)

Command-line tools are more effective than MCPs.
Armin only uses one MCP (Playwright) and relies on command-line tools for everything else.

CLI tools win because they can be embedded in scripts, and agents understand code writing better than MCP interactions.
MCPs create context bloat and are annoying for humans to use and debug.

He's since written a blog post exploring this point more thoroughly.
"Tools: Code Is All You Need"

Context Management Strategies (23:58-26:29)

Effective context management requires custom tools like "MakeGoMethods" that summarize entire codebases efficiently.
A practical approach is providing access to the last 20 lines of logs with clear instructions for getting more detail when needed.
Sub-agents can break down large tasks and run for 12+ hours, but requires careful planning, then creating markdown plans for step-by-step execution.
Context rot - where failed attempts accumulate - causes backtracking and confusion, so aborting when running low on context is often better than letting the system compact.

Advanced Development Patterns (26:29-32:05)

Unified log files (forwarding browser console logs and server logs into the same file) are great for the LLM to understand what is happening.

Creative Use Cases (32:05-37:19)

Claude Code excels at diverse tasks: calling Gemini CLI for PDF processing and summarization, automatically downloading videos and integrating them into presentations, automating Google Chrome to create marketplace listings with proper descriptions and pricing (using Playwright), and performing system administration like Git reconfiguration.

The talk itself was put together using Claude code.
It's an opinion that's shared by others: "Claude Code is My Computer"

Key Takeaways

For Developers

Start Simple: Go, PHP, and basic Python work best with agents
Optimize Your Dev Environment: Agent success depends on tool quality and speed
Embrace Terminal-Based Tools: More flexible than editor integrations for experimentation
Context Management is Important: Use sub-agents, CLI tools, and avoid context rot

Tools Mentioned

Claude Code: Primary agentic coding tool discussed
Cursor: Great IDE integration, but much more limited agent mode that also goes off track more.
OpenCode: Alternative tool mentioned
Gemini CLI: Used for specific tasks like PDF processing
Playwright: Browser automation tool, only MCP the speaker actively uses

The talk emphasizes that while we're in the early days, the combination of advanced LLMs with agentic workflows represents a shift in how software development can be approached.

References

Talk: Andrej Karpathy: Software Is Changing (Again)

2025-06-21T02:02:15.408206+00:00

Andrej Karpathy: Software Is Changing (Again)
Andrej Karpathy's keynote at AI Startup School in San Francisco.

YouTube Video (39m)

Overview

Andrej Karpathy is a machine learning research scientist and a founding member at OpenAI, former Sr. Director of AI at Tesla.

Earlier this year he gave a fascinating the keynote talk at Y Combinator's "AI Startup School in San Francisco" where he talks about the concept of "software 3.0".

The idea is software 1.0 was normal code.
Software 2.0 is machine learning models, which have started being deployed on mass in the last few years.
And now software 3.0 with LLMs. An entire new generation of software.

Talk Chapters

From Andrej Karpathy

0:00 Imo fair to say that software is changing quite fundamentally again. LLMs are a new kind of computer, and you program them in English. Hence I think they are well deserving of a major version upgrade in terms of software.

6:06 LLMs have properties of utilities, of fabs, and of operating systems => New LLM OS, fabbed by labs, and distributed like utilities (for now). Many historical analogies apply - imo we are computing circa ~1960s.

14:39 LLM psychology: LLMs = "people spirits", stochastic simulations of people, where the simulator is an autoregressive Transformer. Since they are trained on human data, they have a kind of emergent psychology, and are simultaneously superhuman in some ways, but also fallible in many others. Given this, how do we productively work with them hand in hand?

Switching gears to opportunities...

18:16 LLMs are "people spirits" => can build partially autonomous products.

29:05 LLMs are programmed in English => make software highly accessible! (yes, vibe coding)

33:36 LLMs are new primary consumer/manipulator of digital information (adding to GUIs/humans and APIs/programs) => Build for agents!

Talk Summary:

Overall Talk Topic

Andrej Karpathy presents a case that software is undergoing a foundational transformation. Large Language Models (LLMs) represent a new kind of computer—programmable in natural language. This shift is significant enough to justify calling it a major new version of software: “Software 3.0.” The talk explores the architecture of LLMs, their emergent psychology, the nature of current opportunities, and the best strategies for building with and for them.

Section 1: Software is Fundamentally Changing (0:49)

LLMs are not just models—they are a new type of computer. They are programmable in English and offer capabilities that differ fundamentally from traditional imperative or neural-network-based software. Given this new paradigm, Karpathy suggests it’s appropriate to upgrade our versioning of software entirely. Software 1.0 was classical logic, Software 2.0 was neural networks with learned weights, and Software 3.0 is built with LLMs using natural language as the programming layer.

Section 2: LLMs as Utilities, Fabs, and Operating Systems (6:06)

Karpathy describes LLMs as being akin to utilities (like electricity), chip fabs (expensive to produce, cheap to use), and operating systems (platforms with APIs and abstractions). Like 1960s computing, most people now use thin clients to access expensive centralized models. While they are currently cloud-hosted utilities, a personal computing model may re-emerge as smaller models become locally deployable.

Section 3: LLM Psychology (14:39)

LLMs behave like "people spirits"—simulations of human minds with a psychology shaped by their training data. They possess strengths like encyclopedic recall and creative problem-solving, but also flaws such as hallucinations, inconsistency, and poor memory. They lack persistence and self-awareness. Effective collaboration with LLMs requires tools that accommodate their abilities while mitigating their weaknesses, especially with tight generate/verify cycles and human oversight.

Section 4: Opportunities: Partially Autonomous Products (18:16)

Because LLMs simulate people, they can drive new forms of human-computer interaction. Products like Cursor and Perplexity wrap LLMs in interfaces that provide automated context, diff-based UIs for human review, and autonomy sliders that balance between user control and model initiative. The most successful applications will support a human-in-the-loop dynamic, allowing LLMs to act semi-independently within guardrails.

Section 5: English as Code – Natural Language Interfaces (29:05)

Programming LLMs in English democratizes access to software creation. Karpathy describes how “vibe coding” and “byte coding” let even non-programmers generate useful applications. However, core infrastructure (e.g., payments, auth) remains complex. While building an iOS app or web tool is now faster than ever, operations are still a bottleneck, suggesting where further automation or tooling improvements could help.

Section 6: Building for Agents (33:36)

LLMs are now a primary manipulator of digital content, joining humans (via GUIs) and traditional programs (via APIs). Developers should design with agents in mind: create markdown-readable interfaces, machine-consumable documentation (like llms.txt), and avoid GUI-only affordances. Early examples like git.ingest and DeepWiki reveal how small design tweaks can expose functionality to agents and unlock powerful behaviors.

Conclusion

LLMs mark the beginning of a new software paradigm. Though they are not perfect, they are powerful and increasingly central to both consumer and developer workflows. Over the next decade, software will shift toward partially autonomous, LLM-driven agents, and the developers who embrace Software 3.0’s strengths—while designing robust human-AI workflows—will define the next era of computing.

Notes

Outages at leading LLM providers act like an “intelligence brownout,” instantly lowering global productivity.
Unlike electricity or GPS, LLMs diffused consumer-first; enterprises and governments are still catching up.
The ability to run small models on a Mac Mini hints at a coming personal-compute era once inference costs fall.
Tesla’s experience shows entire swaths of C++ code can be deleted as neural nets, and soon prompts, absorb that functionality.

YouTube Description

Drawing on his work at Stanford, OpenAI, and Tesla, Andrej sees a shift underway. Software is changing, again. We’ve entered the era of “Software 3.0,” where natural language becomes the new programming interface and models do the rest.

He explores what this shift means for developers, users, and the design of software itself— that we're not just using new tools, but building a new kind of computer.

More content from Andrej: / @andrejkarpathy

Chapters on YouTube

00:00 - Intro
01:25 - Software evolution: From 1.0 to 3.0
04:40 - Programming in English: Rise of Software 3.0
06:10 - LLMs as utilities, fabs, and operating systems
11:04 - The new LLM OS and historical computing analogies
14:39 - Psychology of LLMs: People spirits and cognitive quirks
18:22 - Designing LLM apps with partial autonomy
23:40 - The importance of human-AI collaboration loops
26:00 - Lessons from Tesla Autopilot & autonomy sliders
27:52 - The Iron Man analogy: Augmentation vs. agents
29:06 - Vibe Coding: Everyone is now a programmer
33:39 - Building for agents: Future-ready digital infrastructure
38:14 - Summary: We’re in the 1960s of LLMs — time to build

References:

Talk: Instagram Video Translation System Overview Talk by Meta

2025-05-22T01:01:55.901480+00:00

I came across a really interesting tech talk from an engineer at Meta, going over how the video translation system being implemented on Instagram reels works.

Video: https://www.youtube.com/watch?v=wJOjVr2_WH0
Title: Scaling AI translations at Meta: tackling latency and media processing challenges
Speaker: Jordi Cenzano (Software Engineer, Meta)
Topic: Meta AI’s media translation system with audio and video (lip-sync) translations
Conference: 2025/03/27 MAD Video Tech.

Overview

This talk from an engineer at meta goes through how they built the auto-video translation feature built into Instagram.

The primary model used is the open source seamless-m4t model. But the overall feature requires pre and post processing with other tools and models to get everything feature complete.

Models Used

The pipeline uses more than 10 models, including:

Voice-Ambient Audio Separation Model
Separates speech from background noise for clean source audio.
Language Detection Model
Is the audio even in a supported language?
Sentence Segmentation Model
Seamless is designed for short 10-15s audio clips, so the input audio needs to be broken up into small chunks, which still contain enough context for an accurate translation.
Seamless audio to audio translation model
Does the actual translation from the source language into the target language.
LipSync Model
Takes translated clean audio + original video to generate lip-synced video.
Toxicity Filter / Hallucination Detection Model
Attempts to detect if the translation is more toxic than the input was, and so the translation might have been incorrect.

The talks outlines that more than 10 ML models are used, though only Seamless is named directly.

Translation Pipeline Overview

Upload
1. User uploads video via Instagram or Facebook.
2. Video is uploaded to Meta’s distributed storage system.
Audio Translation
1. Audio Extraction & Decoded into the correct format for processing (AAC → PCM)
2. Pre-Processing:
  • Clean speech extraction (background noise removal)
  • Language detection
  • Sentence segmentation
3. Seamless Model Audio Translation:
  • Input: short PCM audio segments
  • Output: translated PCM audio
4. Post-Processing:
  • Re-align segments (time alignment across languages)
  • (Implied they also adjust the video slightly if more time is needed in the target language?)
  • Recombine segments
  • Re-add ambient noise
  • Re-encode audio (PCM → AAC)
Video (LipSync)
1. Using clean translated audio and original video
2. LipSync Model:
  • Generates new video with synchronized lip movement synced to the new audio
3. Post-process:
  • Overlay visible watermark (e.g., visible tag for AI-generated content)
  • Encode and mux final audio/video
Safety and Approval
• Toxicity filter checks both input and output
• Invisible watermarking
• Optional user approval flow before publishing
• Feedback system for corrections/removals
Playback Logic
• Language-aware serving:
• Prefers original if user device matches
• Auto-serves translated/lip-synced audio and video content otherwise

Future Improvements

The talk also mentioned various points which need improvement.

Add support for more languages (Seamless supports over 100 languages, but the overall system is more limited)
Support multi-speaker content (currently this system only supports a single speaker)
Improve speaker voice cloning (matching the emotion and intonation of the original speaker)
Better support videos with background music (auto detect which song is playing to help separate and re-add the audio)
Improve latency and real-time deployment (make it faster)
Enhance evaluation via better metrics & automation

Given the base audio to audio translation model has been open source for some time the main new information in the talk is how such a model is productised and the amount of pre and post processing needed to deploy the model in a real end user product.

Binary Quantized Embeddings

2025-05-17T05:37:56.798085+00:00

How to use binary quantized embeddings for semantic text search

Have you ever wanted a search that didn't just matched keywords, but searched based on the word or sentence meaning, returning sentences that meant the same, even if spelt slightly differently?
Let's explore how to create a simple retrieval system using binary quantized embeddings for efficient semantic search.

This sort of system is typically used for a Retrieval Augmented Generation (RAG) system. Where you are pre-preparing a prompt to send to an LLM.

What are we building?

We're going to build a lookup system for RAG that can:

Convert text into compact binary embeddings
Store them efficiently in SQLite
Perform very fast semantic searches

Let's start by understanding the core components we'll be using.

What are embeddings?

Embeddings are numerical representations of text that capture semantic meaning. When we convert text to embeddings, similar meanings end up closer together in the embedding space, enabling semantic search by finding embeddings which are "near" our reference embedding we are searching for.

# Simple example of generating an embedding
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('mixedbread-ai/mxbai-embed-large-v1')
text = "Hello world"
embedding = model.encode(text)  # This creates a numerical representation

print(f"Embedding shape: {embedding.shape}")  # Typically (1024,) or similar

While traditional embeddings use floating-point numbers (typically 32-bit), binary quantized embeddings compress each value to just 1 bit, dramatically reducing storage requirements and improving search speed with minimal accuracy loss.

What is Quantization?

Quantization is the act of reducing high-precision float vectors into smaller number types, such as float16, float8, tiny, int, or bit-based formats so they take up less RAM and multiply faster. So by quantizating the float32 array of numbers, info a float16 array you half the size you need to store, and process. But also half the precision. And if you quantize down to a binary value for each float32, you reduce by 32x, going from 4KB per 1024 value (3 bytes per 1024 float) embedding, down to 0.13 KB (128 bytes) — all while our similarity search still returns roughly the same results.

What is SentenceTransformer?

SentenceTransformer is a Python framework that makes it easy to compute sentence and text embeddings. It's built on top of PyTorch and Transformers and gives us access to hundreds of pre-trained models.

What makes SentenceTransformer particularly useful is how it simplifies the process of generating embeddings with just a few lines of code. The library handles all the complexity of transformer models while giving us easy-to-use interfaces.

# Using SentenceTransformer to compare two sentences
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('mixedbread-ai/mxbai-embed-large-v1')

# Generate embeddings for two sentences
embedding1 = model.encode("I love programming")
embedding2 = model.encode("I enjoy coding")

# Calculate similarity (cosine similarity ranges from -1 to 1)
similarity = util.cos_sim(embedding1, embedding2)
print(f"Similarity: {similarity.item():.4f}")  # Should be high (close to 1)

What is SQLite-Vec?

SQLite-Vec is a powerful SQLite extension that adds vector search capabilities to SQLite. It's particularly useful for our search system because it:

Adds vector search operations directly into SQL
Supports binary quantized embeddings out of the box
Works with standard SQLite, which is already included in Python
Provides efficient approximate nearest neighbor search

SQLite-Vec is perfect for our use case because it combines the simplicity of SQLite with specialized vector operations that would otherwise require separate vector databases.

# Basic example of SQLite-Vec setup
import sqlite3
import sqlite_vec

# Connect to SQLite and load the extension
conn = sqlite3.connect("example.db")
conn.enable_load_extension(True)
sqlite_vec.load(conn)
conn.enable_load_extension(False)

# Create a vector table
conn.execute("""
    CREATE VIRTUAL TABLE IF NOT EXISTS vectors 
    USING vec0(
        id INTEGER PRIMARY KEY,
        embedding bit[1024]  -- Binary vectors with 1024 dimensions
    )
""")

Binary quantization: Why compress embeddings?

When working with embeddings, you'll quickly run into either storage, or more likely search speed issues. A standard float32 embedding with 1024 dimensions takes 4KB per document. For a million documents, that's 4GB just for embeddings. And you have to do multidimensional vector distance calculations for each of them when searching (though a lot of that can be cleverly optimised to cut down on how many you need to compute).

Binary quantization compresses each number from a 32-bit float to a single bit (0 or 1), resulting in a 32x reduction in size. Here's how we can quantize embeddings:

from sentence_transformers.quantization import quantize_embeddings

# Generate a standard embedding
embedding = model.encode("Binary quantization makes embeddings tiny!", normalize_embeddings=True)

# Quantize to binary (1 bit per dimension)
binary_embedding = quantize_embeddings(embedding, precision="ubinary")

print(f"Original size: {embedding.nbytes} bytes")
print(f"Binary size: {binary_embedding.nbytes} bytes")
print(f"Compression ratio: {embedding.nbytes / binary_embedding.nbytes}x")

While there's a small trade-off in accuracy, the benefits for most applications are enormous:

32x smaller storage footprint
Much faster similarity calculations using bit operations
Better cache utilization
Lower memory requirements

And although lookup performance is worse than the full 32bit floats, you still get 96.45% accuracy.

Hamming distance: Faster binary embedding lookup speed

One of the most impressive advantages of binary embeddings isn't just their size—it's how much faster and efficient they are to compare. This is due to a fundamental shift in how we measure similarity.

With traditional float32 embeddings, we typically use cosine similarity, which requires several expensive operations:

Vector dot products (multiplication operations)
Vector magnitude calculations (square roots)
Division operations

In contrast, binary embeddings can use Hamming distance, which simply counts how many bits differ between two vectors. This can be implemented using:

An XOR operation (extremely fast on modern CPUs)
A population count (counting the 1s in the result)

These operations are dramatically faster than floating-point math:

# Comparing binary embeddings with Hamming distance
import numpy as np

# Create two binary vectors (already quantized)
binary_vec1 = np.array([0, 1, 1, 0, 1], dtype=np.bool_)
binary_vec2 = np.array([1, 1, 0, 0, 1], dtype=np.bool_)

# Hamming distance calculation (XOR + bit count)
hamming_distance = np.sum(binary_vec1 ^ binary_vec2)
print(f"Hamming distance: {hamming_distance}")  # Number of differing bits

SQLite-Vec takes full advantage of this property. When you store binary embeddings using bit[1024] and the vec_quantize_binary() function, SQLite-Vec automatically uses the much faster Hamming distance calculations during searches.

According to benchmarks from MixedBread.ai and HuggingFace, this approach can yield a 15-45x speedup in retrieval time (with a mean of 25x) compared to traditional float32 embeddings, while retaining over 95% of the retrieval accuracy.

That's why the combination of binary quantization and SQLite-Vec is so powerful for our RAG system—we get both the storage benefits and the computational speedup.

Setup and Installation

Let's get started by installing the required dependencies:

pip install sentence-transformers
pip install sqlite-vec

We'll use the mixedbread-ai/mxbai-embed-large-v1 model, which generates high-quality text embeddings, and was designed to generate embeddings which support being quantized down to binary while retaining lookup accuracy.

Quick test script to check that the dependencies are working.

# Import Needed libraries
from sentence_transformers import SentenceTransformer
import sqlite3
import sqlite_vec

# Load model
model = SentenceTransformer('mixedbread-ai/mxbai-embed-large-v1')

# Test SQLite-Vec
conn = sqlite3.connect(":memory:")
conn.enable_load_extension(True)
sqlite_vec.load(conn)
conn.enable_load_extension(False)

print("Setup complete! Everything is working correctly.")

Creating embeddings with mixedbread-ai/mxbai-embed-large-v1

The mixedbread-ai/mxbai-embed-large-v1 model generates embeddings and works perfectly with binary quantization.

Example usage:

from sentence_transformers import SentenceTransformer
from sentence_transformers.quantization import quantize_embeddings

# Load the model
model = SentenceTransformer('mixedbread-ai/mxbai-embed-large-v1')

# Generate embeddings for some text
text = "This is an example sentence."
embedding = model.encode(text, normalize_embeddings=True)

# Optionally quantize to binary
binary_embedding = quantize_embeddings(embedding, precision="ubinary")

The normalize_embeddings=True parameter is important for binary quantization, as it ensures the values are properly distributed for optimal quantization.

Now let's build a complete RAG system that uses SQLite-Vec for storage and binary quantized embeddings for efficient search:

import os
import sqlite3
import sqlite_vec
from sentence_transformers import SentenceTransformer
from typing import List, Tuple
import json

class RAG:
    """Semantic text search for Retrieval Augmented Generation"""
    
    def __init__(self, db_path: str = None):
        """Initialize the RAG system.
        
        Args:
            db_path: Path to the SQLite database file
        """
        self.db_path = os.path.join(os.path.dirname(__file__), "rag.db") if db_path is None else db_path
        self._init_db()
        # Initialize the embedding model
        self.model = SentenceTransformer('mixedbread-ai/mxbai-embed-large-v1')
    
    def _init_db(self) -> None:
        """Initialize the SQLite database with required tables."""
        conn = sqlite3.connect(self.db_path)
        conn.enable_load_extension(True)
        sqlite_vec.load(conn)
        conn.enable_load_extension(False)
        
        # Create virtual table for vector storage with binary embeddings
        conn.execute("""
            CREATE VIRTUAL TABLE IF NOT EXISTS segments 
            USING vec0(
                id TEXT PRIMARY KEY,       -- Segment ID
                text TEXT,                 -- Segment text
                embedding bit[1024],       -- Binary quantized embedding vector
                metadata TEXT              -- Optional metadata in JSON format
            )
        """)
        
        conn.commit()
        conn.close()
    
    def _get_db_connection(self) -> sqlite3.Connection:
        """Get a database connection with sqlite-vec extension loaded."""
        conn = sqlite3.connect(self.db_path)
        conn.enable_load_extension(True)
        sqlite_vec.load(conn)
        conn.enable_load_extension(False)
        return conn
    
    def load_segment(self, id: str, text: str, metadata: str = "{}") -> None:
        """Load a segment into the database.
        
        Args:
            id: Unique identifier for the segment
            text: The text content
            metadata: Optional JSON metadata string
        """
        if not text or len(text.strip()) == 0:
            return

        # Generate embedding using SentenceTransformer
        embedding = self.model.encode(text, normalize_embeddings=True)
        # Convert embedding to JSON string for SQLite-Vec's vec_quantize_binary function
        embedding_json = json.dumps(embedding.tolist())
        
        # Store in database using vec_quantize_binary
        conn = self._get_db_connection()
        
        conn.execute("""
            INSERT OR REPLACE INTO segments (id, text, embedding, metadata)
            VALUES (?, ?, vec_quantize_binary(?), ?)
        """, (id, text, embedding_json, metadata))
        
        conn.commit()
        conn.close()
    
    def search_segments(self, text: str, limit: int = 10) -> List[Tuple[str, str, float, str]]:
        """Search for segments similar to the given text.
        
        Args:
            text: The text to search for
            limit: Maximum number of results to return
            
        Returns:
            List of tuples containing (id, text, similarity_score, metadata)
        """
        # Generate embedding for query text
        query_embedding = self.model.encode(text, normalize_embeddings=True)
        # Convert query embedding to JSON string
        query_embedding_json = str(query_embedding.tolist())
        
        # Search database using vec_quantize_binary for the query
        conn = self._get_db_connection()
        
        cursor = conn.execute("""
            SELECT id, text, distance, metadata 
            FROM segments 
            WHERE embedding MATCH vec_quantize_binary(?) AND k = ?
            ORDER BY distance
        """, (query_embedding_json, limit))
        
        results = []
        for id, text, distance, metadata in cursor.fetchall():
            # Convert distance to similarity score (normalize to [0,1] range)
            # The distance is a Hamming distance between binary vectors
            similarity = 1.0 - (distance / 1024.0)  # 1024 is our vector dimension
            results.append((id, text, similarity, metadata))
        
        conn.close()
        
        return results
    
    def remove_segment(self, id: str) -> None:
        """Remove a segment from the database.
        
        Args:
            id: The ID of the segment to remove
        """
        conn = self._get_db_connection()
        conn.execute("DELETE FROM segments WHERE id = ?", (id,))
        conn.commit()
        conn.close()

Using the RAG system

Here's how to use the RAG search:

# Initialize the RAG system
rag = RAG()

# Add some documents
rag.load_segment("doc1", "Tokyo is the capital of Japan.")
rag.load_segment("doc2", "Paris is the capital of France.")
rag.load_segment("doc3", "Berlin is the capital of Germany.")
rag.load_segment("doc4", "Rome is the capital of Italy.")

# Search for similar documents
results = rag.search_segments("What is the capital city of Japan?")

# Print results
for id, text, similarity, metadata in results:
    print(f"{id}: '{text}' (similarity: {similarity:.4f})")

Why binary quantization?

Binary quantization offers several advantages:

Storage efficiency: Reduces embedding size by 32x compared to float32
Search speed: Binary operations are extremely fast on modern CPUs
Memory efficiency: Allows storing millions of embeddings in memory
Simplicity: Bit operations are supported by all database systems

For most applications, the small accuracy trade-off is well worth these benefits. When working with large document collections, binary quantized embeddings make semantic search practical and cost-effective.

Conclusion

We've built a simple but powerful RAG system using binary quantized embeddings. This approach makes semantic search efficient and scalable even with limited resources.

References:

Example Code

I've added the above example code to Github, with an extra command line interface to let you load in text and query the DB from the command line here.

Github: ikyle.me-code-examples/Binary Quantized Embeddings Search

Querying the iOS Photo Library

2025-05-15T06:41:31.229506+00:00

To allow users to select a photo from their photo library the simplest option is to use UIImagePickerController to present the built in system interface to pick a photo and return it to your app.
However, UIImagePickerController has limitations, such as no customisations, programmatic library access or even multi-selection. So if you need more access, or more control, than UIImagePickerController provides you'll need to dig into the PhotoKit framework.

Required App Permissions

To access the users Photo library your app will need to ask for permission. First the app has to declare this in the Info.plist file, using the NSPhotoLibraryUsageDescription key and a description (which can be localised).

<key>NSPhotoLibraryUsageDescription</key>
<string>This app wants to use your photos.</string>

You can query the current authorisation status using the API [PHPhotoLibrary authorizationStatus] or PHPhotoLibrary.authorizationStatus().

This API will return one of the following status codes:

// The user has not yet made a choice
PHAuthorizationStatusNotDetermined // ObjC
PHAuthorizationStatus.notDetermined // Swift

// Access is not authorized and the user cannot change this
// (possibly due to restrictions such as parental controls)
PHAuthorizationStatusRestricted
PHAuthorizationStatus.restricted

// The user has explicitly denied the app access
PHAuthorizationStatusDenied
PHAuthorizationStatus.denied

// The user has authorized access to the photo library
PHAuthorizationStatusAuthorized
PHAuthorizationStatus.authorized

The user will be prompted for permission, either when you explicitly ask for permission using the requestAuthorization method on PHPhotoLibrary, or the first time you call an authorisation required Photos framework method.

[PHPhotoLibrary requestAuthorization:^(PHAuthorizationStatus status) {
   NSLog(@"Permission Status: %ld", (long)status);
}];

PHPhotoLibrary.requestAuthorization(handler: { (status) in
   print("Permission status: \(status)")
})

Using the PhotoKit Framework

Querying for Media

How to get the 50 most recently taken photos.

// 50 most recently created images
let options = PHFetchOptions()
options.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: false)]
options.fetchLimit = 50
options.wantsIncrementalChangeDetails = false

let result: PHFetchResult<PHAsset> = PHAsset.fetchAssets(with: .image, options: options)

result.enumerateObjects(options: [], using: { (asset, index, stop) in
    self.assets.append(asset)
})

How to get a list of 50 most recently taken screenshots.

let options = PHFetchOptions()
options.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: false)]
options.fetchLimit = 50

options.predicate = NSPredicate(
   format: "(mediaSubtype & %d) != 0",
   PHAssetMediaSubtype.photoScreenshot.rawValue
)

let result: PHFetchResult<PHAsset> = PHAsset.fetchAssets(with: .image, options: options)

Documentation for the predicate.

A predicate that specifies which properties to select results by and that also specifies any constraints on selection.
Construct a predicate with the properties of the class of objects that you want to fetch, listed in the PHFetchOptions table.
For example, the following code uses a predicate to fetch assets matching a specific set of mediaSubtypes values.
Photos does not support predicates created with the NSPredicate method init or the predicateWithBlock method.

let format: String = "(mediaSubtypes & %d) != 0 || (mediaSubtypes & %d) != 0"
let fetchOptions = PHFetchOptions()
fetchOptions.predicate = NSPredicate(
   format: format,
   argumentArray: [PHAssetMediaSubtype.photoPanorama, PHAssetMediaSubtype.videoHighFrameRate]
)

let fetchResult = PHAsset.fetchAssets(
   with: PHAssetMediaType.image,
   options: fetchOptions
)

Class for Fetch Method	Supported Keys
`PHAsset`	`SELF`, `localIdentifier`, `creationDate`, `modificationDate`, `mediaType`, `mediaSubtypes`, `duration`, `pixelWidth`, `pixelHeight`, `favorite` (or `isFavorite`), `hidden` (or `isHidden`), `burstIdentifier`
`PHAssetCollection`	`SELF`, `localIdentifier`, `localizedTitle` (or `title`), `startDate`, `endDate`, `estimatedAssetCount`
`PHCollectionList`	`SELF`, `localIdentifier`, `localizedTitle` (or `title`), `startDate`, `endDate`
`PHCollection` (can fetch a mix of `PHCollectionList` and `PHAssetCollection` objects)	`SELF`, `localIdentifier`, `localizedTitle` (or `title`), `startDate`, `endDate`

API Docs: https://developer.apple.com/documentation/photos/phfetchoptions#overview

Querying Albums

How to fetch user-created albums and smart albums, then retrieve assets from a specific album.

// Fetch user-created albums
let userAlbums = PHAssetCollection.fetchAssetCollections(
    with: .album,
    subtype: .albumRegular,
    options: nil
)

// Fetch smart albums like Favorites, Recently Added, etc.
let smartAlbums = PHAssetCollection.fetchAssetCollections(
    with: .smartAlbum,
    subtype: .any,
    options: nil
)

// Get assets from a specific album
func getAssetsFromAlbum(album: PHAssetCollection, limit: Int = 0) -> [PHAsset] {
    let options = PHFetchOptions()
    options.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: false)]
    if limit > 0 {
        options.fetchLimit = limit
    }
    
    let result = PHAsset.fetchAssets(in: album, options: options)
    var assets = [PHAsset]()
    
    result.enumerateObjects(options: [], using: { (asset, index, stop) in
        assets.append(asset)
    })
    
    return assets
}

Modifying Albums

How to create new albums, add assets to existing albums, and remove assets from albums.

// Create a new album
func createAlbum(withTitle title: String, completion: @escaping (PHAssetCollection?) -> Void) {
	var placeholder: String?
	
	PHPhotoLibrary.shared().performChanges({
		
		let createRequest: PHAssetCollectionChangeRequest = PHAssetCollectionChangeRequest.creationRequestForAssetCollection(withTitle: title)
		placeholder = createRequest.placeholderForCreatedAssetCollection.localIdentifier
		
	}, completionHandler: { (success: Bool, error: (any Error)?) in
		if success, let identifier = placeholder
		{
			let fetchResult: PHFetchResult<PHAssetCollection> = PHAssetCollection.fetchAssetCollections(
				withLocalIdentifiers: [identifier],
				options: nil
			)
			
			let collection: PHAssetCollection? = fetchResult.firstObject
			
			DispatchQueue.main.async(execute: {
				completion(collection)
			})
		}
		else {
			DispatchQueue.main.async(execute: {
				completion(nil)
			})
		}
	})
}

// Add assets to an album
func addAssets(assets: [PHAsset], toAlbum album: PHAssetCollection, completion: @escaping (Bool) -> Void)
{
	PHPhotoLibrary.shared().performChanges({
		
		if let changeRequest = PHAssetCollectionChangeRequest(for: album) {
			changeRequest.addAssets(PHAsset.fetchAssets(withLocalIdentifiers: assets.map { $0.localIdentifier }, options: nil) as NSFastEnumeration)
		}
		
	}, completionHandler: { (success: Bool, error: (any Error)?) in
		DispatchQueue.main.async(execute: {
			completion(success)
		})
	})
}

// Remove assets from an album
func removeAssets(assets: [PHAsset], fromAlbum album: PHAssetCollection, completion: @escaping (Bool) -> Void)
{
	PHPhotoLibrary.shared().performChanges({
		
		if let changeRequest = PHAssetCollectionChangeRequest(for: album) {
			changeRequest.removeAssets(PHAsset.fetchAssets(withLocalIdentifiers: assets.map { $0.localIdentifier }, options: nil) as NSFastEnumeration)
		}
		
	}, completionHandler: { (success: Bool, error: (any Error)?) in
		DispatchQueue.main.async(execute: {
			completion(success)
		})
	})
}

Getting Folders of Albums

How to fetch folders containing albums and retrieve the albums within each folder.

// Fetch folders that contain albums
func getFolders() -> [PHCollectionList] {
    let fetchOptions = PHFetchOptions()
    fetchOptions.sortDescriptors = [NSSortDescriptor(key: "localizedTitle", ascending: true)]
    
    let foldersFetchResult = PHCollectionList.fetchTopLevelUserCollections(with: fetchOptions)
    var folders = [PHCollectionList]()
    
    foldersFetchResult.enumerateObjects(options: [], using: { (collection, index, stop) in
        if let folder = collection as? PHCollectionList {
            folders.append(folder)
        }
    })
    
    return folders
}

// Get albums within a folder
func getAlbumsInFolder(folder: PHCollectionList) -> [PHAssetCollection] {
    let fetchOptions = PHFetchOptions()
    fetchOptions.sortDescriptors = [NSSortDescriptor(key: "localizedTitle", ascending: true)]
    
    let albumsFetchResult = PHCollection.fetchCollections(in: folder, options: fetchOptions)
    var albums = [PHAssetCollection]()
    
    albumsFetchResult.enumerateObjects(options: [], using: { (collection, index, stop) in
        if let album = collection as? PHAssetCollection {
            albums.append(album)
        }
    })
    
    return albums
}

Modifying Folders

How to create new folders and add albums to existing folders in the photo library.

// Create a new folder
func createFolder(withTitle title: String, completion: @escaping (PHCollectionList?) -> Void) {
	var placeholder: String?
	
	PHPhotoLibrary.shared().performChanges({
		
		let createRequest = PHCollectionListChangeRequest.creationRequestForCollectionList(withTitle: title)
		placeholder = createRequest.placeholderForCreatedCollectionList.localIdentifier
		
	}, completionHandler: { success, error in
		if success, let identifier: String = placeholder
		{
			let fetchResult: PHFetchResult<PHCollectionList> = PHCollectionList.fetchCollectionLists(
				withLocalIdentifiers: [identifier],
				options: nil
			)
			
			let folder: PHCollectionList? = fetchResult.firstObject
			
			DispatchQueue.main.async(execute: {
				completion(folder)
			})
		}
		else {
			DispatchQueue.main.async(execute: {
				completion(nil)
			})
		}
	})
}

// Add albums to a folder
func addAlbums(albums: [PHAssetCollection], toFolder folder: PHCollectionList, completion: @escaping (Bool) -> Void) {
	PHPhotoLibrary.shared().performChanges({
		
		if let changeRequest = PHCollectionListChangeRequest(for: folder)
		{
			changeRequest.addChildCollections(PHAssetCollection.fetchAssetCollections(
				withLocalIdentifiers: albums.map({ $0.localIdentifier }),
				options: nil
			) as NSFastEnumeration)
		}
		
	}, completionHandler: { (success: Bool, error: (any Error)?) in
		DispatchQueue.main.async(execute: {
			completion(success)
		})
	})
}

Getting thumbnails

How to fetch optimized thumbnail images for photo assets to display in a UI collection view.

// Get thumbnail for UICollectionView
func getThumbnail(for asset: PHAsset, targetSize: CGSize, completion: @escaping (UIImage?) -> Void)
{
	let options = PHImageRequestOptions()
	options.deliveryMode = .opportunistic
	options.resizeMode = .exact
	options.isNetworkAccessAllowed = true
	
	PHImageManager.default().requestImage(
		for: asset,
		targetSize: targetSize,
		contentMode: .aspectFill,
		options: options,
		resultHandler: { (image: UIImage?, info: [AnyHashable : Any]?) in
			DispatchQueue.main.async(execute: {
				completion(image)
			})
		})
}

The options for the delivery mode are:

/// Options for delivering requested image data
enum PHImageRequestOptionsDeliveryMode {
   /// Photos automatically provides one or more results in order to balance image quality and responsiveness.
   case opportunistic
   /// Photos provides only the highest-quality image available, regardless of how much time it takes to load.
   case highQualityFormat
   /// Photos provides only a fast-loading image, possibly sacrificing image quality.
   case fastFormat
}

There is also a property to control which version of the photo you would like:

options.version = PHImageRequestOptionsVersion.current

/// Options for requesting an image asset with or without adjustments.
enum PHImageRequestOptionsVersion {
   /// Request the most recent version of the image asset (the one that reflects all edits).
   case current
   /// Request a version of the image asset without adjustments.
   case unadjusted
   /// Request the original, highest-fidelity version of the image asset.
   case original
}

You can also specify how you would like the image resized to the target size you asked for.

options.resizeMode = PHImageRequestOptionsResizeMode.none

/// Options for how to resize the requested image to fit a target size.
enum PHImageRequestOptionsResizeMode {
   /// Photos does not resize the image asset.
   case none
   /// Photos efficiently resizes the image to a size similar to, or slightly larger than, the target size.
   case fast
   /// Photos resizes the image to match the target size exactly.
   case exact
}

You can also specify how the image should be cropped.

options.normalizedCropRect = CGRect.zero

To request a cropped image, specify the crop rectangle in a unit coordinate space relative to the image. In this coordinate system, the point {0.0,0.0} refers to the upper left corner of the image, and the point {1.0,1.0} refers to the opposite corner regardless of the image’s aspect ratio.
This property defaults to CGRectZero, which specifies no cropping.
If you specify a crop rectangle, you must also specify the PHImageRequestOptionsResizeMode.exact option for the resizeMode property.
https://developer.apple.com/documentation/photos/phimagerequestoptions/normalizedcroprect

Getting Full Size Images

How to retrieve full-resolution images from the photo library for detailed viewing or editing.

// Get full size image for editing
func getFullSizeImage(for asset: PHAsset, completion: @escaping (UIImage?, [AnyHashable: Any]?) -> Void) {
	let options = PHImageRequestOptions()
	options.deliveryMode = .highQualityFormat
	options.isNetworkAccessAllowed = true
	options.isSynchronous = false
	
	PHImageManager.default().requestImage(
		for: asset,
		targetSize: PHImageManagerMaximumSize,
		contentMode: .aspectFit,
		options: options,
		resultHandler: { (image: UIImage?, info: [AnyHashable : Any]?) in
			DispatchQueue.main.async(execute: {
				completion(image, info)
			})
		}
	)
}

Editing Metadata

How to update photo metadata including location and favorite status.

// Modify photo location
func updateLocation(for asset: PHAsset, to newLocation: CLLocation, completion: @escaping (Bool) -> Void)
{
	PHPhotoLibrary.shared().performChanges({
		
		let request = PHAssetChangeRequest(for: asset)
		request.location = newLocation
		
	}, completionHandler: { (success: Bool, error: (any Error)?) in
		DispatchQueue.main.async(execute: {
			completion(success)
		})
	})
}

// Favorite or unfavorite an asset
func setFavorite(asset: PHAsset, isFavorite: Bool, completion: @escaping (Bool) -> Void)
{
	PHPhotoLibrary.shared().performChanges({
		
		let request = PHAssetChangeRequest(for: asset)
		request.isFavorite = isFavorite
		
	}, completionHandler: { (success: Bool, error: (any Error)?) in
		DispatchQueue.main.async(execute: {
			completion(success)
		})
	})
}

Getting Videos

How to fetch and stream video assets from the photo library, including iCloud videos.

// Stream iCloud video
func getVideoAsset(for asset: PHAsset, completion: @escaping (AVAsset?, [AnyHashable: Any]?) -> Void) {
	let options = PHVideoRequestOptions()
	options.deliveryMode = .automatic
	options.isNetworkAccessAllowed = true
	
	PHImageManager.default().requestAVAsset(
		forVideo: asset,
		options: options,
		resultHandler: { (avAsset: AVAsset?, audioMix: AVAudioMix?, info: [AnyHashable : Any]?) in
			
			DispatchQueue.main.async(execute: {
				completion(avAsset, info)
			})
		}
	)
}

The options for the delivery mode are:

enum PHVideoRequestOptionsDeliveryMode {
   /// Photos automatically determines which quality of video data to provide based on the request and current conditions.
   case automatic
   /// Photos provides only the highest quality video available.
   case highQualityFormat
   /// Photos provides a video of moderate quality unless a higher quality version is locally cached.
   case mediumQualityFormat
   /// Photos provides whatever quality of video can be most quickly loaded.
   case fastFormat
}

There is also a property to control which version of the video you would like:

options.version = PHVideoRequestOptionsVersion.current

/// Options for requesting a video asset with or without adjustments.
enum PHVideoRequestOptionsVersion {
   /// Request the most recent version of the video asset, reflecting all edits.
   case current
   /// Request a version of the video asset without adjustments.
   case original
}

Playing Live Photos

How to retrieve and display Live Photos with their interactive capabilities.

// Get PHLivePhoto object for display
func getLivePhoto(for asset: PHAsset, targetSize: CGSize, completion: @escaping (PHLivePhoto?, [AnyHashable: Any]?) -> Void) {
	let options = PHLivePhotoRequestOptions()
	options.deliveryMode = .highQualityFormat
	options.isNetworkAccessAllowed = true
	
	PHImageManager.default().requestLivePhoto(
		for: asset,
		targetSize: targetSize,
		contentMode: .aspectFit,
		options: options,
		resultHandler: { (livePhoto: PHLivePhoto?, info: [AnyHashable : Any]?) in
			
			DispatchQueue.main.async(execute: {
				completion(livePhoto, info)
			})
		}
	)
}

Get video from live photo

How to extract and access the video component from a Live Photo asset.

// Extract video component from a Live Photo
func getVideoFromLivePhoto(asset: PHAsset, completion: @escaping (AVAsset?) -> Void)
{
	guard asset.mediaSubtypes.contains(.photoLive) else {
		completion(nil)
		return
	}
	
	let resources: [PHAssetResource] = PHAssetResource.assetResources(for: asset)
	
	guard let videoResource: PHAssetResource = resources.first(where: { $0.type == .pairedVideo }) else {
		completion(nil)
		return
	}
	
	let options = PHAssetResourceRequestOptions()
	options.isNetworkAccessAllowed = true
	
	let fileURL: URL = FileManager.default.temporaryDirectory.appendingPathComponent("\(NSUUID().uuidString).mov")
	
	PHAssetResourceManager.default().requestData(
		for: videoResource,
		options: options,
		dataReceivedHandler: { (data: Data) in
			// This handler might be called multiple times
			try? data.write(to: fileURL, options: .atomic)
		},
		completionHandler: { (error: (any Error)?) in
			if error == nil {
				let asset = AVAsset(url: fileURL)
				DispatchQueue.main.async(execute: {
					completion(asset)
				})
			}
			else {
				DispatchQueue.main.async(execute: {
					completion(nil)
				})
			}
		}
	)
}

Getting updates to your PHFetchResult

How to track and respond to changes in the photo library using the persistent change API.

/// Using the new change history API to track changes
class PhotoLibraryChangeTracker {
	
	private var lastChangeToken: PHPersistentChangeToken?
	
	init() {
		self.lastChangeToken = PHPhotoLibrary.shared().currentChangeToken
	}
	
	/// Fetch all asset changes since the last change update `lastChangeToken`
	/// - Parameter completion: inserted, updated, deleted
	func fetchChanges(completion: @escaping ([String], [String], [String]) -> Void) {
		guard let token: PHPersistentChangeToken = self.lastChangeToken else {
			completion([], [], [])
			return
		}
		
		do {
			let persistentChanges: PHPersistentChangeFetchResult = try PHPhotoLibrary.shared().fetchPersistentChanges(since: token)
			
			var insertedIdentifiers = [String]()
			var updatedIdentifiers = [String]()
			var deletedIdentifiers = [String]()
			
			for persistentChange in persistentChanges {
				do {
					let changeDetails: PHPersistentObjectChangeDetails = try persistentChange.changeDetails(for: .asset)
					
					insertedIdentifiers.append(contentsOf: changeDetails.insertedLocalIdentifiers)
					updatedIdentifiers.append(contentsOf: changeDetails.updatedLocalIdentifiers)
					deletedIdentifiers.append(contentsOf: changeDetails.deletedLocalIdentifiers)
					
					// Store the last change token for future use
					self.lastChangeToken = persistentChange.changeToken
				}
				catch {
					print("ERROR: \(error)")
				}
			}
			
			DispatchQueue.main.async(execute: {
				completion(insertedIdentifiers, updatedIdentifiers, deletedIdentifiers)
			})
		}
		catch
			PHPhotosError.persistentChangeTokenExpired,
			PHPhotosError.persistentChangeDetailsUnavailable
		{
			// Handle errors by refetching tracked objects
			DispatchQueue.main.async(execute: {
				completion([], [], [])
			})
		}
		catch {
			DispatchQueue.main.async(execute: {
				completion([], [], [])
			})
		}
	}
	
}

Show iOS System Dictionary Screen

2025-05-06T01:42:32.435253+00:00

Show iOS System Dictionary Screen

iOS includes a built-in dictionary that can be shown to the user programmatically using UIReferenceLibraryViewController. This allows you to show word definitions directly in your app.

Using UIReferenceLibraryViewController

The UIReferenceLibraryViewController is simple to use, but also very limited.
Initialize it with a term to look up, then present the UIViewController:

import UIKit

// Check if dictionary is available for the word
let term = "serendipity"
let dictionaryAvailable = UIReferenceLibraryViewController.dictionaryHasDefinition(forTerm: term)

if dictionaryAvailable {
    print("Has definition in one of the currently downloaded dictionaries")
}
else {
    print("Does not have a definition in one of the currently downloaded dictionaries")
}

// Create and present the dictionary view controller
let dictionaryViewController = UIReferenceLibraryViewController(term: term)
self.present(dictionaryViewController, animated: true, completion: nil)

Even if there is no definition available, presenting the UIReferenceLibraryViewController anyway is a good idea.
If the user has no dictionaries downloaded, there will be no definition found for any word. However, the presented UIReferenceLibraryViewController will show the user an option to download a dictionary file.

All Dictionaries	Definition

Swift Map vs CompactMap vs FlatMap

2024-10-17T01:57:52.894801+00:00

It can be confusing to remember the difference, so this is a cheatsheet for reference.

Swift `.map()`

map({}) transforms all elements in a list.

let list = ["about", "testing", "1", "100", "300"]
print(list)
// ["about", "testing", "1", "100", "300"]

let mappedList = list.map({ String($0.reversed()) })
print(mappedList)
// ["tuoba", "gnitset", "1", "001", "003"]

.compactMap()

compactMap() compacts the list, by removing the nil items. (filterMap was the original name for the function when it was proposed)

let list = ["about", nil, "testing", "1", nil, "100", "300", nil]
print(list)
// [Optional("about"), nil, Optional("testing"), Optional("1"), nil, Optional("100"), Optional("300"), nil]

let compactedList = list.compactMap({ $0 })
print(compactedList)
// ["about", "testing", "1", "100", "300"]

.flatMap()

flatMap() flattens a list of other lists by combining their contents.

let list = [ ["1", "2", "3"], ["a", "b", "c"], ["kyle", "harry", "tom"] ]
print(list)
// [["1", "2", "3"], ["a", "b", "c"], ["kyle", "harry", "tom"]]

let compactedList = list.flatMap({ $0 })
print(compactedList)
// ["1", "2", "3", "a", "b", "c", "kyle", "harry", "tom"]

Running Local LLMs on macOS

2024-08-08T03:46:31.472765+00:00

To install and run ChatGPT style LLM models locally and offline on macOS the easiest way is with either llama.cpp or Ollama (which basically just wraps llama.cpp).

llama.cpp is one of those open source libraries which is what actually powers most more user facing applications. (like ffmpeg if you do anything with audio or video)

It loads and runs LLM model files and allows you to use them via:

An OpenAI API compatible web server.
Directly from Terminal via the cli
A variety of supported UIs

It has a lot of advanced features and supports all sorts of options.

However, you still have to get the LLM models and make sure they are the correct format, and use the right chat template (which if they are in llama.cpp's file format should be included in the model file).

Ollama

When Ollama runs a model, it basically just starts the llama.cpp CLI with some default parameters.
However, Ollama doesn't just run models.

It also includes a library of pre-quantised (shrunken to load faster and fit in less vram) models available for easy download.
And a homrbrew style CLI interface for downloading the models.

Installation

Download both the GUI tool from https://ollama.com/download and install the cli tool from homebrew.

brew install ollama

Usage

Find a model you want to run.
The best current (2024-08) small models are:

Llama3.1 8b (meta)
Mistral NeMo 12b (mistral)
Gemma 2 9b (Google)

Llama3.1 is considered the best currently available open source model, but the 8b version struggles with tool usage, and the 70b is too big for a lot of computers.
Mistral NeMo 12b is slightly bigger, but handles tool usage much better.
Gemma 2 doesn't support tool usage but is a very capable general model.

Download the model

ollama pull mistral-nemo:12b

Run the Model

To run the llm directly from the command line:

ollama run mistral-nemo:12b

You'll now have an interactive cli prompt you can use to talk to the LLM.
Like a terminal version of ChatGPT.

However, you'll notice the LLMs have mostly been trained to answer with markdown, so a UI is most useful.

Start the web server

ollama serve

Interact via web API

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [
    { "role": "user", "content": "why is the sky blue?" }
  ]
}'

As long as you have the asked for model downloaded it'll load the model and run the command.

Useful Notes

As the web API can load models on demand it can be useful to see what models are currently loaded in ram (by default kept for 4 minutes after the last API call, this can be changed).

ollama ps

NAME      	ID          	SIZE 	PROCESSOR	UNTIL
llama3:70b	bcfb190ca3a7	42 GB	100% GPU 	4 minutes from now

List models downloaded

ollama list

NAME                              	ID          	SIZE  	MODIFIED     
mistral-nemo:latest               	4b300b8c6a97	7.1 GB	7 days ago  	
llama3.1:8b                       	62757c860e01	4.7 GB	7 days ago  	
phi3:medium-128k                  	3aeb385c7040	7.9 GB	4 weeks ago 	
gemma2:latest                     	ff02c3702f32	5.4 GB	4 weeks ago 	
llama3:8b                         	365c0bd3c000	4.7 GB	4 weeks ago 	
phi3:mini-128k                    	d184c916657e	2.2 GB	4 weeks ago 	
phi3:3.8-mini-128k-instruct-q5_K_M	5a696b4e6899	2.8 GB	2 months ago	
phi3:latest                       	a2c89ceaed85	2.3 GB	2 months ago

Ram Limitations

To run an LLM model it needs to fit in your vram, the GPUs ram. On macOS the M series chips having unified memory shared between the CPU and GPU is great for this! On PC systems you are limited to the 16 GB of vRam on the RTX 4080, or 24GB of vram on the RTX 3090 and RTX 4090.

However, Macs can have as much vram as they have normal ram*. The actual limit is the system only gives about 2/3 to the GPU at max, and needs some for the rest of the system, so you don't actually get to use all of it. You can tweak the limit: sudo sysctl iogpu.wired_limit_mb=<mb> (source), but you need to be careful doing so.

All this means when picking a model to download and run, you can't just pick the biggest, and best, model.
You have to pick one that will fit inside your vram.

Models take up roughly their download size + a little extra (for the prompt and processing).

So Llama 3.1 8b takes up 16GB for the base 16 bit floating point model.
However, if the model is quantised down to storing all those model weights as 4 bit ints instead, it is now 4.7 GB.
The model gets a bit dumber doing this, but not by that much most of the time.

If you pick a model which is tool big, then llama.cpp will actually stop trying to load it into the GPU and start using CPU ram instead, doing some of the processing on the CPU and some on the GPU.
This is MUCH slower and should be avoided if you can.
Also as M chip mac's share the memory between the 2 anyway, this isn't as effective and you might still get a crash if trying to load a model that is too big.

Pretty Print JSON in the Terminal

2024-08-08T02:32:53.507775+00:00

I was testing out a web API quickly with curl and it printed minified JSON to the terminal.
Instead of trying to read that densely packed json text, this is how to pretty print the json on the terminal instead.

The tool jq is designed to handle and format JSON data on the command line.

Here’s how you can do it:

Install

Install it using homebrew:

brew install jq

Usage

Use the following command to pretty-print the JSON response from curl:

curl -s <URL> | jq .

-s: This option makes curl operate in silent mode (no progress meter or error messages).
<URL>: Replace this with the URL you are making the request to.
| jq .: This pipes the output of curl to jq, which formats the JSON. (the . is an argument)

Example

curl -s https://api.example.com/data | jq .

This will print the JSON response in a readable, pretty-printed format.

Notes

jq is actually meant not to pretty print json, but to transform and manipulate json data.
The . argument to it tells it to return the input unchanged as the output. This implies (correctly) other arguments change the input json before outputting it.

A fully tutorial on it's abilities is available on it's website: jqlang.github.io/jq/tutorial/

How to Turn On an LG C2 with Home Assistant

2023-02-09T03:54:05.107912+00:00

HomeAssistant automatically detects and supports LG webOS smart TVs with the LG webOS Smart TV integration.

Turn On TV

However, by default you can turn the TV off, but not back on again.
In order to turn the TV on you need to manually setup an action on the device which calls the WakeOnLan service.

As far as I understand, the reason for this is because the WakeOnLan service is a separate integration and integrations can't install/call each other automatically.

Setup

1: Enable the WakeOnLan integration.
(requires editing the configuration.yaml manually).

2: Inside the device details for the TV.
Create a new automation triggered by Device is requested to turn on.

3: Add a Call service action.

4: Select call Wake on LAN send magic packet

5: Add the MAC address of the TV.

The instructions on the Home Assistant documentation doesn't mention this, but for me to get this to work I also needed to fill in the Broadcast address and set it to the local subnet's broadcast IP, and unselect the Broadcast port.

As my IP address range is 192.168.50.1-254 this is 192.168.50.255 for me.
For you it will likely be either 192.168.0.255 or 192.168.1.255.

This automation should be automatically picked up by the webOS integration and means you will now have fully power ON and OFF controls for the TV device.

Finished Setup

Creating a CoreImage Filter with a Metal Shader

2022-07-03T00:56:34.030222+00:00

In my recent article explaining how to make a metaball style iOS Shape Morphing effect I glossed over how to use a Metal CoreImage filter; instead of using the deprecated Core Image Kernel Language Shader.

The reason for glossing over it because it is annoyingly complicated!

Where as showing you how to use the Core Image Kernel Language Shader based shader as as simple as this:

// LumaThresholdFilter.swift

class LumaThresholdFilter: CIFilter
{
	var threshold: CGFloat = 0.5
	
	static let thresholdKernel = CIColorKernel(source:"""
kernel vec4 thresholdFilter(__sample image, float threshold)
{
	float luma = (image.r * 0.2126) + (image.g * 0.7152) + (image.b * 0.0722);
	return (luma > threshold) ? vec4(1.0, 1.0, 1.0, 1.0) : vec4(0.0, 0.0, 0.0, 0.0);
}
""")!
	
	//
	
	@objc dynamic var inputImage : CIImage?
	
	override var outputImage : CIImage!
	{
		guard let inputImage = self.inputImage else
		{
			return nil
		}
		
		let arguments = [inputImage, Float(self.threshold)] as [Any]
		
		return Self.thresholdKernel.apply(extent: inputImage.extent, arguments: arguments)
	}
	
}

To explain how to use the Metal shader version requires quite a few more steps.

Custom Core Image Filters with Metal Shaders

Unlike with Core Image Kernel Language shaders you can't just include the shader as a string. Instead them most be in separate .metal files.

This has positives: like compile time error checking; and negatives: no tricks involving dynamic manipulation/generation of shaders.

So to start with we'll move our thresholdFilter shader function from the string into its own file LumaThreshold.metal.
Except no. We'll actually name it LumaThreshold.ci.metal not because we have to, but because it'll help us later.

// LumaThreshold.ci.metal

#include <metal_stdlib>
using namespace metal;
#include <CoreImage/CoreImage.h>

extern "C" float4 lumaThreshold(coreimage::sample_t pixelColor, float threshold, coreimage::destination destination)
{
	float3 pixelRGB = pixelColor.rgb;
	
	float luma = (pixelRGB.r * 0.2126) + (pixelRGB.g * 0.7152) + (pixelRGB.b * 0.0722);
	
	return (luma > threshold) ? float4(1.0, 1.0, 1.0, 1.0) : float4(0.0, 0.0, 0.0, 0.0);
}

Now to load this in our CoreImage filter subclass we'll load it like this:

static var thresholdKernel: CIColorKernel = { () -> CIColorKernel in
	let url = Bundle.main.url(forResource: "LumaThreshold", withExtension: "ci.metallib")!
	let data = try! Data(contentsOf: url)
	
	do {
		return try CIColorKernel(functionName: "lumaThreshold", fromMetalLibraryData: data)
	}
	catch {
		print("\(error)")
		fatalError("\(error)")
	}
}()

If you were observant you might notice we made a LumaThreshold.ci.metal file in the source code, but are now loading a LumaThreshold.ci.metallib file at runtime.

When we try to build and run the app now we'll get a crash with the error message:

> let url = 
^ MetaballEffectFilter.swift:56: Fatal error: Unexpectedly found nil while unwrapping an Optional value

That's because first we need to preprocess the Metal files; unfortunately, Xcode doesn't do this automatically for you.

The WWDC 2020 video Build Metal-based Core Image kernels with Xcode shows you how to set up the needed custom build steps.
However, if yo follow it exactly you will get more runtime errors, as the instructions are slightly wrong.

As of June 2022 using Xcode 13.4.1 these are the needed steps:

# First convert LumaThreshold.ci.metal to LumaThreshold.ci.air
xcrun metal -c $MTL_HEADER_SEARCH_PATHS -fcikernel  "LumaThreshold.ci.metal" -o "LumaThreshold.ci.air"

# Then convert LumaThreshold.ci.air to LumaThreshold.ci.metallib
xcrun metallib -cikernel "LumaThreshold.ci.air" -o "LumaThreshold.ci.metallib"

To do this automatically we'll setup some custom build rules like suggested in the WWDC video.

In the Xcode Project Settings > Build Rules tab, click on the + icon to setup a new build rule.

Then create 2 new rules:

First the one for *.ci.metal files.
This is why we named it .ci.metal, instead of just .metal. So we can process this separately from any non CoreImage related Metal files we might have.

Source files with names matching: *.ci.metal
[ ] Run once per architecture

Custom script:

xcrun metal -c $MTL_HEADER_SEARCH_PATHS -fcikernel  "${INPUT_FILE_PATH}" -o "${SCRIPT_OUTPUT_FILE_0}"

Output Files

$(DERIVED_FILE_DIR)/$(INPUT_FILE_BASE).air

Next step a step to turn these files into new .air files into *.metallib.

Source files with names matching: *.ci.air
[ ] Run once per architecture

Custom script:

xcrun metallib -cikernel "${INPUT_FILE_PATH}" -o "${SCRIPT_OUTPUT_FILE_0}"

Output Files

$(METAL_LIBRARY_OUTPUT_DIR)/$(INPUT_FILE_BASE).metallib

NOTE: The first above command is slightly different from the one in the WWWDC video.

The WWDC video from 2020 shows this command:

xcrun metal -c -I $MTL_HEADER_SEARCH_PATHS -fcikernel  "${INPUT_FILE_PATH}" -o "${SCRIPT_OUTPUT_FILE_0}"

If you use that however, it will strip out the function during linking and when you run the project you will get this error:

MetaballEffectFilter.swift:64: Fatal error: Error Domain=CIKernel Code=1 "(null)" UserInfo={CINonLocalizedDescriptionKey=Function does not exist in library data.

The problem is the -I remove that and your build scripts setup should work fine.

If we build and run now, we should have a fully working CoreImage filter using a Metal Shader.

Useful Links

Creating Cool UI: iOS Shape Morphing

2022-07-02T02:15:01.083349+00:00

The other day on Twitter I came across a tweet showing a really cool UI demo and wondered how it was done.

Seeing this tweet reminded me of an idea I've had for several years to do a blog post series showing how to actually create various cool UI concept videos, Dribbble posts, or weird UI elements from other apps.

So, as hopefully the first of many, let's see how to a Create Cool UI Component: "Shape Morphing" on iOS.

Metaballs

The technique for organic looking objects morphing between and into each other is known as metaballs.

As two objects get close together, instead of waiting until they start to overlap.

The two objects will instead start stretching towards each other and combining.

iOS Implementation

To implement this effect we need to combine 2 different effects together (at least the way we are going to go about it, there are a few other ways to achieve this effect; which are linked at the bottom of the page).

We need to render our shapes and then apply these effects over the top as post processing steps to the rendered image.

CALayer's do have a .filters property which looks perfect for this!

An array of Core Image filters to apply to the contents of the layer and its sublayers. Animatable

Until you get to the bottom of the documentation.

This property is not supported on layers in iOS

🥲

Sadly such rendering effects aren't supported on iOS.
So instead we'll need to use a rendering API which does let us post-process the rendered result.

On iOS we have the choice of either SpriteKit or SceneKit.

SceneKit has a cool trick of letting you set a UIView as the material of a node, which would be really helpful in applying the trick to UIView's as we could host the views inside the SceneKit view.

However, for now I'll just use SpriteKit as I don't need anything more for this example and it is simpler to work with.

SpriteKit Implementation

First we need to setup our scene:

class SimpleViewController: UIViewController {
	
	// Our SpriteKit Objects
	let skView = SKView()
	let scene = SKScene()
	
	// The two balls
	var blobOne: SKShapeNode? = nil
	var blobTwo: SKShapeNode? = nil
	
	override func viewDidLoad() {
		super.viewDidLoad()
		
		// Setup Scene
		self.scene.scaleMode = .resizeFill
		self.scene.physicsWorld.gravity = CGVector(dx: 0, dy: 0)
		
		self.skView.presentScene(self.scene)
		self.view.addSubview(self.skView);
		
		// Create the 2 balls
		{
			let blob = SKShapeNode(circleOfRadius: 50)
			blob.fillColor = UIColor.yellow
			blob.strokeColor = blob.fillColor
			
			self.scene.addChild(blob)
			self.blobOne = blob
		}();
		
		{
			let blob = SKShapeNode(circleOfRadius: 50)
			blob.fillColor = UIColor.white
			blob.strokeColor = blob.fillColor
			
			self.scene.addChild(blob)
			self.blobTwo = blob
		}();
	}
	
	// Set the positions of everything
	override func viewDidLayoutSubviews() {
		super.viewDidLayoutSubviews()
		let bounds = self.view.bounds
		
		self.skView.frame = bounds
		
		let center = CGPoint(x: bounds.midX, y: bounds.midY)
		
		self.blobOne?.position = center.applying(CGAffineTransform(translationX: -50, y: 0))
		self.blobTwo?.position = center.applying(CGAffineTransform(translationX: 50, y: 0))
	}
}

This gives us our basic scene. Two spheres near each other but only just touching.

Now let's add a blur effect at the end of -viewDidLoad().

// - At the top of the file....
import CoreImage
import CoreImage.CIFilterBuiltins

// - At the bottom of viewDidLoad

// Create 
let blur = CIFilter.gaussianBlur()
blur.radius = 20
self.scene.filter = blur

// Make sure the scene uses the filter we created
self.scene.shouldEnableEffects = true

Now that we've done that we can see how the shapes are merged together as their blurs overlap and merge.

However, this still doesn't look like the effect we want. To achieve that we will need to apply a threshold filter where we make fully opaque everything over a certain threshold, and we hide every pixel below that.
This will make sure our shapes still have precise boundaries and appear to merge, not just blur together.

We can't apply multiple filters to the SpriteKit scene, so we'll need to subclass CIFilter and make our own combined filter.

class MetaballEffectFilter: CIFilter
{
	// Internal Filters
	
	let blurFilter: CIFilter & CIGaussianBlur = {
		let blur = CIFilter.gaussianBlur()
		blur.radius = 30
		return blur
	}()
	
	let thresholdFilter = LumaThresholdFilter()
	
	
	// - CIFilter Subclass Properties
	
	@objc dynamic var inputImage : CIImage?
	
	override var outputImage: CIImage!
	{
		guard let inputImage = self.inputImage else
		{
			return nil
		}
		
		// Blur Image
		self.blurFilter.inputImage = inputImage
		let blurredOutput = self.blurFilter.outputImage
		
		// Clip to the threshold set
		self.thresholdFilter.inputImage = blurredOutput
		return self.thresholdFilter.outputImage
	}
	
}

First we blur the image, then clip the output to the threshold.
But where does this LumaThresholdFilter come from? This is another class we'll need to make ourselves.

The easier way to do this is to use Core Image Kernel Language API which is deprecated. Instead you are meant to use the new metal shader CoreImage filter API.
However, to make CoreImage metal shaders requires several more steps of tweaking your build config by adding custom build rules and build settings. So for now I'll show examples of both the metal and CoreImage shaders but will use the Core Image Kernel Language for the rest of the examples afterwards.

Metal Shader

// LumaThreshold.ci.metal

#include <metal_stdlib>
using namespace metal;
#include <CoreImage/CoreImage.h>

extern "C" float4 lumaThreshold(coreimage::sample_t pixelColor, float threshold, coreimage::destination destination)
{
	float3 pixelRGB = pixelColor.rgb;
	
	float luma = (pixelRGB.r * 0.2126) + (pixelRGB.g * 0.7152) + (pixelRGB.b * 0.0722);
	
	return (luma > threshold) ? float4(1.0, 1.0, 1.0, 1.0) : float4(0.0, 0.0, 0.0, 0.0);
}

Core Image Kernel Language Shader

// LumaThresholdFilter.swift

class LumaThresholdFilter: CIFilter
{
	var threshold: CGFloat = 0.5
	
	static let thresholdKernel = CIColorKernel(source:"""
kernel vec4 thresholdFilter(__sample image, float threshold)
{
	float luma = (image.r * 0.2126) + (image.g * 0.7152) + (image.b * 0.0722);
	return (luma > threshold) ? vec4(1.0, 1.0, 1.0, 1.0) : vec4(0.0, 0.0, 0.0, 0.0);
}
""")!
	
	//
	
	@objc dynamic var inputImage : CIImage?
	
	override var outputImage : CIImage!
	{
		guard let inputImage = self.inputImage else
		{
			return nil
		}
		
		let arguments = [inputImage, Float(self.threshold)] as [Any]
		
		return Self.thresholdKernel.apply(extent: inputImage.extent, arguments: arguments)
	}
	
}

This now gives us our final effect:

From Metaballs to Icon Morphing

Now we have our metaballs effect. But we still don't have the same icon morphing as shown in the original Twitter video. However, we now have everything we need.

Now we just need to fade in the new icon over the top of the old icon, while fading the old icon out.

First we need to swap the 2 circles for 1 icon. We'll also need to keep hold of the filter reference this time, so we can animate it.

var currentIcon: SKSpriteNode?

let filter = MetaballEffectFilter()

And let's add some button on screen for us to use.

// Property

lazy var buttons : [UIButton] = { [unowned self] in
	return [
		"circle.fill",
		"heart.fill",
		"star.fill",
		"bell.fill",
		"bookmark.fill",
		"tag.fill",
		"bolt.fill",
		
		"play.fill",
		"pause.fill",
		"squareshape.fill",
		"key.fill",
		"hexagon.fill",
		"gearshape.fill",
		"car.fill",
	].map({ buttonName in
		
		var config = UIButton.Configuration.filled()
		config.image = UIImage(systemName: buttonName)
		config.baseBackgroundColor = UIColor.white
		config.baseForegroundColor = UIColor.black
		
		let button = UIButton(configuration: config)
		
		button.addAction(UIAction(handler: { [weak self] _ in
			self?.animateIconChange(newIconName: buttonName, duration: 0.5)
		}), for: .touchUpInside)
		
		return button
	})
}()


// ...in viewDidLoad()

for button in self.buttons {
	self.view.addSubview(button)
}


// .. In viewDidLayoutSubviews()

let buttonSize: CGSize = CGSize(width: 70, height: 50)

var x: CGFloat = 0
var y: CGFloat = bounds.height - (self.view.safeAreaInsets.bottom + 5 + buttonSize.height)

for button in self.buttons {
	button.bounds = CGRect(origin: .zero, size: buttonSize)
	button.center.x = x + (buttonSize.width * 0.5)
	button.center.y = y + (buttonSize.height * 0.5)
	
	x += buttonSize.width + 5
	
	if x > bounds.size.width - (buttonSize.width + 5) {
		x = 0
		y -= buttonSize.height + 5
	}
}

Now let's setup showing the icon. (Uses this UIImage helper).

// ...at the end of viewDidLoad()

self.animateIconChange(newIconName: "circle.fill", duration: 0)


// Create new method

func animateIconChange(newIconName: String, duration: CGFloat, showPlayIcon: Bool = false) {
	// Create new icon shape
	let newIconShape: SKSpriteNode? = {
		let iconSize = CGSize(width: 80, height: 80)
		
		guard let image = UIImage(systemName: newIconName)?.withTintColor(UIColor.white).resized(within: iconSize) else { return nil }
		
		let texture = SKTexture(image: image)
		let sprite = SKSpriteNode(texture: texture, size: iconSize)
		return sprite
	}()
	
	newIconShape?.position = CGPoint(
		x: self.view.bounds.midX,
		y: self.view.bounds.midY
	)
	newIconShape?.alpha = 0
	
	// Add new icon
	if let newIconShape = newIconShape {
		self.scene.addChild(newIconShape)
	}
	
	let oldIconShape = self.currentIcon
	self.currentIcon = nil
	
	self.currentIcon = newIconShape
	
	
	if duration == 0 {
		newIconShape?.alpha = 1
		oldIconShape?.removeFromParent()
		return
	}

	...
}

Now we have our basic structure.

To animate the change similarly to how it appears on Twitter we want to do several things in quick succession:

First animate the blur from 0 to something.
Halfway through that, start alpha fading in the new shape and out the old shape.
About 3/4rd way through the shapes fading from one to the other, then animate the blur back down to 0 so by the end you end up left with just the new target shape itself.

	// ... continues
	
	// Animate the change
	
	let fadeDuration = (duration * 0.25)
	
	// Animate in the blur effect
	self.animateBlur(duration: fadeDuration, blur: 5, from: 0)
	
	// Wait then start fading in the new icon and out the old
	DispatchQueue.main.asyncAfter(deadline: .now() + (fadeDuration * 0.75), execute: {
		
		let swapDuration = duration * 0.5
		
		newIconShape?.run(SKAction.fadeAlpha(to: 1, duration: swapDuration))
		oldIconShape?.run(SKAction.fadeAlpha(to: 0, duration: swapDuration))
		
		// Wait, then start returning the view back to a non-blobby version
		DispatchQueue.main.asyncAfter(deadline: .now() + (swapDuration * 0.75), execute: {
			
			self.animateBlur(duration: fadeDuration, blur: 0, from: 5)
			
			// Cleanup
			DispatchQueue.main.asyncAfter(deadline: .now() + fadeDuration, execute: {
				oldIconShape?.removeFromParent()
			})
		})
	})
}

// Helper which animates the blur
func animateBlur(duration: CGFloat, blur targetBlur: CGFloat, from: CGFloat) {
	let blurFade = SKAction.customAction(withDuration: duration, actionBlock: { (node, elapsed) in
		let percent = elapsed / CGFloat(duration)
		
		let difference = (targetBlur - from)
		let currentBlur = from + (difference * percent)
		
		self.filter.blurFilter.setValue(currentBlur, forKey: kCIInputRadiusKey)
		self.scene.shouldEnableEffects = true
	})
	
	self.scene.run(blurFade)
}

As the different parts of the shapes fade in and go over the threshold they appear.
The blur also means the shapes sort of meld together as we sort earlier with the metaballs effect itself.

We basically fade from one icon to the other while animating the metaballs effect on and then back off again.

The result:

You can change the feel of the effect by changing the speed and overlap of different parts of the animation; and how much blur you fade in during the animation.
But I've found those values to look quite nice and match the style of the effect in the original video reasonably well.

Useful Links

UITableViewCell Styles

2022-05-29T00:17:59.498373+00:00

UITableView has various style options UITableViewCell.CellStyle, but the documentation doesn't include any images to show you what each one actually looks like.

UITableView Cells

When creating a table view cell there are 2 included style options UITableViewCell.AccessoryType and UITableViewCell.CellStyle.

override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
    let reuseIdentifier = "reuseIdentifier"
    
    var cell:UITableViewCell? = tableView.dequeueReusableCell(withIdentifier: reuseIdentifier)
    
    if cell == nil {
        cell = UITableViewCell(style: UITableViewCell.CellStyle.default, reuseIdentifier: reuseIdentifier)
    }
    
    cell!.imageView?.image = UIImage(named: "exampleImage")
    cell!.textLabel?.text = "Text Label"
    cell!.detailTextLabel?.text = "Detail Label"
    cell!.accessoryType = UITableViewCell.AccessoryType.disclosureIndicator
    
    return cell!
}

UITableViewCellStyle

All Styles	Including Images

All styles & accessories	All styles, Images & accessories

UITableViewCellAccessoryType

All Styles	With Images

Understanding UIKit: CALayer Anchor Point

2022-04-26T02:11:47.947967+00:00

All on screen elements (normally) on iOS are made of UIView's. They have a frame, which have an x and y origin, and a width and height size.

The frame is actually a derived property. It comes from combining the bounds, center and transform properties. If you apply a transform to scale a view up by 2 its frame will grow, but its bounds will remain the same size.
The bounds is the view's internal size, the frame is the external size.

UIView Frame

You can see in this example, applying the scaling changes the frame, but not the bounds.

However, UIView's are actually a helper which adds touch and other event handling on top of the real on screen element. The real element which controls what appears on screen is the CALayer which backs each and every UIView.

CALayer's location on screen is controlled by very similar, but slightly different set of properties.
The bounds, for internal size.
The position, its location in its parent view.
The transform, which is a 3D transform now (not just a 2D one like UIView's transform property)

And finally the anchorPoint property.

CALayer anchorPoint

By default anchorPoint is (x: 0.5, y: 0.5). This is a relative value.
Instead of being a point value in the parent view/layer like position or bounds, this is relative to its width and height.
(x:0, y:0) is the top left corner.
(x:0.5, y:0.5) the middle of the view.
(x:1, y: 1) is the bottom right edge of the view.

The anchorPoint is the location inside the view/layer that the UIView.center or CALayer.position represents. Normally this defaults to the centre of the view.
However, if you change the anchor point the entire layer shifts over based on this pivot point.

The official documentation puts it nicely.

All geometric manipulations to the view occur about the specified point. For example, applying a rotation transform to a layer with the default anchor point causes the layer to rotate around its center. Changing the anchor point to a different location would cause the layer to rotate around that new point.

This hints at one of the big reasons you might want to use the anchorPoint property. It allows you to easily control how other properties, such as the transform (rotation and scaling) are applied to the views.

If we apply the same rotation to all the views from the above example you can see the different effects it has on each.

Outside the Box

All the examples above have the anchorPoint inside the view's bounds, as we would normally expect, with values between 0 and 1. However, there is nothing that requires that.
We can freely set values above, or below, that range.

An X value of 1 is the full width away from the left edge. So a value of 2 would be 2 full widths of the layer away from the left edge.

Similarly a value of -1 would be 1 full width away from the left edge of the layer.

All the examples below have the same centre point set, but different anchor points (and some rotation).

Although these examples use rotation, the same also applies to scaling. Scaling up or down a layer is done from the point of origin of its anchorPoint.
So if the anchorPoint is outside the bounds, when a layer is scaled up it will appear to expand away from the anchorPoint.

UIBlurEffectStyles

2022-04-01T00:58:43.764302+00:00

Blurred backgrounds are a staple of modern app design, and the Apple provided UIView to achieve that effect with is via applying a UIBlurEffect to a UIVisualEffectView view.

Example

let visualEffectView = UIVisualEffectView(effect: nil)
self.view.addSubview(visualEffectView)

visualEffectView.effect = UIBlurEffect(style: .systemMaterial)

UIVisualEffectView *visualEffectView = [[UIVisualEffectView alloc] initWithEffect:nil];
[self.view addSubview:visualEffectView];

visualEffectView.effect = [UIBlurEffect effectWithStyle:UIBlurEffectStyleSystemMaterial];

UIBlurEffectStyle

The different blur effects available are:

The original 4 from iOS 8

UIBlurEffectStyleExtraLight, // .extraLight
UIBlurEffectStyleLight,      // .light
UIBlurEffectStyleDark,       // .dark
UIBlurEffectStyleExtraDark   // .extraDark

Adaptive Default Styles

These will adapt to light and dark mode automatically as the user switches modes.

UIBlurEffectStyleRegular,  // .regular
UIBlurEffectStyleProminent // .prominent

Adaptive Styles

UIBlurEffectStyleSystemUltraThinMaterial // .systemUltraThinMaterial
UIBlurEffectStyleSystemThinMaterial      // .systemThinMaterial
UIBlurEffectStyleSystemMaterial          // .system
UIBlurEffectStyleSystemThickMaterial     // .systemThickMaterial
UIBlurEffectStyleSystemChromeMaterial.   // .ystemChromeMaterial

Example Project

To visualise these different styles I creates an example project which allows you to see each style in light and dark mode.

Screenshots

Examples of each UIBlurEffectStyle style

UIBlurEffectStyle	Screenshot
ExtraLight	Light

Dark	Regular

Prominent	SystemUltraThinMaterial

SystemThinMaterial	SystemMaterial

SystemThickMaterial	SystemChromeMaterial

SystemUltraThinMaterialLight	SystemThinMaterialLight

SystemMaterialLight	SystemThickMaterialLight

SystemChromeMaterialLight	SystemUltraThinMaterialDark

SystemThinMaterialDark	SystemMaterialDark

SystemThickMaterialDark	SystemChromeMaterialDark

Spherium 1.0 Released

2020-08-13T14:33:58.135459+00:00

Spherium lets you easily view and share 360 degree images saved in your Photo Library.

Spherium will automatically attempt to find the 360º images saved in your photo library from 360º cameras, drones, apps such as Google Street View, or anywhere else you have saved them from.
It looks for equirectangular images, images with a width equal to twice its height, with a minimum resolution of 1000 x 2000 pixels and lets you view and share those 360 photos.

Sharing

Spherium supports taking and sharing normal FOV (field of view) snapshots of 360º images, as if you had taken a normal photo.
This is really helpful for sharing the photos with friends and posting them on social media, like Instagram.

You can take one 360º photo and then take a portrait snapshot of it, for posting as an Instagram Story or WhatsApp status, and then also take a landscape or square snapshot for use in a normal Instagram post from the same photo.

1.0 Released Now

Spherium 1.0 is out now available on the iOS App Store.

Inspiration

I love taking 360º group photos with friends, or of beautiful landscapes. Capturing the entire view all around me. Taking a photo, not just of me and my friends filling the cameras view, but also of our surroundings and the view all around us.

You can take 360º photos with dedicated cameras, such as the Samsung Gear 360, Qoocam 8k, Insta360 One R, Insta360 One X, Go Pro Fusion and GoPro Max; drones some drones, such as the DJI Mavic Air and Spark; or even on your phone with apps, like Google's Street View app.

360º cameras and drones usually have companion apps for the devices that support downloading and viewing the photos they take. The problem is then you end up with a photo collection split across various different apps, some in one app, some in another, and your normal photos in your phones Photo Library. Then there is also the risk the manufacture discontinues support for the app.

The apps always have the ability to save the 360º photos to your Photo Library. Unfortunately iOS does not support viewing 360º photos natively, only showing the raw stretched image itself.

I wanted to be able to save all my photos, wherever they came from, in one place and view them all. That's why I built Spherium.

Spherium 1.0 is available now

How to Read and Write Image File Metadata with CoreGraphics

2020-07-13T23:33:52.630391+00:00

UIImage allows you to very easily save the image to a file, with UIImagePNGRepresentation(_) or UIImageJPEGRepresentation(_, _). However, those new images will be new, with the current date and time, and containing no location or camera metadata. If you want to edit an image without losing those things, that's a problem.

Fortunately, CoreGraphics includes several easy API's to access the image files raw metadata.
However, these are the raw values themselves. If you want a CLLocation object from the GPS metadata you'll have to parse the "{GPS}" section of the metadata yourself.

Read Image Metadata

To read the image file metadata you only need a CoreGraphics image source and then to pass that image source to the CGImageSourceCopyPropertiesAtIndex(_, _, _) function. This function then returns a CFDictionaryRef, which can just be cast to an NSDictionary, containing all the image file metadata.

CGImageSourceRef source = CGImageSourceCreateWithURL((__bridge CFURLRef)url, NULL);
CFDictionaryRef imageProperties = CGImageSourceCopyPropertiesAtIndex(source, 0, NULL);
NSLog(@"%@", (__bridge NSDictionary *)(imageProperties) );

let data = NSData(contentsOf: url)!
let source = CGImageSourceCreateWithData(data as CFData, nil)!
let metadata = CGImageSourceCopyPropertiesAtIndex(source, 0, nil)!
print(metadata)

Example Image Metadata

{
    ColorModel = RGB;
    Depth = 8;
    PixelHeight = 5632;
    PixelWidth = 11264;
    ProfileName = "sRGB IEC61966-2.1";
    "{Exif}" =     {
        ColorSpace = 1;
        DateTimeDigitized = "2000:02:12 20:20:14";
        DateTimeOriginal = "2000:02:12 20:20:14";
        PixelXDimension = 11264;
        PixelYDimension = 5632;
    };
    "{GPS}" =     {
        Altitude = "3.522659318754763";
        AltitudeRef = 0;
        Latitude = "50.79826333333333";
        LatitudeRef = N;
        Longitude = "0.5709116666666667";
        LongitudeRef = W;
    };
    "{IPTC}" =     {
        DateCreated = 20000612;
        DigitalCreationDate = 20000612;
        DigitalCreationTime = 202014;
        TimeCreated = 202014;
    };
    "{JFIF}" =     {
        DensityUnit = 0;
        JFIFVersion =         (
            1,
            0,
            1
        );
        XDensity = 72;
        YDensity = 72;
    };
    "{TIFF}" =     {
        Make = Apple;
        Model = "iPhone11,1 - 13.1.1";
    };
}
{
    "{Exif}" =     {
        DateTimeDigitized = "2010:01:12 20:20:14";
        DateTimeOriginal = "2010:01:12 20:20:14";
    };
    "{GPS}" =     {
        Altitude = "2.522659318754763";
        AltitudeRef = 0;
        Latitude = "53.79826333333333";
        LatitudeRef = N;
        Longitude = "-1.5709116666666667";
        LongitudeRef = W;
    };
}

Write Image Metadata

To write image file metadata is very similar. Provided with an image source, and the type of image file being written to, we can create a new CGImageDestinationRef and then write in the image and metadata with CGImageDestinationAddImageFromSource(_, _, _, _).

CGImageSourceRef source = CGImageSourceCreateWithURL((__bridge CFURLRef)url, NULL);
CFStringRef uniformTypeIdentifier = CGImageSourceGetType(source);

CGImageDestinationRef destination = CGImageDestinationCreateWithURL( (__bridge CFURLRef)url, uniformTypeIdentifier, 1, NULL);
CGImageDestinationAddImageFromSource(destination, source, 0, (__bridge CFDictionaryRef)metadata);
CGImageDestinationFinalize(destination);

guard let source = CGImageSourceCreateWithURL(url as CFURL, nil),
    let uniformTypeIdentifier = CGImageSourceGetType(source) else { return }

guard let destination = CGImageDestinationCreateWithURL(url as CFURL, uniformTypeIdentifier, 1, nil) else { return }
CGImageDestinationAddImageFromSource(destination, source, 0, metadata)
guard CGImageDestinationFinalize(destination) else { return }

Example Projects

The example code Xcode Projects demonstrating how to use the new API are available on Github:

The example projects support iOS and macOS and allow you to select an image file. The XML representation of the metadata dictionary is then loaded into an editable text view. You can then save the edited metadata and export the resulting file.

Using SwiftUI Previews with UIKit and ObjC

2020-07-01T23:34:35.165396+00:00

One of the main selling points of SwiftUI is the live preview feature in Xcode. However, it is also possible to use this new feature with UIKit views, even ones written in Objective C.

Xcode Preview Setup Steps

Starting with a new Objective C Xcode project. Create a new SwiftUI View file.

Accept the creation of the Objective-C bridging header.
Add your UIViewController's to the bridging header.
Replace the SwiftUI view with a SwiftUI UIViewController wrapper.

import SwiftUI

struct VCRepresentable<VCClass:UIViewController>: UIViewControllerRepresentable {
    func updateUIViewController(_ uiViewController: UIViewController, context: Context) {
        // leave this empty
    }
    
    @available(iOS 13.0.0, *)
    func makeUIViewController(context: Context) -> UIViewController {
        return VCClass()
    }
}

struct SwiftUIView_Previews: PreviewProvider {
    static var previews: some View {
       VCRepresentable<ViewController>()
    }
}

Xcode won't auto-refresh any changes made outside of the body method, so to refresh you need to press the Resume button or use the keyboard shortcut CMD+ALT+P.

Xcode takes a shortcut to reload a SwiftUI view, but needs to do a rebuild for changes made outside of the views body method (like changes to the ObjC file). Fortunately, if you are using ObjC, the rebuild will be fast so that doesn't matter very much.

For more information on using Xcode Preview's with UIKit in general @soulchildpls's "Use Xcode Previews with UIKit" article goes into more details.

Example Project

Objective C Example Project

SwiftUI Gradient Text

2020-06-27T23:04:00.593509+00:00

When searching how to implement gradient text with SwiftUI the solution I found was to use a LinearGradient and use the Text element as the mask for that View. However, LinearGradient wants to expand to fill all the available space, and all the examples I found used fixed size Views.

The implementation I came up with to implement this for arbitrary fonts and text was to:

Create a transparent text, to set the needed size.
Put the LinearGradient into the text's background property.
Place a second, duplicate, Text element as the mask for the gradient.

// Custom View
struct GradientText: View {
   
   @State var text: String
   @State var gradient:LinearGradient = LinearGradient(
      gradient: Gradient(colors: [.white, .gray]),
      startPoint: .top,
      endPoint: .bottom
   )
   
   var body: some View {
      Text(text)
      .foregroundColor(Color.clear)
      .background(gradient.mask(Text(self.text)))
   }
   
}

// Example Usage
struct Gradient_Previews: PreviewProvider {
    static var previews: some View {
      GradientText(text: "Hello World Gradient Text!")
         .font(.system(size: 50, weight: .bold))
         .padding()
         .background(Color.black)
         .previewLayout(PreviewLayout.sizeThatFits)
         .padding()
         .previewDisplayName("Example Gradient Text")
    }
}

If anyone knows a better solution without the duplicate Text element please let me know on Twitter or Mastodon.

Using PHPickerViewController to Select a Photo on iOS 14

2020-06-23T03:59:40.595178+00:00

For many years the simplest way to selection photos and videos on iOS has been to use the UIImagePickerController class. The class allowed you to present a built in system UI to select a photo or video and return it to your app, without having to built the selection UI or prompt for access to the photo library.

The code to do so was incredibly simple:

// Objective C
- (void)pickPhoto
{
    UIImagePickerController *imagePicker = [[UIImagePickerController alloc] init];
    imagePicker.sourceType = UIImagePickerControllerSourceTypePhotoLibrary;
    imagePicker.delegate = self;
    [self presentViewController:imagePicker animated:YES completion:nil];
}

// Implement UIImagePickerControllerDelegate method
-(void)imagePickerController:(UIImagePickerController *)picker didFinishPickingMediaWithInfo:(NSDictionary<UIImagePickerControllerInfoKey,id> *)info
{
    UIImage *image = info[UIImagePickerControllerOriginalImage];
    self.imageView.image = image;

    [picker dismissViewControllerAnimated:YES completion:nil];
}

// Swift
func pickPhoto()
{
    let imagePicker = UIImagePickerController()
    imagePicker.sourceType = .photoLibrary
    imagePicker.delegate = self
    present(imagePicker, animated: true)
}

// Implement UIImagePickerControllerDelegate method
public func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey: Any])
{
    imageView.image = (info[.originalImage] as? UIImage)
    picker.dismiss(animated: true, completion: nil)
}

However, UIImagePickerController had a number of disadvantages: it was fairly basic and limited in the UI it gave the users to browse their library with; it only allowed selection of one item at a time, and had only basic filtering support. Now in iOS 14 the UIImagePickerController is "soft deprecated". Although not currently marked as deprecated, if you look at the header file you'll see the API marker with this:

API_DEPRECATED("Will be removed in a future release, use PHPicker.", ios(11, API_TO_BE_DEPRECATED));

API_TO_BE_DEPRECATED is defined with this comment above it.

/* API_TO_BE_DEPRECATED is used as a version number in API that will be deprecated 
 * in an upcoming release. This soft deprecation is an intermediate step before formal 
 * deprecation to notify developers about the API before compiler warnings are generated.
...

Instead iOS 14 now includes the PHPickerViewController with support for multiple selection and a much improved UI.

PHPicker

Rather than being in UIKit, the new PHPicker. classes in iOS 14 are located in the PhotosUI framework and include:

You present a PHPickerViewController, which has a PHPickerConfiguration to tell it how many items to select, and of what type. The types of items allowed are defined by the PHPickerConfiguration's filter property, which has the possible option of being any combination of: images, live photos, or videos. How many items users can select is controlled from the PHPickerConfiguration's selectionLimit property.

PHPickerConfiguration *config = [[PHPickerConfiguration alloc] init];
config.selectionLimit = 3;
config.filter = [PHPickerFilter imagesFilter];

PHPickerViewController *pickerViewController = [[PHPickerViewController alloc] initWithConfiguration:config];
pickerViewController.delegate = self;
[self presentViewController:pickerViewController animated:YES completion:nil];

PHPickerConfiguration, PHPickerFilter, and PHPickerResult all bridge across into Swift as Structs, not as classes.

var config = PHPickerConfiguration()
config.selectionLimit = 3
config.filter = PHPickerFilter.images

let pickerViewController = PHPickerViewController(configuration: config)
pickerViewController.delegate = self
self.present(pickerViewController, animated: true, completion: nil)

The results from PHPickerViewController are returned differently than UIImagePickerController, instead using NSItemProvider, like the drag and drop APIs.

NOTE: There is an assetIdentifier property defined on PHPickerResult, which is documented as Local identifier of the selected asset. However, (as of iOS 14.0 beta 1, in my testing this is always nil; regardless of whether or not the app has been granted full photo library access.
I presume that is meant to be the PHAsset's assetIdentifier allowing us to retrieve the asset object with a call to asset = PHAsset.fetchAssets(withLocalIdentifiers: [photoID], options: nil).firstObject.

// Objective C

-(void)picker:(PHPickerViewController *)picker didFinishPicking:(NSArray<PHPickerResult *> *)results{
   [picker dismissViewControllerAnimated:YES completion:nil];
    
   for (PHPickerResult *result in results)
   {
      // Get UIImage
      [result.itemProvider loadObjectOfClass:[UIImage class] completionHandler:^(__kindof id<NSItemProviderReading>  _Nullable object, NSError * _Nullable error)
      {
         if ([object isKindOfClass:[UIImage class]])
         {
            dispatch_async(dispatch_get_main_queue(), ^{
               NSLog(@"Selected image: %@", (UIImage*)object);
            });
         }
      }];
   }
}

// Swift
func picker(_ picker: PHPickerViewController, didFinishPicking results: [PHPickerResult]) {
   picker.dismiss(animated: true, completion: nil)
   
   for result in results {
      result.itemProvider.loadObject(ofClass: UIImage.self, completionHandler: { (object, error) in
         if let image = object as? UIImage {
            DispatchQueue.main.async {
               // Use UIImage
               print("Selected image: \(image)")
            }
         }
      })
   }
}

Filtering and Multi-selection

By default the selectionLimit is set to 1, and there is no filter.

The change what is selected, and how many, we just need to change the PHPickerConfiguration.
For example: to select up to 10 videos we would need to specify.

PHPickerConfiguration *config = [[PHPickerConfiguration alloc] init];
config.selectionLimit = 10;
config.filter = [PHPickerFilter videosFilter];

PHPickerConfiguration, PHPickerFilter, and PHPickerResult all bridge across into Swift as Structs, not as classes.

var config = PHPickerConfiguration()
config.selectionLimit = 10
config.filter = PHPickerFilter.videos

Setting the selection limit to 0 allows selecting as many items as the system supports.

Example Projects

The example code Xcode Projects demonstrating how to use the new API are available on Github:

External Only Shadow on UIView

2020-06-16T03:06:29.634590+00:00

To add a shadow to a UIView is very easy, all you have to do is drop down to the CALayer underneath and setup it's shadow properties.

// Objective C
view.layer.shadowColor = [UIColor blackColor].CGColor;
view.layer.shadowOffset = CGSizeMake(0, 1);
view.layer.shadowRadius = 10;
view.layer.shadowOpacity = 1.0;

// Swift
view.layer.shadowColor = UIColor.black.cgColor
view.layer.shadowOffset = CGSize(width:0, height:1);
view.layer.shadowRadius = 10;
view.layer.shadowOpacity = 1.0;

However, if your view has a partially, or completely, transparent background color this can look a bit weird.

Original:

// ObjC
view.backgroundColor = [UIColor colorWithRed:0 green:1 blue:0 alpha:0.5];
// Swift
view.backgroundColor = UIColor(red:0, green:1, blue:0, alpha:0.5)

With shadow:

To fix this we can use use CALayer's maskLayer property to create an external only shadow.

First we need to create a mask layer the size and change of our original view.

// ObjC
CAShapeLayer *maskLayer = [CAShapeLayer layer];
maskLayer.frame = view.bounds;
maskLayer.path = [UIBezierPath bezierPathWithRoundedRect:view.bounds cornerRadius:view.layer.cornerRadius].CGPath;

// Swift
let maskLayer = CAShapeLayer()
maskLayer.frame = view.bounds
maskLayer.path = UIBezierPath(roundedRect: view.bounds, cornerRadius: view.layer.cornerRadius).cgPath

Then we need to move it back, up and make it bigger, so it won't clip the shadow itself.

// ObjC
CGFloat shadowBorder = (view.layer.shadowRadius * 2) + 5;
maskLayer.frame = CGRectInset( maskLayer.frame, -shadowBorder, -shadowBorder ); // Make bigger
maskLayer.frame = CGRectOffset( maskLayer.frame, shadowBorder/2.0, shadowBorder/2.0 ); // Move up and left

// Swift

// Make the mask area bigger than the view, so the shadow itself does not get clipped by the mask
let shadowBorder:CGFloat = (view.layer.shadowRadius * 2) + 5;
maskLayer.frame = maskLayer.frame.insetBy(dx:  -shadowBorder, dy:  -shadowBorder)  // Make bigger
maskLayer.frame = maskLayer.frame.offsetBy(dx: shadowBorder/2, dy: shadowBorder/2) // Move up and left

And set the fill rule to allow cut outs inside the shape.

maskLayer.fillRule = kCAFillRuleEvenOdd;

maskLayer.fillRule = CAShapeLayerFillRule.evenOdd

Lastly we create a mutable path and use it to rewrite the mask layer's path to one with a cut out a hole in the center where the original view was.

// ObjC
CGMutablePathRef pathMasking = CGPathCreateMutable();
// Add the outer view frame
CGPathAddPath(pathMasking, NULL, [UIBezierPath bezierPathWithRect:maskLayer.frame].CGPath);
// Translate into the shape back to the smaller original view's frame start point
CGAffineTransform catShiftBorder = CGAffineTransformMakeTranslation( shadowBorder/2.0, shadowBorder/2.0);
// Now add the original path for the cut out the shape of the original view
CGPathAddPath(pathMasking, NULL, CGPathCreateCopyByTransformingPath(maskLayer.path, &catShiftBorder ) );
// Set this big rect with a small cutout rect as the mask
maskLayer.path = pathMasking;

// Swift
let pathMasking = CGMutablePath()
// Add the outer view frame
pathMasking.addPath(UIBezierPath(rect: maskLayer.frame).cgPath)
// Translate into the shape back to the smaller original view's frame start point
var catShiftBorder = CGAffineTransform(translationX: shadowBorder/2, y: shadowBorder/2)
// Now add the original path for the cut out the shape of the original view
pathMasking.addPath(maskLayer.path!.copy(using: &catShiftBorder)!)
// Set this big rect with a small cutout rect as the mask
maskLayer.path = pathMasking;

Not forgetting to actually set the view's maskLayer to our new layer.

view.layer.mask = maskLayer

This has given us our transparent center, but it will also crop out the background color and any child views placed inside.
As a result, this technique requires a full size 'shadow view' added as the bottom most child of the view you want to add a shadow to, rather than applying this directly to the view itself.

Example Xcode Projects

There are Swift and Objective C example Xcode projects showing this technique available on Github:

CAGradientLayer Explained

2020-06-15T22:19:33.819892+00:00

iOS and macOS have built in support for drawing gradients in QuartzCore with the CAGradientLayer class, no custom CoreGraphics drawing in -drawRect: with CGContextDrawLinearGradient necessary.

Another advantage of CAGradientLayer over using CoreGraphics, besides the convenience, is that CAGradientLayer is rendered directly by the GPU itself. Compared to drawing a gradient with CoreGraphics, which will draw into a buffer on the CPU and then send that to the GPU.

Linear (Axial) Gradients

To create a linear gradient from one color to another (or several) the layer's type must be set to kCAGradientLayerAxial (.axial). The colors must be CGColor's, not UIColors. If you forgot this the color object is just ignored completely.

// Objective C

gradientLayer.colors =
@[
    (id)[UIColor colorWithRed: 48.0/255.0 green: 35.0/255.0 blue: 174.0/255.0 alpha: 1.0].CGColor,
    (id)[UIColor colorWithRed: 200.0/255.0 green: 109.0/255.0 blue: 215.0/255.0 alpha: 1.0].CGColor
];

// Swift

gradientLayer.colors =
[
    UIColor(red: 48.0/255.0, green: 35.0/255.0, blue: 174.0/255.0, alpha: 1.0).cgColor,
    UIColor(red: 200.0/255.0, green: 109.0/255.0, blue: 215.0/255.0, alpha: 1.0).cgColor
]

Start Point and End Point

The start and end point of the gradient can be changed to get different effects.
For gradients the X and Y axis go from 0 to 1, across their length. So a start point of (x:0, y:0) is the top left corner.

By default the start point is (0.5, 0.0), half way across the horizontal and at the top vertically, and the end point is the bottom middle of the view (0.5, 1.0).

The code to create this example horizontal gradient is:

// Objective C

// Set type (Axial is already the default value)
gradientLayer.type = kCAGradientLayerAxial;
// Set the colors (these need to be CGColor's, not UIColor's)
gradientLayer.colors =
@[
    (id)[UIColor colorWithRed: 48.0/255.0 green: 35.0/255.0 blue: 174.0/255.0 alpha: 1.0].CGColor,
    (id)[UIColor colorWithRed: 200.0/255.0 green: 109.0/255.0 blue: 215.0/255.0 alpha: 1.0].CGColor
];
// Set the start and end points
gradientLayer.startPoint = CGPointMake(0, 0);
gradientLayer.endPoint = CGPointMake(1, 0);

// Swift

// Set type (Axial is already the default value)
gradientLayer.type = CAGradientLayerType.axial
// Set the colors (these need to be CGColor's, not UIColor's)
gradientLayer.colors =
[
    UIColor(red: 48.0/255.0, green: 35.0/255.0, blue: 174.0/255.0, alpha: 1.0).cgColor,
    UIColor(red: 200.0/255.0, green: 109.0/255.0, blue: 215.0/255.0, alpha: 1.0).cgColor
]
// Set the start and end points
gradientLayer.startPoint = CGPoint(x: 0, y: 0)
gradientLayer.endPoint = CGPoint(x: 1, y: 0)

Multiple Colors and Positions

Most of my uses for gradients only involve 2 colors, but the colors property is an array of colors are you can add as many as you want.

gradientLayer.colors =
@[
    (id)[UIColor blueColor].CGColor,
    (id)[UIColor orangeColor].CGColor,
    (id)[UIColor greenColor].CGColor,
    (id)[UIColor redColor].CGColor,
    (id)[UIColor purpleColor].CGColor
];

The locations property can be used to control how the colors are distributed across the layer.

// ObjC
gradientLayer.locations = @[
    @0,   // blueColor
    @0.1, // orangeColor
    @0.6, // greenColor
    @0.7, // redColor
    @1    // purpleColor
];

// Swift
gradientLayer.locations = [
    0,   // blueColor
    0.1, // orangeColor
    0.6, // greenColor
    0.7, // redColor
    1    // purpleColor
]

Radial Gradients

To create a radial gradient out from one point the layer's type must be set to kCAGradientLayerRadial (.radial).

The start and end point are important, as the endPoint's x and y define the end of the second color.

With a startPoint in the center (0.5, 0.5), an endPoint x of 0 or 1 will draw from the center to the outer edge of the view horizontally.
An endPoint Y of 1 or 0 will draw the gradient to the height of the view.

The code to create this example gradient is:

// Objective C

// Set the type
gradientLayer.type = kCAGradientLayerRadial;
gradientLayer.colors =
@[
    (id)[UIColor colorWithRed: 0.0/255.0 green: 101.0/255.0 blue: 255.0/255.0 alpha: 1.0].CGColor,
    (id)[UIColor colorWithRed: 0.0/255.0 green: 40.0/255.0 blue: 101.0/255.0 alpha: 1.0].CGColor
];
// Start in the center
gradientLayer.startPoint = CGPointMake(0.5, 0.5);
// End at the outer edge of the view
gradientLayer.endPoint = CGPointMake(0, 0.75);

// Swift

// Set type to radial
gradientLayer.type = CAGradientLayerType.radial
// Set the colors
gradientLayer.colors =
[
    UIColor(red: 0.0/255.0, green: 101.0/255.0, blue: 255.0/255.0, alpha: 1.0).cgColor,
    UIColor(red: 0.0/255.0, green: 40.0/255.0, blue: 101.0/255.0, alpha: 1.0).cgColor
]
// Start point of first color in the middle of the view
gradientLayer.startPoint = CGPoint(x: 0.5, y: 0.5)
// End points to the edges of the view
gradientLayer.endPoint = CGPoint(x: 0, y: 0.75)

Angular (Conic) Gradients

Whereas linear and radial gradients have been part of the iOS SDK forever (iOS 3.0 & 3.2 respectively & macOS 10.6) angular, or conic as CoreAnimation calls them, gradients were only added in iOS 12.0 (macOS 10.14).

The startPoint is the point at which the gradient is drawn around.
The endPoint is the point the start/end line draws aiming at.

The code to create the example angular gradient is:

// Objective C

gradientLayer.type = kCAGradientLayerConic;
// Set the colors
gradientLayer.colors =
@[
    (id)[UIColor blueColor].CGColor,
    (id)[UIColor colorWithRed: 50.0/255.0 green: 251.0/255.0 blue: 255.0/255.0 alpha: 1.0].CGColor,
    (id)[UIColor blackColor].CGColor
];
// Start point of first color in the middle of the view
gradientLayer.startPoint = CGPointMake(0.5, 0.5);
// End points to the edges of the view
gradientLayer.endPoint = CGPointMake(0.5, 0);

// Swift

gradientLayer.type = CAGradientLayerType.conic
gradientLayer.colors =
[
    UIColor.blue,
    UIColor(red: 50.0/255.0, green: 251.0/255.0, blue: 255.0/255.0, alpha: 1.0).cgColor,
    UIColor.black
]
gradientLayer.startPoint = CGPoint(x: 0.5, y: 0.5)
gradientLayer.endPoint = CGPoint(x: 0.5, y: 0)

UIView Wrapper

As layers are a little annoying to work with in UIKit, with implicit animations and separate layout, I almost always use it via wrapping it in a custom UIView subclass.
You can actually change the underlying type of a UIView by overriding the +(Class)layerClass class method and returning any CALayer subclass.

Objective C Subclass. Redefining the layer property means view.layer will happily return the correct layer class, and @dynamic layer; tells it not to auto-synthesize anything for that deceleration.

// KHGradientView.h
@import UIKit;
@import QuartzCore;

@interface KHGradientView : UIView
@property(nonatomic, readonly, strong) CAGradientLayer *layer;
@end


// KHGradientView.m
#import "KHGradientView.h"

@implementation KHGradientView
@dynamic layer;

+(Class)layerClass{
    return [CAGradientLayer class];
}

@end

// Swift Subclass

import UIKit

class KHGradientView: UIView {
    override open class var layerClass: AnyClass {
       return CAGradientLayer.classForCoder()
    }
    var gradientLayer : CAGradientLayer {
        return self.layer as! CAGradientLayer
    }
}

It's incredibly minimal, but makes things much easier in the rest of the codebase.

Example Project

The example code Xcode Project to produce the opening example image is available on Github:

Mac Catalyst Title Bar Window Tabs

2020-06-01T00:57:38.826645+00:00

To start with let's take a simple tab bar based app with a label in the UIViewController's' telling us which view we are looking at.
If we just take this app and compile and run it for mac Catalyst this is the result we get.

This works... but feels a lot like running the app in the iOS simulator.
You probably want to change the style, to either the side tabs, like the AppStore app, or titlebar tabs, like the Calendar app.

Most of the time I want the tabs in the titlebar. (if you want the sidebar style, that's UISplitViewController + .primaryBackgroundStyle = .sidebar)

Putting Tabs in the Title Bar

To do that we need to customise the window's tilebar. In iOS, the application window (as in multi-window support, not UIWindow or NSWindow) is now controlled from the UISceneDelegate.

In the SceneDelegate we setup an NSToolbar, set the SceneDelegate as the NSToolbar delegate and then hide both the title bar's title itself, after assigning the toolbar, & the tab bar.
Inside the SceneDelegate class add:

Objective-C (ObjC)
SceneDelegate.m

// SceneDelegate.m
#if TARGET_OS_MACCATALYST
#import <AppKit/AppKit.h>
@interface SceneDelegate () <NSToolbarDelegate>
#else
@interface SceneDelegate ()
#endif


// -[SceneDelegate scene:willConnectToSession:options:] {

#if TARGET_OS_MACCATALYST
NSToolbar *toolbar = [[NSToolbar alloc] initWithIdentifier:@"mainToolbar"];
toolbar.centeredItemIdentifier = @"mainTabsToolbarItem";
toolbar.delegate = self;

[(UIWindowScene*)scene titlebar].toolbar = toolbar;
[(UIWindowScene*)scene titlebar].titleVisibility = UITitlebarTitleVisibilityHidden;

_tabbarController.tabBar.hidden = YES;
#endif

Swift
SceneDelegate.swift

// SceneDelegate.swift
// SceneDelegate.scene(_:willConnectTo:options:) {

#if targetEnvironment(macCatalyst)
let toolbar = NSToolbar(identifier: "mainToolbar")
toolbar.centeredItemIdentifier = NSToolbarItem.Identifier(rawValue: "mainTabsToolbarItem")
toolbar.delegate = self

windowScene.titlebar?.titleVisibility = .hidden
windowScene.titlebar?.toolbar = toolbar

tabBarController.tabBar.isHidden = true
#endif

Most of the NSToolbar stuff is actually configured via the NSToolbarDelegate methods.

Here we are: specifying which items can be added to the toolbar toolbarDefaultItemIdentifiers: & toolbarAllowedItemIdentifiers:; setting up a new NSToolbarItemGroup item, specifying the segment control in the center of the title bar; and then .

ObjC

// SceneDelegate.m
#pragma mark - NSToolbarDelegate

#if TARGET_OS_MACCATALYST

// NSToolbarItemIdentifier is just an NSString*
- (nullable NSToolbarItem *)toolbar:(NSToolbar *)toolbar itemForItemIdentifier:(NSToolbarItemIdentifier)itemIdentifier willBeInsertedIntoToolbar:(BOOL)flag
{
   if ([itemIdentifier isEqualToString:@"mainTabsToolbarItem"]) {
      NSToolbarItemGroup *group = [NSToolbarItemGroup groupWithItemIdentifier:itemIdentifier
                                                       titles:@[@"First", @"Second"]
                                                  selectionMode:NSToolbarItemGroupSelectionModeSelectOne
                                                       labels:@[@"First view", @"Second view"]
                                                       target:self
                                                       action:@selector(toolbarGroupSelectionChanged:)];
      [group setSelectedIndex:0];
      return group;
   }
   
   return nil;
}

/* Returns the ordered list of items to be shown in the toolbar by default.   If during initialization, no overriding values are found in the user defaults, or if the user chooses to revert to the default items this set will be used. */
- (NSArray<NSToolbarItemIdentifier> *)toolbarDefaultItemIdentifiers:(NSToolbar *)toolbar{
   return @[@"mainTabsToolbarItem"];
}

/* Returns the list of all allowed items by identifier.  By default, the toolbar does not assume any items are allowed, even the separator.  So, every allowed item must be explicitly listed.  The set of allowed items is used to construct the customization palette.  The order of items does not necessarily guarantee the order of appearance in the palette.  At minimum, you should return the default item list.*/
- (NSArray<NSToolbarItemIdentifier> *)toolbarAllowedItemIdentifiers:(NSToolbar *)toolbar{
   return [self toolbarDefaultItemIdentifiers:toolbar];
}


-(void)toolbarGroupSelectionChanged:(NSToolbarItemGroup*)sender{
   self.tabbarController.selectedIndex = sender.selectedIndex;
}
#endif

Swift

// SceneDelegate.swift
// MARK: - NSToolbarDelegate

func toolbar(_ toolbar: NSToolbar, itemForItemIdentifier itemIdentifier: NSToolbarItem.Identifier, willBeInsertedIntoToolbar flag: Bool) -> NSToolbarItem?
{
   if (itemIdentifier.rawValue == "mainTabsToolbarItem")
   {
      let group = NSToolbarItemGroup.init(itemIdentifier: itemIdentifier,
                                 titles: ["First", "Second"],
                                 selectionMode: .selectOne,
                                 labels: ["First view", "Second view"],
                                 target: self, action: #selector(toolbarGroupSelectionChanged))
      group.setSelected(true, at: 0)
      return group
   }
   
   return nil
}

func toolbarDefaultItemIdentifiers(_ toolbar: NSToolbar) -> [NSToolbarItem.Identifier] {
   return [NSToolbarItem.Identifier(rawValue: "mainTabsToolbarItem")]
}

func toolbarAllowedItemIdentifiers(_ toolbar: NSToolbar) -> [NSToolbarItem.Identifier] {
   return self.toolbarDefaultItemIdentifiers(toolbar)
}


@objc func toolbarGroupSelectionChanged(_ sender: NSToolbarItemGroup) {
   tabBarController.selectedIndex = sender.selectedIndex
}

If you get the error message:

ERROR: Declaration of 'NSToolbarItemGroup' must be imported from module 'AppKit.NSToolbarItemGroup' before it is required

This means you haven't imported the Appkit headers. The fix is to @include AppKit; or #import <AppKit/AppKit.h>.

Before and after:

Looking at the result it might seem that the labels part has been ignored. However, if you turn on customisation of the toolbar, or remove the hiding of the title in the titlebar you can see where it is used.

Example Projects

Made with Xcode 11.4.1 targeting iOS 13.0

Spherium: 360º Photos Explained

2020-04-27T23:29:15.119018+00:00

Spherium searches your iOS library for 360º degree equirectangular images.

What is a 360 Photo

Example

Normal photos capture what is in front of the camera. 360º photos capture what is all around the camera in every direction.

What is Equirectangular Projection

The wikipedia article on it has more information, but effectively a equirectangular projection is a way of stretching the pixels from a sphere into a flat 2:1 image.

Spherium looks for images which are exactly 2:1 (w:h) to indicate they might be a 360º photo.

How Do I Take a 360 Photo

You typically use a dedicated 360 camera, which captures 2 images from fisheye lens at slightly more than 180 degrees and then merges the two images into one 360 image.
However, there are also apps that can stitch together a 360º panorama on device from a series of photos in different directions; and several drones have a 360 panorama capture mode.

How Do I View 360 Photos

Spherium automatically shows you all the 360 photos in your iOS Photo Library and allows you to view them on your iPhone or iPad.

iOS 13 UIContextMenu UIMenu UIAction

2020-04-26T00:22:31.834634+00:00

Example opening paragraph

Example sub heading

Example content.

How to Add Chapters to MP4s with FFmpeg

2020-04-14T23:54:50.491259+00:00

FFmpeg is one of those pieces of software which can do practically anything, if you can work out how. One of the things it can do it modify the metadata for video files, including adding chapters.

Adding chapters is done via ffmetadata files.

Extract Metadata From Video

ffmpeg -i INPUT.mp4 -f ffmetadata FFMETADATAFILE.txt

Example output from a video file with no chapters.

;FFMETADATA1
title=None
major_brand=isom
minor_version=512
compatible_brands=isomiso2avc1mp41
encoder=Lavf57.56.100

Write Metadata To Video

The metadata file can be edited, including to add chapters, and then ffmpeg can write it to a file.
ffmpeg doesn't mutate the existing video, but instead creates a new video with the new metadata, existing video and existing audio.

ffmpeg -i INPUT.mp4 -i FFMETADATAFILE.txt -map_metadata 1 -codec copy OUTPUT.mp4

Adding Chapters

Adding chapters is done by editing the metadata file and adding sections like this:

[CHAPTER]
TIMEBASE=1/1000
START=1
END=448000
title=The Pledge

[CHAPTER]
TIMEBASE=1/1000
START=448001
END= 3883999
title=The Turn

[CHAPTER]
TIMEBASE=1/1000
START=3884000
END=4418000
title= The Prestige

Helper Script

The requirement to specify times as an integer of some time base, instead of the 1:30:20.5 style format usually used for displaying time is very off-putting to me, so I created a small helper script so I could just note down times while watching a video and have the metadata entries automatically worked out how my automatically.

The script has a few requirements:

It expects to be in the same directory as a 'chapters.txt' file to read from, and an 'FFMETADATAFILE' file to output to.
It expects the chapters.txt file to be a txt file with 1 chapter per line starting with 0:00:00 style time stamps and followed by the title of the chapter
It appends to an existing FFMETADATAFILE
It expects no existing chapters in the file
It expects an final END chapter which is ignored, but provides the end point for the last real chapter.

Example Chapters.txt

0:23:20 Start
0:40:30 First Performance
0:40:56 Break
1:04:44 Second Performance
1:24:45 Crowd Shots
1:27:45 Credits

Helper script

import re

chapters = list()

with open('chapters.txt', 'r') as f:
   for line in f:
      x = re.match(r"(\d):(\d{2}):(\d{2}) (.*)", line)
      hrs = int(x.group(1))
      mins = int(x.group(2))
      secs = int(x.group(3))
      title = x.group(4)

      minutes = (hrs * 60) + mins
      seconds = secs + (minutes * 60)
      timestamp = (seconds * 1000)
      chap = {
         "title": title,
         "startTime": timestamp
      }
      chapters.append(chap)

text = ""

for i in range(len(chapters)-1):
   chap = chapters[i]
   title = chap['title']
   start = chap['startTime']
   end = chapters[i+1]['startTime']-1
   text += f"""
[CHAPTER]
TIMEBASE=1/1000
START={start}
END={end}
title={title}
"""


with open("FFMETADATAFILE", "a") as myfile:
    myfile.write(text)

The script could be modified to remove the need for a chapters.txt file by swapping line 5's with. ... statement for this.

chapters_text = """0:40:30 First Performance
0:40:56 Break
... etc ...
"""
f = chapters_text.splitlines()

Example Usage With a Video With No Chapters

Get existing video metadata

ffmpeg -i INPUT.mp4 -f ffmetadata FFMETADATAFILE

Check there are no existing chapters.
Watch the video, noting chapters into a chapters.txt file as you go.
Place FFMETADATAFILE , chapters.txt, and the video file in the same directory.

Run the helper script to append chapters to FFMETADATAFILE.

python3 helper.py

Create a new video, copying the video and audio from the original without re-encoding.

ffmpeg -i INPUT.mp4 -i FFMETADATAFILE -map_metadata 1 -codec copy OUTPUT.mp4

Useful FFmpeg Commands for Sharing Videos

2020-04-27T20:40:55.314617+00:00

A collection of FFmpeg commands I find helpful when working with and sharing videos.

Convert to MP4

ffmpeg -i input.avi output.mp4

Convert from MKV to MP4 Without Re-encoding

This obviously only works the MKV contains a video that can be put in an MP4, like h264 or h265.

ffmpeg -i input.mkv -c copy output.mp4

Or you can specify the audio and video separately (and convert the audio by leaving it out, for example).

ffmpeg -i input.mkv -vcodec copy -acodec copy output.mp4
ffmpeg -i input.mkv -vcodec copy output.mp4

Change Framerate

ffmpeg -i input60fps.mp4 -r 30 output30fps.mp4

Compress For Sharing On The Web

ffmpeg -i input.mp4 -c:v libx264 -crf 25 -c:a aac -movflags faststart output.mp4

The compression factor is set with -crf, the ffmpeg default for h264 is 23 and 28 for h265.

With x264 and x265, you can set the values between 0 and 51, where lower values would result in better quality, at the expense of higher file sizes. Higher values mean more compression, but at some point you will notice the quality degradation.

Quote for: CRF Guide (Constant Rate Factor in x264, x265 and libvpx)

Extract Part of Video

-ss to seek to a part of the video.
-t for the duration, or -to for the time in the video

The behaviour depends on the position of the -ss parameter. These commands will extract the 10s from 0:30-0:40 in the original video.

ffmpeg -i in.mp4 -ss 30 -t 10 -c copy out.mp4
ffmpeg -i in.mp4 -ss 30 -to 40 -c copy out.mp4

However, if the seek is placed before the file input then timestamps are relative to the seek and these commands now behave differently from each other. With the first output being the same as the original commands, but the second now extracting the 40s long segment from 0:30 to 1:10 in the video.

ffmpeg -ss 30 -i in.mp4 -t 10 -c copy out.mp4
ffmpeg -ss 30 -i in.mp4 -to 40 -c copy out.mp4

Extract Audio From Video

To extract audio from a video you just have to specify an audio file output and it'll ignore the video.

ffmpeg -i input.mkv output.mp3

To extract the raw audio from an MP4 without re-encoding (this depends on what codec the audio data actually is inside the mp4, for most H264 video mp4 files this is AAC audio).

ffmpeg -i input.mp4 -c copy output.aac

Extract Frame Image

Good for quickly getting a thumbnail of the video.

ffmpeg -ss 00:23:00 -i input.mp4 -frames:v 1 output.jpg
ffmpeg -i input.mp4 -ss 00:00:10.5 -vframes 1 output.png

How to Add iOS 13.4 Device Support to Xcode 11.3 on Mojave

2020-04-04T01:03:06.264285+00:00

The latest version of iOS (13.4) was recently released, and with it the new version of Xcode. This new version of Xcode only runs on macOS Catalina. If, like me, you have been avoiding Catalina as much as possible this is a problem, because if you try to use the last version of Xcode that supports Mojave (Xcode 11.3) to Build and Run on a device that is running iOS 13.4 it will give you an error (as it only has support for iOS versions up to iOS 13.3).

However, you can add extra iOS version support to Xcode by taking the DeviceSupport files from one version and adding them to another version (both to add support for newer, and older versions of iOS).

Find or download a version of Xcode that support the iOS version you want, in this case the latest one.
Extract the Xcode.app bundle, navigate to Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport/
Take the folder for the iOS version you need and place it in the equivalent directory inside the version of Xcode you are running.
At this point you'll need to restart Xcode at the minimum, though in my case I had to reboot completely.
Now connect the iOS device running that iOS version and attempt to Build and Run to it.
Xcode should not pull down the symbol files it needs from the device and you are good to go.

This will not give you the new SDK, but should allow you to continue using your existing Xcode for a while longer.

View Remote Server Logs Locally using GoAccess

2020-04-02T03:33:41.875442+00:00

Since I removed Google Analytics I've had 0 visibility into my websites traffic, and I'd really like to at least see some stats like the page views.

My website is a mix of Nginx serving static files and forwarding dynamic pages to a python web server using Flask.
After a bit of searching I realised I could just use Nginx's access logs directly, rather than introducing anything new. GoAccess seemed to be the most recommended tool, but I wanted to avoid installing anything new on the server if I could help it, so I was really looking for a local tool.

Fortunately, it is possible to run GoAccess on your local machine and view the server logs remotely over SSH.

How To

First make sure the server logs are accessible to the user you will be SSH'ing in with, if they aren't then SSH in and fix the permissions first: sudo chmod -R +r /var/log/nginx/.

On macOS installing GoAccess is as simple as running:

brew install goaccess

Then to view a single log file you can ssh into the server, cat the log file and pipe the output directly to GoAccess on your local machine.

ssh USER@IP_ADDRESS 'cat /var/log/nginx/access.log' | goaccess -a --log-format=COMBINED

After a while nginx will compress the log files and start again so you end up with a directory looking like:

access.log        access.log.2.gz   access.log.3.gz
access.log.4.gz   access.log.5.gz   access.log.6.gz

There are now multiple log files, most of which are compressed.

To view all the log files you can run this command to uncompress them, and concatenate them together before feeding them into GoAccess.

ssh USER@IP_ADDRESS 'zcat -f /var/log/nginx/access.log*' | goaccess --log-format=COMBINED

More details about actually using GoAccess to navigate the logs you have opened can be found on GoAccess's website.

Using UIImagePickerController on iOS

2020-04-01T00:52:13.995134+00:00

The simplest way to allow users to select a photo is to use the UIImagePickerController class, which provides a built in UI for users to access their photos.

Doing so is incredibly simple.

Declare that your class conforms to the UIImagePickerControllerDelegate and UINavigationControllerDelegate protocols.
Implement the -imagePickerController:didFinishPickingMediaWithInfo: method.
Create and present an UIImagePickerController view controller class.
Do something with the image returned

- (void)pickPhoto {
    UIImagePickerController *imagePicker = [[UIImagePickerController alloc] init];
    imagePicker.sourceType = UIImagePickerControllerSourceTypePhotoLibrary;
    imagePicker.delegate = self;
    [self presentViewController:imagePicker animated:YES completion:nil];
}
// Implement UIImagePickerControllerDelegate method
-(void)imagePickerController:(UIImagePickerController *)picker didFinishPickingMediaWithInfo:(NSDictionary<UIImagePickerControllerInfoKey,id> *)info{
    UIImage *image = info[UIImagePickerControllerOriginalImage];
    self.imageView.image = image;
    [picker dismissViewControllerAnimated:YES completion:nil];
}

func pickPhoto() {
    let imagePicker = UIImagePickerController()
    imagePicker.sourceType = .photoLibrary
    imagePicker.delegate = self
    present(imagePicker, animated: true)
}
// Implement UIImagePickerControllerDelegate method
public func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey: Any])
    imageView.image = (info[.originalImage] as? UIImage)
    picker.dismiss(animated: true, completion: nil)
}

Footnotes

UIImagePickerControllerSourceType

UIImagePickerControllerSourceTypePhotoLibrary (.photoLibrary) allows users to pick from their full photo library, including browsing their albums.
UIImagePickerControllerSourceTypeSavedPhotosAlbum (.savedPhotosAlbum) limits the selection to just their chronological 'Moments' screen, camera roll.
UIImagePickerControllerSourceTypeCamera (.camera) shows a camera interface so the user can take a picture their and then in the app.

Note: using UIImagePickerControllerSourceTypeCamera (.camera) will crash the app unless the info.plist contains the NSCameraUsageDescription key with the following error message; unlike the other two, which use the act of the user selecting a photo as implied consent to give that photo too the app, and do not prompt for permission since iOS 11, when it was changed to run as a separate process (and so no longer require the NSPhotoLibraryUsageDescription key).

[access] This app has crashed because it attempted to access privacy-sensitive data without a usage description.
The app's Info.plist must contain an NSCameraUsageDescription key with a string value explaining to the user how the app uses this data.

Select Photos, Videos or Both

By default UIImagePickerController will only show and select images from a users photo library. However this is actually controlled via the mediaTypes property, which acts as a filter. To select photos and videos you would set the array to include videos.

@import MobileCoreServices; // where kUTTypeImage and kUTTypeMovie are defined
imagePicker.mediaTypes = @[(NSString*)kUTTypeImage, (NSString*)kUTTypeMovie];

import MobileCoreServices
imagePicker.mediaTypes = [(kUTTypeImage as String), (kUTTypeMovie as String)]

To extract the video you'll need to get the UIImagePickerControllerMediaURL (.mediaURL) key from the returning info dictionary, instead of the UIImagePickerControllerOriginalImage (originalImage) value.

NOTE: The developer documentation for the property includes this interesting comment regarding those Live Photos that you've enabled effects like 'Bounce and Loop' on.
If you want to display a Live Photo rendered as a Loop or a Bounce, you must include the kUTTypeMovie identifier.

Checking Capabilities

The UIImagePickerController also includes several class methods to check availability of different features.

if ([UIImagePickerController isSourceTypeAvailable:UIImagePickerControllerSourceTypeCamera]) {
    // This device has a camera
}
NSArray *mediaTypes = [UIImagePickerController availableMediaTypesForSourceType:UIImagePickerControllerSourceTypeCamera];
// Can this camera capture video, or just Photos (i.e: the first iPhone vs the iPhone 3G)

if UIImagePickerController.isSourceTypeAvailable(.camera) {
    // This device has a camera
}
let mediaTypes:[String]? = UIImagePickerController.availableMediaTypes(for: .camera)
// Can this camera capture video, or just Photos (i.e: the first iPhone vs the iPhone 3G)

Other

There are also more APIs, such as for getting the availability of flash, checking front and back cameras availability, checking the capabilities of each camera separately, and for allowing basic editing of the photo before it is given to your app. Official UIImagePickerController Class Documentation

Limitations

UIImagePickerController is an incredibly useful and convenient class for quickly adding image selection into an app. But it does have limitations:

no multi-selection support
limited filtering options (only photos, or videos, or both)
no direct programmatic access to the photo library (only the photo you are given)
limited camera access or UI customization.

If you want any of those features then you'll need to use AVFoundation to make a custom camera UI, or PhotoKit (which will be the subject of the next blog post) to make a custom image picker.

QRCode Generation and Scanning on iOS 13

2020-03-11T00:06:48.117661+00:00

Generating QRCodes using CoreImage

iOS has built in support for generating several different types of 1D and 2D bar codes.

The API in the CoreImage framework used to generate these codes is CIFilter, which returns a CIImage. We can then turn this into a UIImage to display on screen otherwise use.

The filter is given an NSData object as the input, and a string representing the amount of error correction to be used when making the QRCode (L 7%, M 15%, Q 25%, or H 30%).

let qrFilter = CIFilter(name: "CIQRCodeGenerator")
// Text
qrFilter?.setValue(text.data(using: .isoLatin1), forKey: "inputMessage")
// Error correction
qrFilter?.setValue("M", forKey: "inputCorrectionLevel")
return qrFilter?.outputImage

CIFilter *qrFilter = [CIFilter filterWithName:@"CIQRCodeGenerator"];
// Set the QR Code contents
[qrFilter setValue:[qrString dataUsingEncoding:NSISOLatin1StringEncoding] forKey:@"inputMessage"];
// Error correction
[qrFilter setValue:@"M" forKey:@"inputCorrectionLevel"];
return qrFilter.outputImage;

One small oddity is that, unlike might be expected, the string is not encoded with UTF8, but instead ISO Latin 1 string encoding.

Scaling

The resulting image is a QRCode, but is only as big as it needs to be so when you put it in your image view it will be scaled up and look blurry. So before using it you likely want to scale up the CIImage to the size of the image view that it will be displayed in.

let qrCode:CIImage = createQRCode()
let viewWidth = imageView.bounds.size.width;
let scale = viewWidth/qrCode.extent.size.width;
let scaledImage = qrCode.transformed(by: CGAffineTransform(scaleX: scale, y: scale))
imageView.image = UIImage(ciImage: scaledImage)

CIImage *qrCode = [self createQRForString:contents];
CGFloat viewWidth = self.imageView.bounds.size.width;
CGFloat scale = viewWidth/qrCode.extent.size.width;
CIImage *scaledImage = [qrCode imageByApplyingTransform:CGAffineTransformMakeScale(scale, scale)];
self.imageView.image = [UIImage imageWithCIImage:scaledImage];

Reading QRCodes with AVFoundation Metadata

Reading a QRCode is similarly easy on iOS. When setting up a camera capture session (getting output from the camera) you can also attach various metadata outputs, for things such as faces, certain objects and QRCodes.

After getting a configured AVCaptureSession we can add our metadata output.

let metadataOutput = AVCaptureMetadataOutput()

if (captureSession.canAddOutput(metadataOutput)) {
   captureSession.addOutput(metadataOutput)

   metadataOutput.setMetadataObjectsDelegate(self, queue: DispatchQueue.main)
   metadataOutput.metadataObjectTypes = [.qr]
}

AVCaptureMetadataOutput *metadataOutput = [[AVCaptureMetadataOutput alloc] init];

if ([captureSession canAddOutput:metadataOutput]) {
    [captureSession addOutput:metadataOutput];

    [metadataOutput setMetadataObjectsDelegate:self queue:dispatch_get_main_queue()];
    metadataOutput.metadataObjectTypes = @[AVMetadataObjectTypeQRCode];
}

Then we just need to implement AVCaptureMetadataOutputObjectsDelegate and we have the QRCode's contents

func metadataOutput(_ output: AVCaptureMetadataOutput, didOutput metadataObjects: [AVMetadataObject], from connection: AVCaptureConnection) {
    if let metadataObject = metadataObjects.first {
        guard let readableObject = metadataObject as? AVMetadataMachineReadableCodeObject else { return }
        guard let stringValue = readableObject.stringValue else { return }

        print("QR Code: \(stringValue)")
    }
}

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputMetadataObjects:(NSArray *)metadataObjects fromConnection:(AVCaptureConnection *)connection{
    AVMetadataObject *metadata = [metadataObjects firstObject];
    if ([metadata isKindOfClass:[AVMetadataMachineReadableCodeObject class]]) {
        NSString *stringValue = [(AVMetadataMachineReadableCodeObject *)metadata stringValue];
        NSLog(@"QR Code: %@", stringValue);
    }
}

You can also ask the camera preview layer to translate the detected object into it's local coordinate system, allowing you to overlay something on the camera view that matches the QRCode's position.

let qrCodeObject = previewLayer.transformedMetadataObject(for: readableObject)
showQRCodeBounds(frame: qrCodeObject?.bounds)

QRCode Generator and Scanner Example Project

The example code in this blog post comes from an example project on Github, which I wrote to demonstrate how to generate a QR code image and how to scan a QR code with the devices camera. (There is also a less feature complete ObjC example project)

GeneratorViewController.swift	ReaderViewController.swift

Project Structure

The Xcode project uses the tab based application template, with the application and scene delegates unchanged from the Xcode template.
The 2 main files are GeneratorViewController.swift and ReaderViewController.swift and the code inside is very similar to the included example code in this blog post, just with the extra surrounding code needed to glue it together into a usable application.

iOS Multiple Window Support Without Storyboards

2020-03-10T23:18:48.695121+00:00

If, like me, you prefer doing things programatically, instead of using interface builder, then the Main.Storyboard file doesn't last beyond creating the new project.

However, now with iOS 13 and multiple window there is now a SceneDelegate as well as an AppDelegate to deal with.
Fortunately, it isn't much more complex than before, just moved around slightly.

AppDelegate (iOS 12)

With iOS 12 you would setup an application programatically by deleting the Main.Storyboard and then creating the window in the AppDelegate's application:willFinishLaunchingWithOptions: or application:didFinishLaunchingWithOptions:.

func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
    self.window = UIWindow(frame: UIScreen.main.bounds)
    self.window.rootViewController = ViewController()
    self.window.makeKeyAndVisible()
    return true
}

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
    self.window = [[UIWindow alloc] initWithFrame:[[UIScreen mainScreen] bounds]];
    self.window.rootViewController = [[ViewController alloc] init];
    [self.window makeKeyAndVisible];
    return YES;
}

SceneDelegate (iOS 13)

The info.plist file still has the UIMainStoryboardFile key-value pair, but now it also has a UISceneStoryboardFile key inside the UIApplicationSceneManifest -> UISceneConfigurations array.

Once those 2 are deleted, the Storyboard is no longer involved in creating your apps UI and you need to do that in code yourself.
The location to do so is now in the SceneDelegate's -scene:willConnectToSession:options: method.

func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
    // Use this method to optionally configure and attach the UIWindow `window` to the provided UIWindowScene `scene`.
    guard let windowScene = (scene as? UIWindowScene) else { return }
    
    let window = UIWindow(windowScene: windowScene)
    window.rootViewController = ViewController()
    self.window = window
    window.makeKeyAndVisible()
}

- (void)scene:(UIScene *)scene willConnectToSession:(UISceneSession *)session options:(UISceneConnectionOptions *)connectionOptions {
    self.window = [[UIWindow alloc] initWithWindowScene:(UIWindowScene*)scene];
    self.window.rootViewController = [[ViewController alloc] init];
    [self.window makeKeyAndVisible];
}

The only difference compared to the old -application:willFinishLaunchingWithOptions: code is that we now need to tell the UIWindow which scene it is part of. This can be done using the new UIWindow(windowScene: windowScene) init method, but windows can also be dynamically moved between scenes so there is also a windowScene property on UIWindow instances which can be changed at any time.

Automatically Correct Insta360 Studio Snapshot Dates

2020-02-14T21:23:47.172803+00:00

When you import a 360º photo into the desktop Insta360 Studio software, one of the export options is to take a "normal" photo (called "FreeCapture" in Insta360's software, or "Overcapture" in the GoPro Fusion software) from the original 360º media, saving this as a snapshot.

Unfortunately these snapshots are just that, exported snapshots of the preview area with no camera metadata.
Lack of metadata is especially annoying when sorting by date (as all phones do), as re-framed photos will not appear next to the 360º photos themselves, but next will appear as being from the day you processed it.

Fortunately, the name includes all the date and time information needed.

IMG_20191230_120453_00_105_2020-02-14_17-17-37_screenshot.jpg

Pattern
IMG_|yr|mon|day|_|hr|min|sec|_00_#_[date screenshot saved]_screenshot.jpg

This means it is easy to create a simple script to fix the files after the fact, allowing you to use Insta360 Studio to snapshot lots of different angles of photos and video and then run a script to clean things up afterwards (instead of manually fixing each image file as you go).

Date Correction Script

The script needs to follow these simple instructions:

Find all jpg's with screenshot in the name.
Extract the date from the file name.
Modify the exif date taken data to match.
Save the new metadata.

(Though there are other steps, like merging in some metadata from the original 360 file, which could be taken to improve it.)

The script

The only external library used is one to extract the metadata and modify it, Piexif.

import os
import re
import datetime
import piexif

filepath = os.path.abspath(".")

# Find files with `screenshot` in the name.
files = [f for f in os.listdir(filepath) if "screenshot" in f and f.endswith(".jpg")]
print(files)

for filename in files:
    print(filename)
    # Extract datetime
    date_rex = r"(20\d\d)(\d\d)(\d\d)_(\d\d)(\d\d)(\d\d)"
    date_ranges = re.search(date_rex, filename)
    # Date
    year = date_ranges.group(1)
    month = date_ranges.group(2)
    day = date_ranges.group(3)
    # Time
    hour = date_ranges.group(4)
    minute = date_ranges.group(5)
    second = date_ranges.group(6)
    # date text
    dt_text = f"{year}:{month}:{day} {hour}:{month}:{second}"
    print(dt_text)
    # Image file path
    image_filepath = os.path.join(filepath, filename)
    print(image_filepath)
    exif_dict = piexif.load(image_filepath)
    print(exif_dict) # Before
    exif_dict["0th"][piexif.ImageIFD.DateTime] = dt_text
    exif_dict["Exif"][piexif.ExifIFD.DateTimeOriginal] = dt_text
    print(exif_dict) # After
    # Save new metadata
    exif_bytes = piexif.dump(exif_dict)
    piexif.insert(exif_bytes, image_filepath)

This python script should be safe to re-run multiple times on the same directory (although it could cut down on fileIO by comparing the existing date metadata to see if it is correct before overriding it).

Example output from piexif.load(img) before and after:

Before:
{'0th': {}, 'Exif': {}, 'GPS': {}, 'Interop': {}, '1st': {}, 'thumbnail': None}

After:
{'0th': {306: '2019:12:30 12:04:53'}, 'Exif': {36867: '2019:12:30 12:04:53'}, 'GPS': {}, 'Interop': {}, '1st': {}, 'thumbnail': None}

The script is also saved as a gist on Github.

My New Website

2017-01-15T02:36:14.846980+00:00

Since I started learning programming I've had a few different websites. I've used Weebly, Wordpress and Tumblr. But I'd always wanted to (and had never gotten around to) make my own website.
Now, finally, I have.

TLDR: The website is written in Python using Flask, running on an Ubuntu VPS from DigitalOcean and the blog posts are written in CommonMark (for now).
The front end is hand written HTML and CSS with minimal Javascript (2 main functions).

Visual Design

Before I started on writing any server side code, I created a few example pages to establish the visual style I wanted.

I am not a designer. I am a programmer who likes good UI and wishes he was good at design.

So to compensate, I look around for examples of design I like and save them. I started doing this with designs I particularly like on dribble and other sites.
At the time of writing this, my "UI" folder on my NAS contains 17 sub folders and 1089 images (yes, I should probably just use pinterest. I don't because I want them locally so I can run image/color analysis algorithms on them, if I ever get around to writing any that is...).

UI Prototypes

I'm never sure how to start things. Whether it is a piece of writing, UI design or a website. I struggle to think of how to introduce things. One option I could have used was to simply have my website start on the blog index. However, I wanted multiple different sections to the website. Just as importantly I wanted them to feel like different sections, not simply set apart pages of the blog. The home page was inspired by @chpwn and a few other developers (though the rest have all changed their websites' designs).

For the rest of the site I used the top navigation bar (found on almost every website) and played around until I was happy with the design. Though I ended up tweaking things here and there anyway when I came to implement it properly.

If you are wondering about the example blog post, the title is obviously nonsense but, rather than use Lorem Ipsum for the body text, I used this incredible story about the Graphing Calculator Apple bundled with the original PowerPC computers.

The Final Designs

I added a drop shadow to the navigation bar, vertically centred the home page and went back on my decision to let the content pages (such as the portfolio project pages and blog posts) go full width no matter the screen width. Overall though I kept things mostly the same.

For the projects themselves I didn't actually decide on the design until I started writing the actual site. I originally planned for each of the projects to get a custom design specific to that project. That plan ended when I realised I don't have enough imagination to pull off more than 10 custom designs, each for incredibly similar projects.

In the end I created a template I can customise to different projects and alter slightly for different types of projects. The app pages get the app icon and a link to the AppStore. The other types of projects simply get links to whatever official website the project has.

I spent a lot of time finding personal websites I liked the designs of and then looking at, and comparing, the “about me” pages. In the end what I ended up writing turned into a mini-portfolio page on its own. I’m not completely happy with the design but I’ll keep working on it over time.

I had originally planned to have the blog posts go full width across the browser window. However, I made this decision while using a 13" MacBook Pro as my main computer. In the interim, between deciding that and implementing it, I switched to using a desktop computer with extra wide screen monitor and reconsidered my earlier position.
For overall style I tried to emulate the look of Github’s README files.

A feature I particularly like on Medium, that lots of writing applications have (but very few websites), is the estimated reading time they display right at the top of the page. As soon as I load the article I know roughly how long I need, and if I should set it aside to read later.
If it is within the last week, the published, and modified (I believe both should be shown on blog posts) labels display how long ago that event was, instead of the actual date. However, the actual date and time is always included in the ‘title’ attribute. Hovering over the relative timestamps should prompt the browser to display the real time stamp.

For categorising posts I decided to use tags instead of categories. In my mind categories are like folders. A post can be put inside a single category. Whereas, multiple tags can be attached to a single article. So if I write about releasing a new iOS fitness app I can tag it with: iOS, fitness, press release, etc...
Later I can lookup all the iOS related blog posts I’ve written, or every press release, etc...

Code Design and Implementation

Project Structure

The public facing web server itself is Nginx, running on an Ubuntu VPS from DigitalOcean. Nginx is set up to try to serve files in the following order:

The static resources directory.
The static pages directory.
The cache directory.
Finally it will forward the request to the python web app.

This setup minimises the requests that make it to the python application and allows me to easily cache blog posts, and any other pages I want to be dynamic some of the time.

server {
    listen       80;
    server_name  ikyle.me www.ikyle.me;
    charset      utf-8;

    root /var/www/ikyle;
    index index.html index.htm;

    location / {
        try_files /static_files$uri /static_files$uri/index.html
        /static_pages$uri /static_pages$uri/index.html
        /cache$uri /cache$uri/index.html @blog;
    }

    location @blog {
        include uwsgi_params;
        uwsgi_pass unix:/var/www/ikyle/ikyle_uwsgi.sock;
    }
}

One optimisation I have been thinking about doing is to: move the cache in front of the static files directory; extend the python application to monitor for changes to css and javascript files inside the static directory; and have it automatically run a minifier on those files, placing the output in the cache directory.

I don’t know what the conventions are for structuring Python applications, but this is the file structure I went with.

Root
+-- error_pages/
|   Static error pages (404, etc...) are stored in here
+-- static_files/
|   Static files (css, js, images, etc...) go in here (can be cached)
+-- static_pages/
|   Static pages from the website (such as the about me and portfolio pages) go in here. (not to be cached)
|
+-- cache/
|   Rendered blog posts, etc go in here.
|
|
+-- portfolio/
|     +-- input/
|     |     +-- categories.json
|     |     |   Defines the different categories of portfolio projects, and their order
|     |     *-- 'folder_name'/
|     |         Projects each have their own folder
|     |           +-- media/
|     |           |   This folder contains all the images and videos for the project
|     |           +-- info.json
|     |           |   This file defines everything about the project
|     |           *-- 'file'.md
|     |               Any markdown files will be rendering into HTML files and placed as directories to the project itself
|     |
|     +-- templates/
|     |   The templates for the porfolio pages go in here
|     |
|     `-- renderer.py
|         Processes the input files and generates the resulting HTML pages
|
|
+-- blog.db
|   The SQLite file that acts as the websites database.
+-- app.py
|   Application entry point
+-- api/
|     +-- api.py
|     |   The python api through which the websites posts are accessed and searched
|     `-- utility.py
|         Functions shared across different pages, such as rendering Markdown and caching pages.
`-- web/
      +-- templates/
      |   The html templates that generate the web pages.
      |     +-- blog/
      |         The folder the blog templates are stored
      |           `-- admin/
      |               The folder the admin templates are stored
      |
      +-- webapi.py
      |   The public facing json web api endpoint.
      `-- blog.py
          The public html blog

The top few directories are simply to place HTML, CSS and other static files in. The next part is the portfolio directory. This acts as its own subproject, separate from the rest of the application.

As most of my projects are very similar (predominantly being iOS apps) I decided that I would create a template, like I had set up for the blog posts.
To create a new project page I create a new directory for it. Inside that directory I create an info.json file and fill out the relevant information, such as: category; title; description; description; any press or reviews it has received; relevant images and videos; and links to related web pages.
The rendered.py script takes the information from the info.json files, passes it into the template and saves the result, along with the media, into the static_pages directory.

At the moment, I just run this locally on my computer and sync the resulting files onto the server. Adding a new project to my portfolio isn’t something I do regularly enough for me to have created a web UI for this yet.

Finally, the last chunk is the actual Flask application itself.

The Flask Application

The application is separated into 3 sections: the api; the blog; and the webapi.

The api directory contains a utilities file, with functions I use throughout the application, and the api.py file that handles all database access for blog posts.

As you might have noticed in the file directory structure, I use SQLite for the database. Before you start, yes I know.

Think of SQLite not as a replacement for Oracle but as a replacement for fopen()

However, the reasons I choose to use SQLite are as follows.

My dataset is small.
The dataset is my blog posts. Even after years of active blogging, we are only talking a few thousand text documents.
My expected load is minimal.
As much as I’d like to imagine otherwise, my posts are unlikely to generate very much traffic.
I’m lazy.
Choosing SQLite gave me: a quick and easy setup on the web server; the ability to run the website locally on OS X with minimal fuss; trivial backups; and easy full text search.

Having said that, I am reconsidering the way I handle storing blog posts, which I’ll detail more in the possible changes section at the end of this post.

The api.py file handles opening, closing and querying the database. Posts are returned as a custom Dict subclass that allows you to access keys like attributes.

# Convience Dict Subclass
#  Allows dict["name"] to be queried via dict.name
class DBItem(dict):
    def __getattr__(self, name):
        if name in self:
            return self[name]
        else:
            return None
    def __setattr__(self, name, value):
        self[name] = value
    def __delattr__(self, name):
        if name in self:
            del self[name]

The biggest part of the file is the get_posts() function. Rather than using several different functions to query different things, this takes several parameters, validates they are within allowed ranges, and builds the correct query string. Then it processes the result into an array of post objects.

# Search posts
def get_posts(query=None, sortDescending=True, before=None, after=None, summary=True, limit=25, tags=None):
    ...
# How many results does the search potentially return
def get_posts_count(query=None, before=None, after=None, tags=None):
    ...

This function is delicate enough that, if this was for something other than my personal blog, I would separate it into 3 distinct parts (the validation; query string building and result building) which could have tests run against them. However, setting up and writing tests for this, (and the different URL endpoints) isn’t worth it to me. If there is a bug I know exactly which file handles that part of the website and can deploy a fix within minutes.

The 2 web facing parts of the website are split between webapi.py and blog.py, which are each separate Flask Blueprints.

app.py sets up Flask and adds my custom functions to the jinja2 template engine and registers the webapi and blog blueprints to the relevant url_prefix.

webapi.py is basically a thin JSON wrapper around the blog api.

For example: ikyle.me/blog/api/post?id=1 will return the first blog post’s Dict (which should be this post) dumped as a JSON file (including the content as markdown, rather than HTML).
This is the relevant code (just with some sanity checks removed).

post = api.get_post( request.args.get('id') )
r = flask.make_response( json.dumps(post) );
r.mimetype = 'application/json';
return r

I created the JSON api so that I have easy API access to my blog, in case I want to make an iOS app or something similar. However, the only endpoints that require authentication are the ones to create, modify or delete a blog post. The two needed for browsing are open.

/blog/api/post?id=
/blog/api/search

blog.py is similar to webapi.py but with the extras required for managing a website rather than an API endpoint. It queries the api, renders the posts, handles (most) server errors and generates the blog's Atom feed (yes it has one, I just haven’t added an icon linking to it yet).

The Atom feed is generated by the AtomFeed class from the Werkzeug contrib package.

I also use Flask-Login to manage access to a set of very basic admin features I built into the website. These consist of a basic blog post editor and buttons that appear allowing me to create, edit or delete posts.

As I (for the moment) use unmodified CommonMark, I am able to use the JavaScript reference implementation to locally generate a live preview of the post as I am writing it. This is almost identical to the commonmark.js dingus. One of the extra things I need to do is refresh the syntax highlighting.

var commonmark = window.commonmark;
var writer = new commonmark.HtmlRenderer();
var htmlwriter = new commonmark.HtmlRenderer();
var xmlwriter = new commonmark.XmlRenderer();
var reader = new commonmark.Parser();

function debounce(func, wait, immediate){ /* https://davidwalsh.name/javascript-debounce-function */}

document.addEventListener("DOMContentLoaded", function(event) {
    hljs.configure({languages: []}); // Don't auto detect languages
    var article_text = document.getElementById("article_textarea");
    var preview_holder = document.getElementById("article_preview_div");

    var render_markdown = debounce(function() {
        var parsed = reader.parse(article_text.value);
        var renderedHTML = writer.render(parsed);

        preview_holder.innerHTML = renderedHTML;
        highlight_code(preview_holder);
    }, 250);

    article_text.onchange = function() { render_markdown(); };
    article_text.oninput  = function() { render_markdown(); };
    // Call them once at the start
    render_markdown();
}

function highlight_code(element) {
    var elements = element.getElementsByTagName('pre');
    for (var i = 0; i < elements.length; i++) {
        var pre_block = elements[i];
        var possible_block = pre_block.children[0];
        if (possible_block.tagName == "CODE") {
            hljs.highlightBlock(possible_block);
        }
    }
}

Future Changes

I’m happy I have the site at a point I can push it live. However, this is not how the website will always be. Even now, there are lots of things I want to change.

At the moment, the website isn’t optimised very much. I designed a cache directory into the design from the beginning, but don’t use it at present.

Both the blog posts and atom feed are dynamically generated on every request. An obvious gain would be to cache both of these.

The atom feed could be saved and only regenerated every time a blog post is changed, rather than on every request.

Blog posts have friendly dynamic timestamps, but only for the first week (which is also the time a post is most likely to be edited). Thereafter, they could be cached indefinitely, only needing to be regenerated if modified. Alternatively, the friendly timestamps could be generated locally with javascript, allowing the posts to be cached from the start.

I am currently writing this post using iA Writer on my MacBook, rather than in the post editor I spent the time to make, because I didn’t think to add the ability to save (or even the concept of) drafts.

Using CommonMark has its advantages. I can use any CommonMark compatible renderer unmodified and my blog posts will render correctly.
Despite this, I don’t think my blog will use standard CommonMark for very long.

One missing feature I was shocked to find out isn’t part of either Markdown or CommonMark is footnotes. I was particularly surprised given that John Gruber is the original author of Markdown and his website, daringfireball.net, uses footnotes in almost every post.

This should be a footnote.[^comment]

[^comment]: Sadly this syntax isn't supported for some reason.

I would have used this feature extensively while writing this article. Almost everywhere I have included a comment in (brackets), and some of the comments blocked by commas, would have instead been a footnote. I left out a lot of small related comments I have about different sections of this post because I would have put them in a footnote, and adding them into the main body of the text would divert that section off course too far from its original topic.

If I’m going to give up compatibility with the CommonMark spec I might as well go all the way. Other non-standard extensions I’d like include:

Tables
Checkboxes
The image syntax extended to include videos automatically, based on the file extension.

John Gruber has stated that for things Markdown doesn’t include you should use HTML directly. However, one of my reasons for using Markdown is to avoid using HTML directly.
Keep in mind that making a native iOS app is always faintly at the back of my mind, even if I end up never getting around to it. Including HTML in your markdown might be fine if the markdown is just going to be rendered out to HTML. But, if the result is going to be rendered differently (by iOS’s text rendering engine for example) HMTL is just going to get in the way and cause you issues.

Talking about custom changes to Markdown leads me neatly into the next point. I will, probably, completely change the way blog posts are handled.

At the moment, blog posts are stored in the SQLite database, and blog post’s images are handled... well that’s another feature, like drafts, I didn’t realise I needed to handle until I started writing this blog post. I manually rsync’d the images for this post onto the server.

An article I read a little while ago which really caught my imagination, and that I haven’t seen mentioned much online, was this blog post announcing iA Writer 4.

In it they mentioned a quote from John Gruber about Markdown’s image syntax (which I keep forgetting).

My best idea for good Markdown img syntax would be to just paste in a URL ending in .jpg/.png/.gif etc.

I’ve already mentioned I think the image syntax should be expanded to include .mp4, .mp3, etc... However, iA Labs took the idea a step further. This allows tables, source code and even other bits of text to be separated into external csv, source and text files.

They even have an example spec for file transclusion, as the concept is named.

My revised handling of blog posts would look much more like iA Writer’s handling of its library, or even how I handle the portfolio, than it looks at the moment.

Root
+-- blog/
|     +-- posts/
|     |     *-- "year"/
|     |     |   Each year with its own directory.
|     |     |     *-- "post_title"/
|     |     |     |   Each post has its own directory
|     |     |     +-- info.json
|     |     |     |   This file defines the title, description, author, tags, and publish/modification date
|     |     |     +-- post.md
|     |     |     |   This would be the custom CommonMark for the post itself.
|     |     |     `-- resources/
|     |     |     |   This directory would be contain the images, text files and other files referenced in the post.md file.

I considered including the metadata at the top of the post.md file, like some websites do, but I prefer a slightly more structured layout.

Ideally, the python application would monitor the posts directory for changes and delete the cached HTML for the relevant blog post when needed. Then, after a short delay it’d re-render the post’s cached HTML file.

I would still use the SQLite database. However, instead of being the primary location for posts to be stored it would mirror the tags, metadata and the full text of blog posts. Rather than loading in the markdown directly, I’d parse the markdown and extract just the text (in much the same way I currently generate the post’s word count) loading that into the database, to make use of the SQLite full text search.

Other items I have notes to add in the future include:

Image uploading from my post editor
A better admin page
A web based portfolio project creation page/editor

Summary

I’m never completely satisfied with anything I make, there’s always another feature I could add, a better way to do something, a more efficient way to achieve what I already have.

However, I am proud of this website.

I have done other websites before, but they all used existing templates, a ready made CMS or were just a single static HTML page. This was my first real server side project, and one of my first attempts at doing the design for a larger project myself.