ruby 43 lines · 8 steps

Normalizing text files to clean UTF-8 in Ruby

A class that detects a file's encoding, decodes it safely, and rewrites it as valid UTF-8.

Explained by highlit
Intermediate encoding text-processing file-io charset-detection
1require "charlock_holmes"
2 
3class TextFileNormalizer
4 DEFAULT_CONFIDENCE = 60
5 
6 def initialize(path, fallback: "ISO-8859-1")
7 @path = path
8 @fallback = fallback
9 end
10 
11 def normalize!
12 utf8 = to_utf8
13 File.write(@path, utf8)
14 utf8
15 end
16 
17 def to_utf8
18 raw = File.binread(@path)
19 encoding = detect(raw)
20 
21 decoded = raw.dup.force_encoding(encoding)
22 unless decoded.valid_encoding?
23 decoded = raw.dup.force_encoding(@fallback)
24 end
25 
26 decoded
27 .encode("UTF-8", invalid: :replace, undef: :replace, replace: "\uFFFD")
28 .delete("\uFEFF")
29 end
30 
31 private
32 
33 def detect(raw)
34 result = CharlockHolmes::EncodingDetector.detect(raw)
35 return @fallback unless result
36 
37 if result[:confidence].to_i >= DEFAULT_CONFIDENCE
38 result[:encoding]
39 else
40 @fallback
41 end
42 end
43end
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Statistical charset detection is a guess, so gate it behind a confidence threshold and keep a fallback.
  2. 2force_encoding only relabels bytes, so validate with valid_encoding? before trusting the result.
  3. 3encode with invalid/undef :replace guarantees output that is always valid UTF-8 even from messy input.

Related explainers

ruby 
require 'date'
require 'set'
 
class BusinessDayCalculator

Counting business days in Ruby

dates sets ranges
Intermediate 8 steps
ruby 
class Document < ApplicationRecord
  class StaleObjectError < StandardError
    def initialize(id)
      super("Document ##{id} was modified by another process")

Optimistic locking with retries in Rails

optimistic-locking concurrency transactions
Advanced 8 steps
ruby 
class BackfillUsersFullName < ActiveRecord::Migration[7.1]
  disable_ddl_transaction!
 
  BATCH_SIZE = 5_000

How to backfill a column safely in Rails

migrations batching backfill
Intermediate 7 steps
ruby 
class ProjectsController < ApplicationController
  def new
    @project = current_account.projects.build
    load_form_collections

The new/create pattern in a Rails controller

mvc strong-parameters tenant-scoping
Intermediate 7 steps
ruby 
class Membership < ApplicationRecord
  belongs_to :organization
  belongs_to :user
 

Enforcing scoped uniqueness in Rails

validations uniqueness callbacks
Intermediate 8 steps
python 
import heapq
from collections import Counter
from typing import Iterable, Hashable
 

Finding the top-N items in a stream

heaps counting generators
Intermediate 5 steps

Share this explainer

Here's the card — post it anywhere.

Normalizing text files to clean UTF-8 in Ruby — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code