ruby 43 lines · 8 steps

Normalizing text files to clean UTF-8 in Ruby

A class that detects a file's encoding, decodes it safely, and rewrites it as valid UTF-8.

Explained by highlit
Intermediate encoding text-processing file-io charset-detection
1require "charlock_holmes"
2 
3class TextFileNormalizer
4 DEFAULT_CONFIDENCE = 60
5 
6 def initialize(path, fallback: "ISO-8859-1")
7 @path = path
8 @fallback = fallback
9 end
10 
11 def normalize!
12 utf8 = to_utf8
13 File.write(@path, utf8)
14 utf8
15 end
16 
17 def to_utf8
18 raw = File.binread(@path)
19 encoding = detect(raw)
20 
21 decoded = raw.dup.force_encoding(encoding)
22 unless decoded.valid_encoding?
23 decoded = raw.dup.force_encoding(@fallback)
24 end
25 
26 decoded
27 .encode("UTF-8", invalid: :replace, undef: :replace, replace: "\uFFFD")
28 .delete("\uFEFF")
29 end
30 
31 private
32 
33 def detect(raw)
34 result = CharlockHolmes::EncodingDetector.detect(raw)
35 return @fallback unless result
36 
37 if result[:confidence].to_i >= DEFAULT_CONFIDENCE
38 result[:encoding]
39 else
40 @fallback
41 end
42 end
43end
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Statistical charset detection is a guess, so gate it behind a confidence threshold and keep a fallback.
  2. 2force_encoding only relabels bytes, so validate with valid_encoding? before trusting the result.
  3. 3encode with invalid/undef :replace guarantees output that is always valid UTF-8 even from messy input.

Related explainers

ruby 
require "shellwords"
require "open3"
 
module Backup

Building safe shell commands in Ruby

shell-out subprocess command-injection
Intermediate 7 steps
ruby 
class UserAgentParser
  BROWSERS = [
    [/Edg\/([\d.]+)/, "Edge"],
    [/OPR\/([\d.]+)/, "Opera"],

Parsing user-agent strings in Ruby

regex pattern-matching lookup-tables
Intermediate 8 steps
ruby 
class LogAggregator
  BUCKET_FORMAT = "%Y-%m-%dT%H:%M"
 
  def initialize(entries)

Bucketing log entries by the minute in Ruby

aggregation hashing enumerable
Intermediate 5 steps
ruby 
class WeeklySignupsReport
  DEFAULT_WEEKS = 12
 
  def initialize(weeks: DEFAULT_WEEKS, source: User.all)

Building a weekly signups report in Rails

service object aggregation group by
Intermediate 7 steps
ruby 
class ApplicationController < ActionController::Base
  EXPERIMENTS = {
    checkout_button_color: %w[control blue green],
    onboarding_flow: %w[control streamlined]

How A/B test cohorts are assigned in Rails

a-b-testing cookies hashing
Intermediate 8 steps
ruby 
class SnippetHighlighter
  CONTEXT_RADIUS = 60
  MAX_TERMS = 8
 

Building search-result snippets in Ruby

regular-expressions text-processing search
Intermediate 9 steps

Share this explainer

Here's the card — post it anywhere.

Normalizing text files to clean UTF-8 in Ruby — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code