python 38 lines · 8 steps

Reading text files of unknown encoding in Python

A function that reads bytes and decodes them by checking BOMs, guessing with chardet, then trying a ranked list of fallbacks.

Explained by highlit
Intermediate encoding byte-order-mark fallback file-io robustness
1from pathlib import Path
2 
3import chardet
4 
5 
6def read_text_safely(path: str | Path, *, sample_size: int = 65536) -> str:
7 path = Path(path)
8 raw = path.read_bytes()
9 
10 if not raw:
11 return ""
12 
13 if raw.startswith(b"\xef\xbb\xbf"):
14 return raw.decode("utf-8-sig")
15 if raw.startswith((b"\xff\xfe", b"\xfe\xff")):
16 return raw.decode("utf-16")
17 
18 detection = chardet.detect(raw[:sample_size])
19 encoding = detection.get("encoding")
20 confidence = detection.get("confidence") or 0.0
21 
22 candidates: list[str] = []
23 if encoding and confidence >= 0.5:
24 candidates.append(encoding)
25 candidates.extend(["utf-8", "cp1252", "latin-1"])
26 
27 seen: set[str] = set()
28 for candidate in candidates:
29 key = candidate.lower()
30 if key in seen:
31 continue
32 seen.add(key)
33 try:
34 return raw.decode(candidate)
35 except (UnicodeDecodeError, LookupError):
36 continue
37 
38 return raw.decode("latin-1", errors="replace")
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A byte-order mark is the most reliable signal of encoding, so check it before guessing.
  2. 2Layering detection with an ordered list of fallbacks gives you a decode that almost never throws.
  3. 3Deduplicating candidates avoids re-attempting an encoding the detector already suggested.

Related explainers

python 
from flask import Blueprint, jsonify, request, abort
 
v1 = Blueprint("users_v1", __name__)
v2 = Blueprint("users_v2", __name__)

Versioning a Flask API with Blueprints

api versioning blueprints rest
Intermediate 7 steps
python 
import time
import threading
from enum import Enum
from functools import wraps

Building a circuit breaker in Python

circuit-breaker resilience decorators
Advanced 7 steps
python 
from django.contrib.postgres.search import SearchQuery, SearchRank, SearchVector
from django.core.cache import cache
from django.db.models import F
from django.http import JsonResponse

Ranked full-text search in Django

full-text-search debouncing caching
Advanced 9 steps
python 
import os
from datetime import timedelta
 
 

Class-based config in a Flask app factory

configuration app-factory inheritance
Intermediate 7 steps
python 
import time
 
from flask import Flask, g, request
 

Timing requests with Flask hooks

middleware request-lifecycle instrumentation
Intermediate 5 steps
rust 
use std::collections::HashMap;
use std::fs::File;
use std::io::{self, BufRead, BufReader};
use std::path::Path;

Parsing INI files in Rust

parsing file-io hashmap
Intermediate 9 steps

Share this explainer

Here's the card — post it anywhere.

Reading text files of unknown encoding in Python — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code