java 59 lines · 8 steps

Detecting text encoding from raw bytes in Java

A byte-level detector that checks for a BOM first, then validates UTF-8 structure to distinguish UTF-8, ASCII, and unknown data.

Explained by highlit
Intermediate byte-manipulation encoding-detection bitwise-operations validation
1public final class EncodingDetector {
2 
3 public enum Encoding {
4 UTF_8, UTF_16LE, UTF_16BE, UTF_32LE, UTF_32BE, ASCII, UNKNOWN
5 }
6 
7 public static Encoding detect(byte[] data) {
8 Encoding fromBom = detectBom(data);
9 if (fromBom != Encoding.UNKNOWN) {
10 return fromBom;
11 }
12 if (looksLikeUtf8(data)) {
13 return isPureAscii(data) ? Encoding.ASCII : Encoding.UTF_8;
14 }
15 return Encoding.UNKNOWN;
16 }
17 
18 private static Encoding detectBom(byte[] d) {
19 if (startsWith(d, 0x00, 0x00, 0xFE, 0xFF)) return Encoding.UTF_32BE;
20 if (startsWith(d, 0xFF, 0xFE, 0x00, 0x00)) return Encoding.UTF_32LE;
21 if (startsWith(d, 0xEF, 0xBB, 0xBF)) return Encoding.UTF_8;
22 if (startsWith(d, 0xFE, 0xFF)) return Encoding.UTF_16BE;
23 if (startsWith(d, 0xFF, 0xFE)) return Encoding.UTF_16LE;
24 return Encoding.UNKNOWN;
25 }
26 
27 private static boolean startsWith(byte[] d, int... prefix) {
28 if (d.length < prefix.length) return false;
29 for (int i = 0; i < prefix.length; i++) {
30 if ((d[i] & 0xFF) != prefix[i]) return false;
31 }
32 return true;
33 }
34 
35 private static boolean looksLikeUtf8(byte[] d) {
36 int i = 0;
37 while (i < d.length) {
38 int b = d[i] & 0xFF;
39 int trailing;
40 if (b <= 0x7F) trailing = 0;
41 else if ((b & 0xE0) == 0xC0) trailing = 1;
42 else if ((b & 0xF0) == 0xE0) trailing = 2;
43 else if ((b & 0xF8) == 0xF0) trailing = 3;
44 else return false;
45 for (int j = 0; j < trailing; j++) {
46 if (++i >= d.length || (d[i] & 0xC0) != 0x80) return false;
47 }
48 i++;
49 }
50 return true;
51 }
52 
53 private static boolean isPureAscii(byte[] d) {
54 for (byte b : d) {
55 if ((b & 0x80) != 0) return false;
56 }
57 return true;
58 }
59}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A byte-order mark is the most reliable signal, so check it before falling back to heuristics.
  2. 2UTF-8's self-synchronizing structure lets you validate a stream by counting the trailing bytes each leader byte promises.
  3. 3Masking bytes with 0xFF avoids Java's signed-byte sign extension when comparing against unsigned values.

Related explainers

php 
<?php
 
namespace App\Services\Checkout;
 

Validating coupons with Laravel's Pipeline

pipeline chain of responsibility transactions
Intermediate 7 steps
java 
@Component
@Converter
public class EncryptedStringConverter implements AttributeConverter<String, String> {
 

Transparent column encryption in Spring & JPA

encryption aes-gcm jpa-converter
Advanced 10 steps
php 
<?php
 
namespace App\Services;
 

How a password strength validator works in PHP

validation regular-expressions data-driven
Intermediate 8 steps
java 
package com.acme.billing.config;
 
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.ConfigurationProperties;

Feature-flagged beans with Spring @ConditionalOnProperty

feature-flags conditional-beans strategy-pattern
Intermediate 5 steps
rust 
use chrono::{Duration, NaiveDate};
 
#[derive(Debug)]
pub struct DateRange {

Parsing and iterating date ranges in Rust

error-handling iterators parsing
Intermediate 7 steps
java 
public static Map<String, String> parseCookieHeader(String header) {
    Map<String, String> cookies = new LinkedHashMap<>();
    if (header == null || header.isBlank()) {
        return cookies;

Parsing an HTTP Cookie header in Java

string-parsing http url-decoding
Intermediate 6 steps

Share this explainer

Here's the card — post it anywhere.

Detecting text encoding from raw bytes in Java — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code