Reading CSV Files of Unknown Encoding into a Pandas DataFrame

(A) Data Collection
library - charset_normalizer
library - pandas
Using charset_normalizer Python library to find out the encoding model of CSV files, to load them into a workspace pandas DataFrames.
Author

Mai Tanaka

Published

July 14, 2026

Illustration of figure holding the decoder key that is able to open a file of unknown encoding.

Illustration of figure holding the decoder key that is able to open a file of unknown encoding.

Introduction

Japanese data files with a bit of history are notorious for having outdated or rarely used encodings. I encountered such a case when generating interactive maps of all the weather stations in Japan. I kept on getting UnicodeDecodeError messages when trying to open a CSV file, which told me that the file was not using the most popular encoding of utf-8.

This short tutorial outlines the steps I took to find out the encoding of the mystery file using the charset_normalizer library. If you are also struggling to open a file with unknown encoding, the steps below should be able to help you.

What You’ll Learn in this Tutorial

By the end of this short tutorial, you’ll learn how to:

  • Open a mystery CSV file as raw binary data using 'rb' designation on the open() function
  • Detect the encoding of the mystery CSV file by using charset_normalizer.detect() on the raw binary data
  • Use the detected encoding to read the mystery CSV file as a pandas DataFrame

If you prefer to skip the explanations and jump straight to the implementation, you can download the code from my GitHub repository.

Here is the list of things you’ll need to run the code.

Prerequisites

  • A copy of either the reading-files-unknown-encoding.ipynb Jupyter notebook or the reading-files-unknown-encoding.py Python script from my GitHub repository
  • data/ subfolder containing ame_master_20260324.csv file (a file with unknown encoding)
  • Python libraries
    • charset_normalizer
    • pandas

Jargon

Raw binary data: Information written in the most basic language of computers, basically comprising solely of 0’s and 1’s. Unreadable to the human reader.

Encoding: The process of using a model to convert a collection of characters into numeric values, representable with 0’s and 1’s. The encoded data cannot be read by a human or a computer program, given that the computer program does not know which encoding model was used. The most commonly used encoding model is 'utf-8'.

Decoding: The process of applying the appropriate encoding model to a file. The decoded file can be read by humans, given that the they know the language it was originally written in.

Find Out the Encoding Used

To read a mystery CSV file into a pandas DataFrame, we will need to:

  1. Open the mystery CSV file as raw binary data
  2. Detect the encoding model used by the mystery CSV file
  3. Use the detected encoding to read the mystery CSV file as a pandas DataFrame

In this blog post, we’ll be using the charset_normalizer library to detect the type of encoding used from the binary file (step 2). It will save you from the time and frustration of manually testing every possible encoding model until you get a hit.

So let’s go ahead and import charset_normalizer library for detecting the encoding, as well as the pandas library for reading the table in the CSV file.

# Import libraries for detecting encoding and reading the table
import charset_normalizer 
import pandas as pd

I’ll detect the encoding used in the CSV file called 'ame_master_20260324.csv'. This file contains tabular data on all active weather stations in Japan and was downloaded as a ZIP file from the Japan Meteorological Agency (JMA) website.

Let’s go ahead and implement the first two steps:

  1. Read the file as raw binary data using open(), with "rb" to designate read binary
  2. Detect the encoding of the binary data with charset_normalizer.detect() function
# File path to the CSV file
fileName = './data/ame_master_20260324.csv'

# Step 1: Read file as raw binary data
with open(fileName, "rb") as f:
    raw_data = f.read()

# Step 2: Use charset_normalizer to detect the encoding
detected_data = charset_normalizer.detect(raw_data)
detected_encoding = detected_data["encoding"]

# print out the detected encoding
if detected_data['encoding'] is not None:
    print('Detected encoding: ', detected_encoding)
Detected encoding:  CP932

Thus, the encoding paradigm of the 'ame_master_20260324.csv' CSV file is CP932, an extension of the shift_jis encoding.

Once the encoding is known, we can move onto the third and final step:

  1. Read the CSV file as a pandas DataFrame using pd.read_csv() with the detected encoding
# Read the CSV file using the detected encoding
amedas_df = pd.read_csv(fileName, encoding=detected_encoding)

We’ve successfully read the CSV file without a UnicodeDecodeError message. We can also confirm that the data is appropriately read into the workspace as a pandas DataFrame with the df.head() method.

amedas_df.head()
都府県振興局 観測所番号 種類 観測所名 ｶﾀｶﾅ名 気象情報等に表記する名称 所在地 緯度(度) 緯度(分) 経度(度) 経度(分) 海面上の高さ(ｍ) 風速計の高さ(ｍ) 温度計の高さ(ｍ) 観測開始年月日 備考1 備考2
0 宗谷 11001 四 宗谷岬 ｿｳﾔﾐｻｷ 稚内市宗谷岬 稚内市宗谷岬 45 31.2 141 56.1 26 10.1 3.1 昭53.10.30 － －
1 宗谷 11016 官 稚内 ﾜｯｶﾅｲ 稚内市開運 稚内市開運　稚内地方気象台 45 24.9 141 40.7 3 24.1 － #昭50.4.1 11903 －
2 宗谷 11046 四 礼文 ﾚﾌﾞﾝ 礼文町香深 礼文郡礼文町大字香深村トンナイ 45 18.3 141 2.7 65 9.9 3.1 平15.10.17 － －
3 宗谷 11061 官 声問 ｺｴﾄｲ 稚内空港 稚内市大字声問村字声問　稚内航空気象観測所 45 24.2 141 48.1 8 10 － 平15.1.1 11904 日照・湿度・気圧を除く
4 宗谷 11076 四 浜鬼志別 ﾊﾏｵﾆｼﾍﾞﾂ 猿払村浜鬼志別 宗谷郡猿払村浜鬼志別 45 20.1 142 10.2 13 9.9 2.7 昭53.10.30 11900 －

Much of the column headers and data elements are written in Japanese, so you’ll have to take my word that it’s working properly.

Summary

That’s it!

You now know how to open a file with unknown encoding using the charset_normalizer library. A quick summary of the steps we took:

  1. Opened a CSV file with unknown encoding as raw binary data with the open() function, using the 'rb' designation
  2. Detected the encoding used in the CSV file by applying charset_normalizer.detect() on the raw binary data
  3. Applied the detected encoding to read the mystery CSV file as a pandas DataFrame

For more tips, tricks and tutorials, be sure to check out the blog posts in the Further Readings.

Further Readings