Transliteration of Japanese Hiragana and Katakana Using the Roman Alphabet

Tutorial on how to convert or transliterate Japanese words written in hiragana or katakana into the Roman alphabet using jaconv or pykakasi.

A machine is fed with Japanese text and generates a polished output.

1 Introduction

The Japanese language has its own alphabet.

Actually, you can consider it to have three alphabets. Hiragana has 46 basic letters and generally have a rounded or curved look (Figure 1, purple letters on the right). Each hiragana letter has a katanaka equivalent, which tends to have a jagged or edgy aesthetic (Figure 1, black letters on the left). Aside from these two sets of alphabets, there are the famous kanji characters. There are a couple thousands of these.

In this blog post, we’ll only cover the hiragana and katanaka letters. I’ll show you how to convert or transliterate Japanese words with the Roman alphabet using the jaconv and pykakasi libraries.

Figure 1: Japanese letters written in hiragana (right) and their katakana equivalent (left). Credit: Ian Remsen, CC0, via Wikimedia Commons

1.1 What You’ll Learn in This Tutorial

By the end of this tutorial, you’ll learn how to:

• Transliterate hiragana and katanaka into the Roman alphabet using the jaconv library
• Transliterate hiragana and katakana into the Roman alphabet using the pykakasi library
• Deal with the unique case of half-width katakana

If you prefer to skip the explanations and jump straight to the implementation, you can download the code from my GitHub repository.

Here is the list of things you’ll need to run the code.

1.2 Prerequisites

• A copy of either the transliterate-japanese-romaji.ipynb Jupyter notebook or transliterate-japanese-romaji.py Python script from my GitHub repository
• Python libraries
  • jaconv
  • pykakasi

1.3 Jargon

Romaji: The representation of Japanese words using the English or Roman alphabet.

Romanization: The process of converting Japanese words written in Japanese into its phonetic equivalent, represented with Roman letters.

Transliteration: The general process of spelling out words in one language using the alphabet of another language. The aim is to represent the foreign words based solely on their phonetics, or how they are pronounced.

2 Transliteration

Pronunciation of Japanese hiragana and katakana letters is quite straight forward. In the English language, the same letters in a different context can be pronounced differently (think of “p” and “ch” in “psychology” compared to “pizza” or “change”). On the other hand, each Japanese letter is, on the most part, attributed one pronunciation only.

This allows for transliteration of Japanese words into English with ease. In other words, spelling out the Japanese word based solely on how it sounds, using the Roman alphabet.

For example, the top letters in Figure 1 are both pronounced as “ka”, while the bottom letters are “na”. They are not assigned any other phonetic sound. Thus, if we were to transliterate the Japanese word in the figure, it would be “kana”.

When Japanese words are written using the Roman or English alphabet (such as “kana” above), they are considered to be written in “romaji (ローマ字)”, which literally translates to “Roman letters”. Romaji is the most common way that Japanese is typed using a computer keyboard.

While there are several Python libraries that transliterate Japanese words into the Roman alphabet, I’ll use two libraries in this blog post: the jaconv library and the pykakasi library.

# Import the two libraries to transliterate Japanese into the Roman alphabet
import jaconv
import pykakasi

The two libraries differ in:

• How the user calls methods to convert Japanese letters into Roman letters
• The Japanese alphabet it can deal with

Let’s go ahead and try transliterating the hiragana alphabet into English.

2.1 Romanizing Hiragana with jaconv and pykakasi

Let’s try converting a couple Japanese words, written in hiragana, into its romanized versions. I’ve picked Japanese words that are relatively well known in English.

# Collection of hiragana text [Tokyo, Osaka, Pokemon, sushi, ramen]
hiragana_text = ["とうきょう", "おおさか", "ぽけもん", "すし", "らーめん"]

To transliterate hiragana using the jaconv library, we use the jaconv.kana2alphabet() method.

# Simple case of transliterating hiragana using `jaconv`
jaconv_hiragana = jaconv.kana2alphabet(hiragana_text[0])
print(f'`jaconv.kana2alphabet()` converts {hiragana_text[0]} to {jaconv_hiragana}')

`jaconv.kana2alphabet()` converts とうきょう to toukyou

You will notice that the spelling of the output is not what we are accustomed to. Usually, the city of Tokyo is spelled without the u’s after the o’s. However, when transliterating, we generally spell out each Japanese letter used. As such, “とうきょう (Tokyo)” is transliterated to “と (to)”, “う (u)”, “きょ (kyo)”, “う (u)”.

With the pykakasi library, the conversion, regardless of the alphabet used, is all grouped into the .convert() method inside the pykakasi.kakasi() object.

# Make a pykakasi.kakasi 
kks = pykakasi.kakasi()

# Convert the hiragana text
kks_result = kks.convert(hiragana_text[0])

# Print out result
for item in kks_result:
    for key, value in item.items():
        print(f'{key:<10}{value}')

orig      とうきょう
hira      とうきょう
kana      トウキョウ
hepburn   toukyou
kunrei    toukyou
passport  tokyou

The pykakasi library automatically detects whether the Japanese text is written in hiragana or katakana, and does not require any arguments other than the text to be converted when calling the .convert() method. Additionally, it outputs a list of dictionaries with three romaji versions of the Japanese word:

• hepburn
• kunrei
• passport

These are different standards, set by different authorities, in transcribing Japanese text into English letters.

Let’s see how jaconv and the three romanization standards in pykakasi convert the full list of hiragana words:

# Make a pykakasi.kakasi 
kks = pykakasi.kakasi()

# Work through each text item
for hiragana_now in hiragana_text:

    # Starting hiragana
    print(hiragana_now, " converts to")

    # Convert using jaconv
    jaconv_hiragana = jaconv.kana2alphabet(hiragana_now)
    print(f'\t {jaconv_hiragana:<14} jaconv')

    # Convert using pykakasi
    kks_result = kks.convert(hiragana_now)[0]
    print(f'\t {kks_result['hepburn']:<14} pykakasi - hepburn')
    print(f'\t {kks_result['kunrei']:<14} pykakasi - kunrei')
    print(f'\t {kks_result['passport']:<14} pykakasi - passport')

とうきょう  converts to
     toukyou        jaconv
     toukyou        pykakasi - hepburn
     toukyou        pykakasi - kunrei
     tokyou         pykakasi - passport
おおさか  converts to
     oosaka         jaconv
     oosaka         pykakasi - hepburn
     oosaka         pykakasi - kunrei
     osaka          pykakasi - passport
ぽけもん  converts to
     pokemon        jaconv
     pokemon        pykakasi - hepburn
     pokemon        pykakasi - kunrei
     pokemon        pykakasi - passport
すし  converts to
     sushi          jaconv
     sushi          pykakasi - hepburn
     susi           pykakasi - kunrei
     sushi          pykakasi - passport
らーめん  converts to
     ra-men         jaconv
     raamen         pykakasi - hepburn
     raamen         pykakasi - kunrei
     raamen         pykakasi - passport

We see the following differences:

• The passport convention in pykakasi library tends to abbreviate the elongated sounds in Tokyo and Osaka while the other libraries tend to transliterate
• The kunrei standard in pykakasi transcribes the “shi” in “sushi” as “si”
• The elongation bar in “らーめん” is retained in jaconv but transcribed as the last known vowel with pykakasi

Which convention you use comes down to personal preference.

2.2 Romanizing Katakana with jaconv and pykakasi

The process of transliterating words written in katakana is similar to the transliteration of hiragana words. With the pykakasi library, the method calls are identical to hiragana: the generation of the pykakasi.kakasi() object and the subsequent method call to .convert(). With jaconv library, the .kata2alphabet() method is called in place of .kana2alphabet().

Let’s go ahead and convert the same list of five Japanese words, now written in katakana, into their romaji equivalents with the two libraries.

# Collection of katakana text
fw_katakana_text = ["トウキョウ", "オオサカ", "ポケモン", "スシ", "ラーメン"]

# Make a pykakasi.kakasi 
kks = pykakasi.kakasi()

# Work through each text item
for katakana_now in fw_katakana_text:

    # Starting katakana
    print(katakana_now, " converts to")

    # Convert using jaconv
    jaconv_hiragana = jaconv.kata2alphabet(katakana_now)
    print(f'\t {jaconv_hiragana:<14} jaconv')

    # Convert using pykakasi
    kks_result = kks.convert(katakana_now)[0]
    print(f'\t {kks_result['hepburn']:<14} pykakasi - hepburn')
    print(f'\t {kks_result['kunrei']:<14} pykakasi - kunrei')
    print(f'\t {kks_result['passport']:<14} pykakasi - passport')
    

トウキョウ  converts to
     toukyou        jaconv
     toukyou        pykakasi - hepburn
     toukyou        pykakasi - kunrei
     tokyou         pykakasi - passport
オオサカ  converts to
     oosaka         jaconv
     oosaka         pykakasi - hepburn
     oosaka         pykakasi - kunrei
     osaka          pykakasi - passport
ポケモン  converts to
     pokemon        jaconv
     pokemon        pykakasi - hepburn
     pokemon        pykakasi - kunrei
     pokemon        pykakasi - passport
スシ  converts to
     sushi          jaconv
     sushi          pykakasi - hepburn
     susi           pykakasi - kunrei
     sushi          pykakasi - passport
ラーメン  converts to
     ra-men         jaconv
     raamen         pykakasi - hepburn
     raamen         pykakasi - kunrei
     raamen         pykakasi - passport

The transliterated outcome is the same as the hiragana version.

2.3 Romanizing Half-Width Katakana with jaconv

In general, when transliterating Japanese hiragana or katakana words, a simple call of the jaconv methods, or the pykakasi methods will suffice.

The pykakasi library is an attractive choice when the user does not know whether the input Japanese words are written in hiragana or katakana. It is also quite powerful as it can deal with Japanese strings with a mix of hiragana, katakana and kanji. It simply outputs each component as an element in the list.

However, the jaconv library has one advantage over the pykakasi library. Namely, it can deal with half-width katakana.

Sometimes, Japanese data contains very compact letters. For example, “ﾄｳｷｮｳ” instead of “トウキョウ”. The former are “半角 (hankaku)” letters, or “half-width”, while the latter are “全角 (zenkaku)” letters, or “full-width”. To the human eye, the half-width katakana letters looks more or less the same as the full-width letters. However, to a computer, they have completely different encodings.

2.3.1 A Bit of History of the Half-Width and Full-Width Letters

The difference comes from the history of the computer. In the early days, computers read code and strings a byte (8 bits) at a time. This gave room for \(2^8\) or 256 letters in one byte, which was more than accommodating for upper and lower cases of the 26 letters of the Roman alphabet, numbers and some special letters.

However, the computer architecture became cramped for space when the Japanese alphabet needed to be added to the existing letter set. With close to 50 letters in each of the Japanese hiragana and katanakana alphabets, the early programmers opted to add on just one set of the katakana alphabet: the half-width katakana.

Later on, when people wanted to type using hiragana, katakana and kanji, 8 bits was not enough. As such, the Japanese alphabet started using their own encodings that used 2 bytes (16 bits) or more. This later version is the zenkaku or full-width letters that are more common today.

2.3.2 Issues of Transliterating Half-Width Katakana

Because hankakau or half-width katakana have a different encoding from the full-width versions, we encounter issues when trying to transliterate. Below, I’ve provided the same five Japanese words written in half-width styles, and try to transliterate them with jaconv and pykakasi.

# Collection of katakana text
hw_katakana_text = ["ﾄｳｷｮｳ", "ｷｮｳﾄ", "ｵｵｻｶ", "ﾎﾟｹﾓﾝ", "ｽｼ", "ﾗｰﾒﾝ"] 

# Make a pykakasi.kakasi 
kks = pykakasi.kakasi()

# Work through each text item
for katakana_now in hw_katakana_text:

    # Starting katakana
    print(katakana_now, " converts to")

    # Convert using jaconv
    jaconv_hiragana = jaconv.kata2alphabet(katakana_now)
    print(f'\t {jaconv_hiragana:<14} jaconv')

    # Convert using pykakasi
    kks_result = kks.convert(katakana_now)
    for item in kks_result:
        print(f'\t {item['hepburn']:<14} pykakasi - hepburn')
    for item in kks_result:
        print(f'\t {item['kunrei']:<14} pykakasi - kunrei')
    for item in kks_result:
        print(f'\t {item['passport']:<14} pykakasi - passport')
    

ﾄｳｷｮｳ  converts to
     ﾄｳｷｮｳ          jaconv
     toukyou        pykakasi - hepburn
     toukyou        pykakasi - kunrei
     tokyou         pykakasi - passport
ｷｮｳﾄ  converts to
     ｷｮｳﾄ           jaconv
     kyouto         pykakasi - hepburn
     kyouto         pykakasi - kunrei
     kyouto         pykakasi - passport
ｵｵｻｶ  converts to
     ｵｵｻｶ           jaconv
     oosaka         pykakasi - hepburn
     oosaka         pykakasi - kunrei
     osaka          pykakasi - passport
ﾎﾟｹﾓﾝ  converts to
     ﾎﾟｹﾓﾝ          jaconv
     ho             pykakasi - hepburn
     ho             pykakasi - hepburn
     kemon          pykakasi - hepburn
     ho             pykakasi - kunrei
     ho             pykakasi - kunrei
     kemon          pykakasi - kunrei
     ho             pykakasi - passport
     ho             pykakasi - passport
     kemon          pykakasi - passport
ｽｼ  converts to
     ｽｼ             jaconv
     sushi          pykakasi - hepburn
     susi           pykakasi - kunrei
     sushi          pykakasi - passport
ﾗｰﾒﾝ  converts to
     ﾗｰﾒﾝ           jaconv
     ra゜men         pykakasi - hepburn
     ra゜men         pykakasi - kunrei
     ra゜men         pykakasi - passport

We see that the jaconv.kata2alphabet() method does not work at all, returning the half-width katakana as is. Additionally, the pykakasi conversion works for some of the letters, but does not work for special characters such as the “ﾎﾟ (po)” letter and the elongation bar in ra-men.

2.3.3 Convert Half-Width to Full-Width Katakana Before Transliteration

In order to convert half-width katakana words into romaji, we must first convert half-width to full-width katakana. Luckily, the jaconv.hankaku2zenkaku() method is build just for this purpose.

# Convert half-width to full-width before tranliteration
fullwidth_now = jaconv.hankaku2zenkaku(hw_katakana_text[0])
print(f'`jaconv.hankaku2zenkaku()` converts {hw_katakana_text[0]} to {fullwidth_now}')
jaconv_output = jaconv.kata2alphabet(fullwidth_now)
print(f'`jaconv.kata2alphabet()` converts {fullwidth_now} to {jaconv_output}')

`jaconv.hankaku2zenkaku()` converts ﾄｳｷｮｳ to トウキョウ
`jaconv.kata2alphabet()` converts トウキョウ to toukyou

After the half-width katakana is converted to its full-width equivalent, the jaconv.kata2alphabet() method can transliterate it to the Roman alphabet.

Similarly, the full-width katakana can be used by the pykakasi library for transliteration. Let’s see the effects of the half- to full-width conversion on both libraries using all the listed Japanese words.

# Collection of katakana text
hw_katakana_text = ["ﾄｳｷｮｳ", "ｷｮｳﾄ", "ｵｵｻｶ", "ﾎﾟｹﾓﾝ", "ｽｼ", "ﾗｰﾒﾝ"] 

# Make a pykakasi.kakasi 
kks = pykakasi.kakasi()

# Work through each text item
for katakana_now in hw_katakana_text:

    # Convert from half-width to full-width katakana
    fullwidth_now = jaconv.hankaku2zenkaku(katakana_now)

    # Starting katakana
    print(katakana_now, " converts to ", fullwidth_now, " which ultimately converts to ")

    # Convert using jaconv
    jaconv_hiragana = jaconv.kata2alphabet(fullwidth_now)
    print(f'\t {jaconv_hiragana:<14} jaconv')

    # Convert using pykakasi
    kks_result = kks.convert(fullwidth_now)
    for item in kks_result:
        print(f'\t {item['hepburn']:<14} pykakasi - hepburn')
    for item in kks_result:
        print(f'\t {item['kunrei']:<14} pykakasi - kunrei')
    for item in kks_result:
        print(f'\t {item['passport']:<14} pykakasi - passport')
    

ﾄｳｷｮｳ  converts to  トウキョウ  which ultimately converts to 
     toukyou        jaconv
     toukyou        pykakasi - hepburn
     toukyou        pykakasi - kunrei
     tokyou         pykakasi - passport
ｷｮｳﾄ  converts to  キョウト  which ultimately converts to 
     kyouto         jaconv
     kyouto         pykakasi - hepburn
     kyouto         pykakasi - kunrei
     kyouto         pykakasi - passport
ｵｵｻｶ  converts to  オオサカ  which ultimately converts to 
     oosaka         jaconv
     oosaka         pykakasi - hepburn
     oosaka         pykakasi - kunrei
     osaka          pykakasi - passport
ﾎﾟｹﾓﾝ  converts to  ポケモン  which ultimately converts to 
     pokemon        jaconv
     pokemon        pykakasi - hepburn
     pokemon        pykakasi - kunrei
     pokemon        pykakasi - passport
ｽｼ  converts to  スシ  which ultimately converts to 
     sushi          jaconv
     sushi          pykakasi - hepburn
     susi           pykakasi - kunrei
     sushi          pykakasi - passport
ﾗｰﾒﾝ  converts to  ラーメン  which ultimately converts to 
     ra-men         jaconv
     raamen         pykakasi - hepburn
     raamen         pykakasi - kunrei
     raamen         pykakasi - passport

Both the jaconv and pykakasi libraries are successful in transliterating the Japanese words once the jaconv.hankaku2zenkaku() method has been applied.

3 Summary

In this short tutorial, you learned how to:

• Transliterate hiragana into the Roman alphabet using the jaconv and pykakasi libraries
• Transliterate katakana into the Roman alphabet using the jaconv and pykakasi libraries
• Convert hankaku (half-width) katakana into zenkaku (full-width) katakana using jaconv.hankaku2zenkaku() before transliteration

Ultimately, the choice of Python library to use in your transliteration task is personal preference. My basic recommendations are:

• Use jaconv library if you need to deal with half-width katakana
• Use pykakasi library if you need to deal with a mix of hiragana, katakana and kanji
• When in doubt, use both

4 Further Readings

• Getting a UnicodeDecodeError when opening a mystery CSV file? Check out this blog post on reading a CSV file with unknown encoding into a pandas DataFrame