ProgrammingInPython/modules/Unicode.html at gh-pages · UWPCE-PythonCert/ProgrammingInPython

595 lines (573 loc) · 41.8 KB
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Unicode in Python &mdash; Programming in Python 7.0 documentation</title>
      <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
      <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
        <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
    <script src="../_static/js/theme.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="Iterators and Generators" href="IteratorsAndGenerators.html" />
    <link rel="prev" title="Persistence and Serialization" href="PersistanceAndSerialization.html" /> 
<body class="wy-body-for-nav"> 
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #4b2e83" >
            <a href="../index.html">
            <img src="../_static/UWPCE_logo_full.png" class="logo" alt="Logo"/>
          </a>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
        </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
              <p class="caption" role="heading"><span class="caption-text">Topics in the Program</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../topics/01-setting_up/index.html">1. Setting up your Environment</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/02-basic_python/index.html">2. Basic Python</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/03-recursion_booleans/index.html">3. Booleans and Recursion</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/04-sequences_iteration/index.html">4. Sequences and Iteration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/05-text_handling/index.html">5. Basic Text Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/06-exceptions/index.html">6. Exception Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/07-unit_testing/index.html">7. Unit Testing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/08-dicts_sets/index.html">8. Dictionaries and Sets</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/09-files/index.html">9. File Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/10-modules_packages/index.html">10. Modules and Packages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/11-argument_passing/index.html">11. Advanced Argument Passing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/12-comprehensions/index.html">12. Comprehensions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/13-intro_oo/index.html">13. Intro to Object Oriented Programing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/14-magic_methods/index.html">14. Properties and Magic Methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/15-subclassing/index.html">15. Subclassing and Inheritance</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/16-multiple_inheritance/index.html">16. Multiple Inheritance</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/17-functional_programming/index.html">17. Introduction to Functional Programming</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/18-advanced_testing/index.html">18. Advanced Testing</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../topics/99-extras/index.html">19. Extra Topics</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="Pep8.html">Coding Style and Linting</a></li>
<li class="toctree-l2"><a class="reference internal" href="CodeReviews.html">Code Reviews</a></li>
<li class="toctree-l2"><a class="reference internal" href="PersistanceAndSerialization.html">Persistence and Serialization</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Unicode in Python</a></li>
<li class="toctree-l2"><a class="reference internal" href="IteratorsAndGenerators.html">Iterators and Generators</a></li>
<li class="toctree-l2"><a class="reference internal" href="Decorators.html">Decorators</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/mailroom/mailroom-decorator.html">Mailroom – Decoratoring it</a></li>
<li class="toctree-l2"><a class="reference internal" href="ContextManagers.html">Context Managers</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/context-managers-exercise.html">A Couple Handy Context Managers</a></li>
<li class="toctree-l2"><a class="reference internal" href="MetaProgramming.html">Metaprogramming</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/mailroom/mailroom-meta.html">Mailroom – metaprogramming it!</a></li>
<li class="toctree-l2"><a class="reference internal" href="Logging.html">Logging and the logging module</a></li>
<li class="toctree-l2"><a class="reference internal" href="Debugging.html">Debugging</a></li>
<li class="toctree-l2"><a class="reference internal" href="NoSQL.html">No SQL Databases</a></li>
<li class="toctree-l2"><a class="reference internal" href="GraphDatabases.html">Graph Databases</a></li>
<li class="toctree-l2"><a class="reference internal" href="Concurrency.html">Concurrent Programming</a></li>
<li class="toctree-l2"><a class="reference internal" href="Async.html">Asychronous Programming</a></li>
<li class="toctree-l2"><a class="reference internal" href="Coroutines.html">Notes on Coroutines</a></li>
<li class="toctree-l2"><a class="reference internal" href="ThreadingMultiprocessing.html">Threading and multiprocessing</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/threaded_downloader.html">Threaded Web Scraper</a></li>
<li class="toctree-l2"><a class="reference internal" href="Profiling.html">Performance and Profiling</a></li>
        </div>
      </div>
    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu"  style="background: #4b2e83" >
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">Programming in Python</a>
      </nav>
      <div class="wy-nav-content">
        <div class="rst-content style-external-links">
          <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
          <li><a href="../topics/99-extras/index.html"><span class="section-number">19. </span>Extra Topics</a> &raquo;</li>
      <li>Unicode in Python</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../_sources/modules/Unicode.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul><div class="rst-breadcrumbs-buttons" role="navigation" aria-label="Sequential page navigation">
        <a href="PersistanceAndSerialization.html" class="btn btn-neutral float-left" title="Persistence and Serialization" accesskey="p"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="IteratorsAndGenerators.html" class="btn btn-neutral float-right" title="Iterators and Generators" accesskey="n">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
  <div class="section" id="unicode-in-python">
<span id="unicode"></span><h1>Unicode in Python<a class="headerlink" href="#unicode-in-python" title="Permalink to this headline"></a></h1>
<p>A quick run-down of Unicode,</p>
<p>Its use in Python 2 and 3,</p>
<p>and some of the gotchas that arise.</p>
<div class="section" id="history">
<h2>History<a class="headerlink" href="#history" title="Permalink to this headline"></a></h2>
<p>A bit about where all this mess came from…</p>
<div class="section" id="what-the-heck-is-unicode-anyway">
<h3>What the heck is Unicode anyway?<a class="headerlink" href="#what-the-heck-is-unicode-anyway" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li><p>First there was chaos…</p>
<li><p>Different machines used different encodings – different ways of mapping
the binary data that the computer stores to letters.</p></li>
<li><p>Then there was ASCII – and all was good (7 bit), 127 characters</p>
<li><p>(for English speakers, anyway)</p></li>
<li><p>But each vendor used the top half of 8bit bytes (127-255) for different things.</p>
<li><p>MacRoman, Windows 1252, etc…</p></li>
<li><p>There is now “latin-1”, a 1-byte encoding suitable for European languages – but still a lot of old files around that use the old ones.</p></li>
<li><p>Non-Western European languages required totally incompatible 1-byte encodings</p></li>
<li><p>This means there was no way to mix languages with different alphabets in the same document (web page, etc.)</p></li>
<div class="section" id="enter-unicode">
<h3>Enter Unicode<a class="headerlink" href="#enter-unicode" title="Permalink to this headline"></a></h3>
<dl class="simple">
<dt>The Unicode idea is pretty simple:</dt><dd><ul class="simple">
<li><p>One “code point” for all characters in all languages</p></li>
<dt>But how do you express that in bytes?</dt><dd><ul class="simple">
<li><p>Early days: we can fit all the code points in a two byte integer (65536 characters)</p></li>
<li><p>Turns out that didn’t work – 65536 is not enough for all languages. So we now need 32 bit integer to hold all of Unicode “raw” (UTC-4).</p></li>
<li><p>But it’s a waste of space to use 4 full bytes for each character, when so many don’t require that much space.</p></li>
<dt>Enter “encodings”:</dt><dd><ul class="simple">
<li><p>An encoding is a way to map specific bytes to a code point.</p></li>
<li><p>Each code point can be represented by one or more bytes.</p></li>
<li><p>Each encoding is different – if you don’t know the encoding, you don’t know how to interpret the bytes! (though maybe you can guess)</p></li>
<div class="section" id="id1">
<h3>Unicode<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h3>
<p>A good start:</p>
<p>The Absolute Minimum Every Software Developer Absolutely,
Positively Must Know About Unicode and Character Sets (No Excuses!)</p>
<p><a class="reference external" href="http://www.joelonsoftware.com/articles/Unicode.html">http://www.joelonsoftware.com/articles/Unicode.html</a></p>
<p><strong>Everything is Bytes</strong></p>
<ul class="simple">
<li><p>If it’s on disk or on a network, it’s bytes</p></li>
<li><p>Python provides some abstractions to make it easier to deal with bytes</p></li>
<p><strong>Unicode is a Biggie</strong></p>
<p>Actually, dealing with numbers rather than bytes is big</p>
<p>– but we take that for granted.</p>
<div class="section" id="mechanics">
<h2>Mechanics<a class="headerlink" href="#mechanics" title="Permalink to this headline"></a></h2>
<div class="section" id="what-are-strings">
<h3>What are strings?<a class="headerlink" href="#what-are-strings" title="Permalink to this headline"></a></h3>
<p>Py2 strings were simply sequences of bytes.  When text was one per character that worked fine.</p>
<p>Py3 strings (or Unicode strings in py2) are sequences of “platonic characters”.</p>
<p>It’s almost one code point per character – there are complications
with combined characters: accents, etc – but we can mostly ignore those – you will get far thinking of a code point as a character.</p>
<p>Platonic characters cannot be written to disk or network!</p>
<p>(ANSI: one character == one byte – it was so easy!)</p>
<div class="section" id="strings-vs-unicode">
<h3>Strings vs Unicode<a class="headerlink" href="#strings-vs-unicode" title="Permalink to this headline"></a></h3>
<p>Python 2 had two types that let you work with text:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">str</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">unicode</span></code></p></li>
<p>And two ways to work with binary data:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">str</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">bytes()</span></code>  (and <code class="docutils literal notranslate"><span class="pre">bytearray</span></code>)</p></li>
<p><strong>but:</strong></p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [86]: </span><span class="nb">str</span> <span class="ow">is</span> <span class="nb">bytes</span>
<span class="gh">Out[86]: </span><span class="go">True</span>
</pre></div>
<p><code class="docutils literal notranslate"><span class="pre">bytes</span></code> is there in py2 for py3 compatibility – but it’s good for making your intentions clear, too.</p>
<p>py3 is more clear:</p>
<blockquote>
<div><p><code class="docutils literal notranslate"><span class="pre">str</span></code> for text
<code class="docutils literal notranslate"><span class="pre">bytes</span></code> for binary data</p>
</div></blockquote>
<div class="section" id="id2">
<h3>Unicode<a class="headerlink" href="#id2" title="Permalink to this headline"></a></h3>
<p>The py3 string (py2 <code class="docutils literal notranslate"><span class="pre">Unicode</span></code>) object lets you work with characters, instead of bytes.</p>
<p>It has all the same methods you’d expect a string object to have.</p>
<div class="section" id="encoding-decoding">
<h3>Encoding / Decoding<a class="headerlink" href="#encoding-decoding" title="Permalink to this headline"></a></h3>
<p>If you need to deal with the actual bytes for some reason, you may need to convert between a string object and a particular set of bytes.</p>
<p><strong>“encoding”</strong> is converting from a string object to bytes</p>
<p><strong>“decoding”</strong> is converting from bytes to a string object</p>
<p>(sometimes this feels backwards…)</p>
<p>And can get even more confusing with py2 strings being <em>both</em> text and bytes!</p>
<p>This is actually one of the biggest differences between Python 2 and Python 3. As an ordinary user (particularly one that used English…), you may not notice – text is text, and things generally “just work”, but under the hood it is very different, and folks writting libraries for things like Internet protocols struggled with the differences.</p>
<div class="section" id="using-unicode-in-py2">
<h3>Using Unicode in Py2<a class="headerlink" href="#using-unicode-in-py2" title="Permalink to this headline"></a></h3>
<p>If you do need to write Python2 code, you really should use Unicode. If you don’t – skip ahead to “Unicode Literals”.</p>
<p>Here are the basics:</p>
<div class="section" id="built-in-functions">
<h4>Built in functions<a class="headerlink" href="#built-in-functions" title="Permalink to this headline"></a></h4>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nb">ord</span><span class="p">()</span>
<span class="nb">chr</span><span class="p">()</span>
<span class="n">unichr</span><span class="p">()</span>
<span class="nb">str</span><span class="p">()</span>
<span class="n">unicode</span><span class="p">()</span>
</pre></div>
<div class="section" id="the-codecs-module">
<h4>The codecs module<a class="headerlink" href="#the-codecs-module" title="Permalink to this headline"></a></h4>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">codecs</span>
<span class="n">codecs</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span>
<span class="n">codecs</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="n">codecs</span><span class="o">.</span><span class="n">open</span><span class="p">()</span> <span class="c1"># better to use ``io.open``</span>
</pre></div>
<div class="section" id="encoding-and-decoding">
<h3>Encoding and Decoding<a class="headerlink" href="#encoding-and-decoding" title="Permalink to this headline"></a></h3>
<p>(Python 2!)</p>
<p><strong>Encoding:</strong> text to bytes – you get a bytes (str) object</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [17]: </span><span class="sa">u</span><span class="s2">&quot;this&quot;</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="gh">Out[17]: </span><span class="go">&#39;this&#39;</span>
<span class="gp">In [18]: </span><span class="sa">u</span><span class="s2">&quot;this&quot;</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-16&#39;</span><span class="p">)</span>
<span class="gh">Out[18]: </span><span class="go">&#39;\xff\xfet\x00h\x00i\x00s\x00&#39;</span>
</pre></div>
<p><strong>Decoding</strong> bytes to text – you get a unicode object</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [2]: </span><span class="n">text</span> <span class="o">=</span>  <span class="s1">&#39;</span><span class="se">\xff\xfe</span><span class="s1">.&quot;+&quot;x</span><span class="se">\x00\xb2\x00</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-16&#39;</span><span class="p">)</span>
<span class="gp">In [3]: </span><span class="nb">type</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
<span class="gh">Out[3]: </span><span class="go">unicode</span>
<span class="gp">In [4]: </span><span class="nb">print</span> <span class="n">text</span>
<span class="go">∮∫x²</span>
</pre></div>
<div class="section" id="unicode-literals">
<h3>Unicode Literals<a class="headerlink" href="#unicode-literals" title="Permalink to this headline"></a></h3>
<p>How do you get text that isn’t plain English text?</p>
<ol class="arabic simple">
<li><p>Use Unicode in your source files:</p></li>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># -*- coding: utf-8 -*-</span>
</pre></div>
<p>(This is only required on Py2 – the UTF-8 encoding is default for Python 3)</p>
<ol class="arabic simple" start="2">
<li><p>Escape the Unicode characters:</p></li>
<p>Either by its hexadecimal value, or its name:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span> <span class="sa">u</span><span class="s2">&quot;The integral sign: </span><span class="se">\u222B</span><span class="s2">&quot;</span>
<span class="nb">print</span> <span class="sa">u</span><span class="s2">&quot;The integral sign: </span><span class="se">\N{integral}</span><span class="s2">&quot;</span>
</pre></div>
<p>Lots of tables of code points are available online:</p>
<p>One example:  <a class="reference external" href="http://inamidst.com/stuff/unidata/">http://inamidst.com/stuff/unidata/</a></p>
<p><a class="reference download internal" download="" href="../_downloads/da2475249504456dba1a0c7efa6f352d/hello_unicode.py"><code class="xref download docutils literal notranslate"><span class="pre">hello_unicode.py</span></code></a>.</p>
<div class="section" id="using-unicode">
<h3>Using Unicode<a class="headerlink" href="#using-unicode" title="Permalink to this headline"></a></h3>
<p>Use <code class="docutils literal notranslate"><span class="pre">unicode</span></code> objects in all your code</p>
<p><strong>Decode on input</strong></p>
<p><strong>Encode on output</strong></p>
<p>Many packages do this for you: <em>XML processing, databases, …</em></p>
<p><strong>Gotcha:</strong></p>
<p>Python has a default encoding. On Mac and Unix systems, it’s usually utf-8 – Windows? not nearly as consistent.</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [7]: </span><span class="n">sys</span><span class="o">.</span><span class="n">getdefaultencoding</span><span class="p">()</span>
<span class="gh">Out[7]: </span><span class="go">&#39;utf-8&#39;</span>
</pre></div>
<p>Try this on your machine, and see what you get.</p>
<p>The default encoding will get used in unexpected places!
Notably in text files by default.</p>
<div class="section" id="using-unicode-everywhere">
<h3>Using Unicode Everywhere<a class="headerlink" href="#using-unicode-everywhere" title="Permalink to this headline"></a></h3>
<p>Python 2.6 and above have a nice feature to make it easier to use Unicode everywhere</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">unicode_literals</span>
</pre></div>
<p>After running that line, the <code class="docutils literal notranslate"><span class="pre">u''</span></code> is assumed</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [1]: </span><span class="n">s</span> <span class="o">=</span> <span class="s2">&quot;this is a regular py2 string&quot;</span>
<span class="gp">In [2]: </span><span class="nb">print</span> <span class="nb">type</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="go">&lt;type &#39;str&#39;&gt;</span>
<span class="gp">In [3]: </span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">unicode_literals</span>
<span class="gp">In [4]: </span><span class="n">s</span> <span class="o">=</span> <span class="s2">&quot;this is now a unicode string&quot;</span>
<span class="gp">In [5]: </span><span class="nb">type</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="gh">Out[5]: </span><span class="go">unicode</span>
</pre></div>
<p>NOTE: You can still get py2 strings from other sources! So you still need to think about <code class="docutils literal notranslate"><span class="pre">str</span></code> vs <code class="docutils literal notranslate"><span class="pre">unicdode</span></code></p>
<p>This is a really good idea if you want to write code compatible with Python2 and 3.</p>
<div class="section" id="encodings">
<h3>Encodings<a class="headerlink" href="#encodings" title="Permalink to this headline"></a></h3>
<p>What encoding should I use???</p>
<p>There are a lot:</p>
<p><a class="reference external" href="http://en.wikipedia.org/wiki/Comparison_of_Unicode_encodings">http://en.wikipedia.org/wiki/Comparison_of_Unicode_encodings</a></p>
<p>But only a couple you are likely to need:</p>
<ul class="simple">
<li><p>utf-8  (<code class="docutils literal notranslate"><span class="pre">*nix</span></code>)</p></li>
<li><p>utf-16  (Windows)</p></li>
<p>and of course, still the one-bytes ones.</p>
<ul class="simple">
<li><p>ASCII</p></li>
<li><p>Latin-1</p></li>
<div class="section" id="utf-8">
<h3>UTF-8<a class="headerlink" href="#utf-8" title="Permalink to this headline"></a></h3>
<p>Probably the one you’ll use most – most common in Internet protocols (xml, JSON, etc.)</p>
<p>Nice properties:</p>
<ul class="simple">
<li><p>ASCII compatible: First 127 characters are the same as ASCII</p></li>
<li><p>Any ascii string is a utf-8 string</p></li>
<li><p>Compact for mostly-English text.</p></li>
<p>Gotchas:</p>
<ul class="simple">
<li><p>“higher” code points may use more than one byte: up to 4 for one character</p></li>
<li><p>ASCII compatible means in may work with default encoding in tests – but then blow up with real data…</p></li>
<div class="section" id="utf-16">
<h3>UTF-16<a class="headerlink" href="#utf-16" title="Permalink to this headline"></a></h3>
<p>Kind of like UTF-8, except it uses at least 16bits (2 bytes) for each character: NOT ASCII compatible.</p>
<p>But it still needs more than two bytes for some code points, so you still can’t simply process it as two bytes per character.</p>
<p>In C/C++, it is held in a “wide char” or “wide string”.</p>
<p>MS Windows uses UTF-16, as does (I think) Java.</p>
<div class="section" id="utf-16-criticism">
<h3>UTF-16 criticism<a class="headerlink" href="#utf-16-criticism" title="Permalink to this headline"></a></h3>
<p>There is a lot of criticism on the net about UTF-16 – it’s kind of the worst of both worlds:</p>
<ul class="simple">
<li><p>You can’t assume every character is the same number of bytes</p></li>
<li><p>It takes up more memory than UTF-8</p></li>
<p><a class="reference external" href="http://programmers.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful">UTF-16 Considered Harmful</a></p>
<p>But to be fair:</p>
<p>Early versions of Unicode: everything fit into two bytes (65536 code points). MS and Java were fairly early adopters, and it seemed simple enough to just use 2 bytes per character.</p>
<p>When it turned out that 4 bytes were really needed, they were kind of stuck in the middle.</p>
<div class="section" id="latin-1">
<h3>Latin-1<a class="headerlink" href="#latin-1" title="Permalink to this headline"></a></h3>
<p><strong>NOT Unicode</strong>:</p>
<p>A 1-byte per char encoding.</p>
<ul class="simple">
<li><p>Superset of ASCII suitable for Western European languages.</p></li>
<li><p>The most common one-byte per char encoding for European text.</p></li>
<li><p>Nice property – every byte value from 1 to 255 is a valid character ( at least in Python )</p></li>
<li><p>You will never get an UnicodeDecodeError if you try to decode arbitrary bytes with latin-1.</p></li>
<li><p>And it can “round-trip” through a unicode object.</p></li>
<li><p>Useful if you don’t know the encoding – at least it won’t raise an Exception</p></li>
<li><p>Useful if you need to work with combined text+binary data.</p></li>
<p><a class="reference download internal" download="" href="../_downloads/c71f489791c94e7196c17ed340e069b0/latin1_test.py"><code class="xref download docutils literal notranslate"><span class="pre">latin1_test.py</span></code></a>.</p>
<div class="section" id="unicode-docs">
<h3>Unicode Docs<a class="headerlink" href="#unicode-docs" title="Permalink to this headline"></a></h3>
<p>Python Docs Unicode HowTo:</p>
<p><a class="reference external" href="http://docs.python.org/howto/unicode.html">http://docs.python.org/howto/unicode.html</a></p>
<p>“Reading Unicode from a file is therefore simple”</p>
<p>use plain old open:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nb">open</span><span class="p">(</span><span class="s1">&#39;unicode.rst&#39;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">:</span>
    <span class="nb">print</span> <span class="nb">repr</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
</pre></div>
<dl class="simple">
<dt>Encodings Built-in to Python:</dt><dd><p><a class="reference external" href="http://docs.python.org/3/library/codecs.html#standard-encodings">http://docs.python.org/3/library/codecs.html#standard-encodings</a></p>
<div class="section" id="gotchas-in-python-2">
<h3>Gotchas in Python 2<a class="headerlink" href="#gotchas-in-python-2" title="Permalink to this headline"></a></h3>
<p>(Probaly do’nt need to worry about this anymore!)</p>
<p>File names, etc:</p>
<p>If you pass in unicode, you get unicode</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [9]: </span><span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="s1">&#39;./&#39;</span><span class="p">)</span>
<span class="gh">Out[9]: </span><span class="go">[&#39;hello_unicode.py&#39;, &#39;text.utf16&#39;, &#39;text.utf32&#39;]</span>
<span class="gp">In [10]: </span><span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="sa">u</span><span class="s1">&#39;./&#39;</span><span class="p">)</span>
<span class="gh">Out[10]: </span><span class="go">[u&#39;hello_unicode.py&#39;, u&#39;text.utf16&#39;, u&#39;text.utf32&#39;]</span>
</pre></div>
<p>Python deals with the file system encoding for you…</p>
<p>But: some more obscure calls don’t support unicode filenames:</p>
<p><code class="docutils literal notranslate"><span class="pre">os.statvfs()</span></code> (<a class="reference external" href="http://bugs.python.org/issue18695">http://bugs.python.org/issue18695</a>)</p>
<p>Exception messages:</p>
<blockquote>
<div><ul class="simple">
<li><p>Py2 Exceptions use str when they print messages.</p></li>
<li><p>But what if you pass in a unicode object?</p>
<li><p>It is encoded with the default encoding.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">UnicodeDecodeError</span></code> Inside an Exception????</p></li>
<p>NOPE: it swallows it instead.</p>
</div></blockquote>
<p><a class="reference download internal" download="" href="../_downloads/bb22abb4db949be359ed622d72ba6613/exception_test.py"><code class="xref download docutils literal notranslate"><span class="pre">exception_test.py</span></code></a>.</p>
<div class="section" id="unicode-in-python-3">
<h3>Unicode in Python 3<a class="headerlink" href="#unicode-in-python-3" title="Permalink to this headline"></a></h3>
<p>The “string” object <strong>is</strong> Unicode (always).</p>
<p>Py3 has two distinct concepts:</p>
<ul class="simple">
<li><p>“text” – uses the str object (which is always Unicode!)</p></li>
<li><p>“binary data” – uses bytes or bytearray</p></li>
<p>Everything that’s about text is Unicode.</p>
<p>Everything that requires binary data uses bytes.</p>
<p>It’s all much cleaner.</p>
<p>(by the way, the recent implementations are very efficient…)</p>
<p>So you can pretty much ignore encodings and all that for most basic text processing.
If you do find yourself needing to deal with binary data, you ay need to encode/decode stuff yourself.
In which case, Python provides an <code class="docutils literal notranslate"><span class="pre">.encode()</span></code> method on strings that encode the string to a bytes object with the encoding you select:</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [3]: </span><span class="n">this_in_utf16</span> <span class="o">=</span> <span class="s2">&quot;this&quot;</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-16&#39;</span><span class="p">)</span>
<span class="gp">In [4]: </span><span class="n">this_in_utf16</span>
<span class="gh">Out[4]: </span><span class="go">b&#39;\xff\xfet\x00h\x00i\x00s\x00&#39;</span>
</pre></div>
<p>And bytes objects have a <code class="docutils literal notranslate"><span class="pre">.decode</span></code> method that decodes the bytes and makes a string object:</p>
<blockquote>
<div><p>In [5]: this_in_utf16.decode(‘utf-16’)
Out[5]: ‘this’</p>
</div></blockquote>
<p>It’s all quite simple an robust.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>During the long and painful transition from Python2 to Python3, the Unicode-always string type was a major source of complaints.  There are many rants and <a class="reference external" href="http://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/">well thought out posts</a> about it still available on the internet. It was enough to think that Python had made a huge mistake.</p>
<p>But there are a couple key points to remember:</p>
<ul class="simple">
<li><p>The primary people struggling were those that wrote (or worked with) libraries that had to deal with protocols that used both binary and text data in the same data stream.</p></li>
<li><p>As of Python 3.4 or so, the python string object had grown the features it needed to support even those ugly binary+text use cases.</p></li>
<p>For a typical user, the Python3 text model is MUCH easier to deal with and less error prone.</p>
<div class="section" id="exercises">
<h2>Exercises<a class="headerlink" href="#exercises" title="Permalink to this headline"></a></h2>
<div class="section" id="basic-unicode-lab">
<h3>Basic Unicode LAB<a class="headerlink" href="#basic-unicode-lab" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li><p>Find some nifty non-ascii characters you might use.</p>
<li><p>Create a unicode object with them in two different ways.</p></li>
<li><p><a class="reference download internal" download="" href="../_downloads/da2475249504456dba1a0c7efa6f352d/hello_unicode.py"><code class="xref download docutils literal notranslate"><span class="pre">here</span></code></a> is one example</p></li>
<li><p>Read the contents into unicode objects:</p></li>
<blockquote>
<div><ul class="simple">
<li><p><a class="reference download internal" download="" href="../_downloads/8cfc8668d4a488ede3e3281cb5fcd659/ICanEatGlass.utf8.txt"><code class="xref download docutils literal notranslate"><span class="pre">ICanEatGlass.utf8.txt</span></code></a></p></li>
<li><p><a class="reference download internal" download="" href="../_downloads/d8a2901456b14379487d17a1625a4d3b/ICanEatGlass.utf16.txt"><code class="xref download docutils literal notranslate"><span class="pre">ICanEatGlass.utf16.txt</span></code></a></p></li>
</div></blockquote>
<p>and / or</p>
<blockquote>
<div><ul class="simple">
<li><p><a class="reference download internal" download="" href="../_downloads/8996f535304b3e2e4a45ee10526d0bed/text.utf8"><code class="xref download docutils literal notranslate"><span class="pre">text.utf8</span></code></a></p></li>
<li><p><a class="reference download internal" download="" href="../_downloads/f945dd005bb544a3776f773f5b1cee8f/text.utf16"><code class="xref download docutils literal notranslate"><span class="pre">text.utf16</span></code></a></p></li>
<li><p><a class="reference download internal" download="" href="../_downloads/755eb41102dde9e2c2ec1db66c41339b/text.utf32"><code class="xref download docutils literal notranslate"><span class="pre">text.utf32</span></code></a></p></li>
</div></blockquote>
<ul class="simple">
<li><p>write some of the text from the first exercise to file – read that file back in.</p></li>
<div class="section" id="some-help">
<h3>Some Help<a class="headerlink" href="#some-help" title="Permalink to this headline"></a></h3>
<p>reference: <a class="reference external" href="http://inamidst.com/stuff/unidata/">http://inamidst.com/stuff/unidata/</a></p>
<p>NOTE: if your terminal does not support unicode – you’ll get an error trying to print.
Try a different terminal or IDE, or google for a solution.</p>
<div class="section" id="challenge-unicode-lab">
<h3>Challenge Unicode LAB<a class="headerlink" href="#challenge-unicode-lab" title="Permalink to this headline"></a></h3>
<p>Here is an error in Python2:</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [38]: </span><span class="sa">u</span><span class="s1">&#39;to </span><span class="se">\N{INFINITY}</span><span class="s1"> and beyond!&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">UnicodeEncodeError</span><span class="g g-Whitespace">                        </span>Traceback (most recent call last)
<span class="nn">&lt;ipython-input-38-7f87d44dfcfa&gt;</span> in <span class="ni">&lt;module&gt;</span><span class="nt">()</span>
<span class="ne">----&gt; </span><span class="mi">1</span> <span class="sa">u</span><span class="s1">&#39;to </span><span class="se">\N{INFINITY}</span><span class="s1"> and beyond!&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="nn">/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.pyc</span> in <span class="ni">decode</span><span class="nt">(input, errors)</span>
<span class="g g-Whitespace">     </span><span class="mi">14</span>
<span class="g g-Whitespace">     </span><span class="mi">15</span> <span class="k">def</span> <span class="nf">decode</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">errors</span><span class="o">=</span><span class="s1">&#39;strict&#39;</span><span class="p">):</span>
<span class="ne">---&gt; </span><span class="mi">16</span>     <span class="k">return</span> <span class="n">codecs</span><span class="o">.</span><span class="n">utf_8_decode</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">errors</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span>
<span class="g g-Whitespace">     </span><span class="mi">17</span>
<span class="g g-Whitespace">     </span><span class="mi">18</span> <span class="k">class</span> <span class="nc">IncrementalEncoder</span><span class="p">(</span><span class="n">codecs</span><span class="o">.</span><span class="n">IncrementalEncoder</span><span class="p">):</span>
<span class="nn">UnicodeEncodeError: &#39;ascii&#39; codec can&#39;t encode character u&#39;\u221e&#39; in position 3: ordinal not</span> in <span class="ni">range</span><span class="nt">(128)</span>
</pre></div>
<p>But why would you <strong>decode</strong> a unicode object?</p>
<p>And it should be a no-op – why the exception?</p>
<p>And why ‘ascii’? I specified ‘utf-8’!</p>
<p>It’s there for backward compatibility</p>
<p>What’s happening under the hood</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="sa">u</span><span class="s1">&#39;to </span><span class="se">\N{INFINITY}</span><span class="s1"> and beyond!&#39;</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span>
</pre></div>
<p>It encodes with the default encoding (ascii), then decodes</p>
<p>In this case, it barfs on attempting to encode to ‘ascii’</p>
<p>So never call decode on a unicode object!</p>
<p>But what if someone passes one into a function of yours that’s expecting a py2 string?</p>
<p>Type checking and converting – yeach!</p>
<p>Read:</p>
<p><a class="reference external" href="http://axialcorps.com/2014/03/20/unicode-str/">http://axialcorps.com/2014/03/20/unicode-str/</a></p>
<p>See if you can figure out the decorators:</p>
<p><a class="reference download internal" download="" href="../_downloads/1299d43327253c58b8a484d1185dc914/unicodify.py"><code class="xref download docutils literal notranslate"><span class="pre">unicodify.py</span></code></a>.</p>
<p>(This is advanced Python JuJu: Aren’t you glad I didn’t ask you to write that yourself?)</p>
           </div>
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="PersistanceAndSerialization.html" class="btn btn-neutral float-left" title="Persistence and Serialization" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="IteratorsAndGenerators.html" class="btn btn-neutral float-right" title="Iterators and Generators" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
  <div role="contentinfo">
    <p>&#169; Copyright 2020, University of Washington, Natasha Aleksandrova, Christopher Barker, Brian Dorsey, Cris Ewing, Christy Heaton, Jon Jacky, Maria McKinley, Andy Miles, Rick Riehle, Joseph Schilz, Joseph Sheedy, Hosung Song. Creative Commons Attribution-ShareAlike 4.0 license.</p>
  Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    provided by <a href="https://readthedocs.org">Read the Docs</a>.
        </div>
      </div>
    </section>
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
  </script> 
Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

Unicode.html

Latest commit

History

Unicode.html

File metadata and controls
