Detect encoding and make everything UTF-8

auto detect encoding online
php detect file encoding
php convert ascii to utf-8
php convert to utf-8
mb_detect_encoding not working
convert special characters to utf-8
iconv utf-8
convert iconv utf 8

I'm reading out lots of texts from various RSS feeds and inserting them into my database.

Of course, there are several different character encodings used in the feeds, e.g. UTF-8 and ISO 8859-1.

Unfortunately, there are sometimes problems with the encodings of the texts. Example:

  1. The "ß" in "Fußball" should look like this in my database: "Ÿ". If it is a "Ÿ", it is displayed correctly.

  2. Sometimes, the "ß" in "Fußball" looks like this in my database: "ß". Then it is displayed wrongly, of course.

  3. In other cases, the "ß" is saved as a "ß" - so without any change. Then it is also displayed wrongly.

What can I do to avoid the cases 2 and 3?

How can I make everything the same encoding, preferably UTF-8? When must I use utf8_encode(), when must I use utf8_decode() (it's clear what the effect is but when must I use the functions?) and when must I do nothing with the input?

How do I make everything the same encoding? Perhaps with the function mb_detect_encoding()? Can I write a function for this? So my problems are:

  1. How do I find out what encoding the text uses?
  2. How do I convert it to UTF-8 - whatever the old encoding is?

Would a function like this work?

function correct_encoding($text) {
    $current_encoding = mb_detect_encoding($text, 'auto');
    $text = iconv($current_encoding, 'UTF-8', $text);
    return $text;
}

I've tested it, but it doesn't work. What's wrong with it?

If you apply utf8_encode() to an already UTF-8 string, it will return garbled UTF-8 output.

I made a function that addresses all this issues. It´s called Encoding::toUTF8().

You don't need to know what the encoding of your strings is. It can be Latin1 (ISO 8859-1), Windows-1252 or UTF-8, or the string can have a mix of them. Encoding::toUTF8() will convert everything to UTF-8.

I did it because a service was giving me a feed of data all messed up, mixing UTF-8 and Latin1 in the same string.

Usage:

require_once('Encoding.php');
use \ForceUTF8\Encoding;  // It's namespaced now.

$utf8_string = Encoding::toUTF8($utf8_or_latin1_or_mixed_string);

$latin1_string = Encoding::toLatin1($utf8_or_latin1_or_mixed_string);

Download:

https://github.com/neitanod/forceutf8

I've included another function, Encoding::fixUFT8(), which will fix every UTF-8 string that looks garbled.

Usage:

require_once('Encoding.php');
use \ForceUTF8\Encoding;  // It's namespaced now.

$utf8_string = Encoding::fixUTF8($garbled_utf8_string);

Examples:

echo Encoding::fixUTF8("Fédération Camerounaise de Football");
echo Encoding::fixUTF8("Fédération Camerounaise de Football");
echo Encoding::fixUTF8("FÃÂédÃÂération Camerounaise de Football");
echo Encoding::fixUTF8("Fédération Camerounaise de Football");

will output:

Fédération Camerounaise de Football
Fédération Camerounaise de Football
Fédération Camerounaise de Football
Fédération Camerounaise de Football

I've transformed the function (forceUTF8) into a family of static functions on a class called Encoding. The new function is Encoding::toUTF8().

mb_detect_encoding - Manual, it returns the detected charset, which you can use to convert the string to utf-8 with iconv. <?php function convertToUTF8($str) { $enc = mb_detect_encoding($​str); if  Your encoding looks like you encoded into UTF-8 twice; that is, from some other encoding, into UTF-8, and again into UTF-8. As if you had iso-8859-1, converted from iso-8859-1 to utf-8, and treated the new string as iso-8859-1 for another conversion into UTF-8.

You first have to detect what encoding has been used. As you’re parsing RSS feeds (probably via HTTP), you should read the encoding from the charset parameter of the Content-Type HTTP header field. If it is not present, read the encoding from the encoding attribute of the XML processing instruction. If that’s missing too, use UTF-8 as defined in the specification.


Edit   Here is what I probably would do:

I’d use cURL to send and fetch the response. That allows you to set specific header fields and fetch the response header as well. After fetching the response, you have to parse the HTTP response and split it into header and body. The header should then contain the Content-Type header field that contains the MIME type and (hopefully) the charset parameter with the encoding/charset too. If not, we’ll analyse the XML PI for the presence of the encoding attribute and get the encoding from there. If that’s also missing, the XML specs define to use UTF-8 as encoding.

$url = 'http://www.lr-online.de/storage/rss/rss/sport.xml';

$accept = array(
    'type' => array('application/rss+xml', 'application/xml', 'application/rdf+xml', 'text/xml'),
    'charset' => array_diff(mb_list_encodings(), array('pass', 'auto', 'wchar', 'byte2be', 'byte2le', 'byte4be', 'byte4le', 'BASE64', 'UUENCODE', 'HTML-ENTITIES', 'Quoted-Printable', '7bit', '8bit'))
);
$header = array(
    'Accept: '.implode(', ', $accept['type']),
    'Accept-Charset: '.implode(', ', $accept['charset']),
);
$encoding = null;
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
$response = curl_exec($curl);
if (!$response) {
    // error fetching the response
} else {
    $offset = strpos($response, "\r\n\r\n");
    $header = substr($response, 0, $offset);
    if (!$header || !preg_match('/^Content-Type:\s+([^;]+)(?:;\s*charset=(.*))?/im', $header, $match)) {
        // error parsing the response
    } else {
        if (!in_array(strtolower($match[1]), array_map('strtolower', $accept['type']))) {
            // type not accepted
        }
        $encoding = trim($match[2], '"\'');
    }
    if (!$encoding) {
        $body = substr($response, $offset + 4);
        if (preg_match('/^<\?xml\s+version=(?:"[^"]*"|\'[^\']*\')\s+encoding=("[^"]*"|\'[^\']*\')/s', $body, $match)) {
            $encoding = trim($match[1], '"\'');
        }
    }
    if (!$encoding) {
        $encoding = 'utf-8';
    } else {
        if (!in_array($encoding, array_map('strtolower', $accept['charset']))) {
            // encoding not accepted
        }
        if ($encoding != 'utf-8') {
            $body = mb_convert_encoding($body, 'utf-8', $encoding);
        }
    }
    $simpleXML = simplexml_load_string($body, null, LIBXML_NOERROR);
    if (!$simpleXML) {
        // parse error
    } else {
        echo $simpleXML->asXML();
    }
}

php: Detect encoding and make everything UTF-8, To make sure all recipients of a document can display and interpret it properly, it is very If the encoding selected or detected is US-ASCII , UTF-8 , UTF-16 , or it is encoded in UTF-8, and so you don't actually have to check anything else.). Your application must detect all of that ! If your UTF-8 application is not appropriately designed, it may be vulnerable to hackers. When you design your application, you must keep in mind all of that. Here, as a simple example, is a function to detect UTF-8 encoding and to extract Unicode out of a string of char. (function not tested).

Detecting the encoding is hard.

mb_detect_encoding works by guessing, based on a number of candidates that you pass it. In some encodings, certain byte-sequences are invalid, an therefore it can distinguish between various candidates. Unfortunately, there are a lot of encodings, where the same bytes are valid (but different). In these cases, there is no way to determine the encoding; You can implement your own logic to make guesses in these cases. For example, data coming from a Japanese site might be more likely to have a Japanese encoding.

As long as you only deal with Western European languages, the three major encodings to consider are utf-8, iso-8859-1 and cp-1252. Since these are defaults for many platforms, they are also the most likely to be reported wrongly about. Eg. if people use different encodings, they are likely to be frank about it, since else their software would break very often. Therefore, a good strategy is to trust the provider, unless the encoding is reported as one of those three. You should still doublecheck that it is indeed valid, using mb_check_encoding (note that valid is not the same as being - the same input may be valid for many encodings). If it is one of those, you can then use mb_detect_encoding to distinguish between them. Luckily that is fairly deterministic; You just need to use the proper detect-sequence, which is UTF-8,ISO-8859-1,WINDOWS-1252.

Once you've detected the encoding you need to convert it to your internal representation (UTF-8 is the only sane choice). The function utf8_encode transforms ISO-8859-1 to UTF-8, so it can only used for that particular input type. For other encodings, use mb_convert_encoding.

Checking the character encoding using the validator, Which character encoding should I use for my content, and how do I apply it to my Choose UTF-8 for all content and consider converting any content in legacy If you really can't use a Unicode encoding, check that there is wide browser  However, when mb_detect_encoding() is supplied an ASCII string (special characters in the Latin1-fields from 192-255) it detects it as UTF-8, hence in the following attempt to convert everything to proper UTF-8 all special characters are removed.

A really nice way to implement an isUTF8-function can be found on php.net:

function isUTF8($string) {
    return (utf8_encode(utf8_decode($string)) == $string);
}

Choosing & applying a character encoding, UTF-16 and UTF-8 are variable-length encodings. turn they need to be told or try to detect the encoding of everything that has to do with text. During my work in updating some old projects im working through some old ANSI/ASCII files and encodings. I want to have everything running utf-8 to make sure that i can support all kinds of languag

This cheatsheet lists some common caveats related to UTF-8 handling in PHP: http://developer.loftdigital.com/blog/php-utf-8-cheatsheet

This function detecting multibyte characters in a string might also prove helpful (source):

function detectUTF8($string)
{
    return preg_match('%(?:
        [\xC2-\xDF][\x80-\xBF]             # non-overlong 2-byte
        |\xE0[\xA0-\xBF][\x80-\xBF]        # excluding overlongs
        |[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2} # straight 3-byte
        |\xED[\x80-\x9F][\x80-\xBF]        # excluding surrogates
        |\xF0[\x90-\xBF][\x80-\xBF]{2}     # planes 1-3
        |[\xF1-\xF3][\x80-\xBF]{3}         # planes 4-15
        |\xF4[\x80-\x8F][\x80-\xBF]{2}     # plane 16
        )+%xs', 
    $string);
}

What Every Programmer Absolutely, Positively Needs to Know , Most stringi functions always return UTF-8 encoded strings, regardless of the input ASCII and ISO-8859-1 (the most widely used character sets) to make it easier that whenever your native encoding is being detected automatically on ICU's  1 Answer 1. mb_detect_encoding gives you the first encoding in which the tested string is valid. If left to its own devices, it tests for ASCII before UTF-8. Since a URL-encoded string consists solely of a subset of ASCII characters, it is valid ASCII and mb_detect_encoding will tell you so.

stringi-encoding function, Automatically detect encoding. Make sure this option is checked, otherwise UltraEdit won't do anything to try and automatically detect the encoding of the files you  If present, then use that encoding. If absent, then assume UTF-8, which is the default XML encoding. If you need to support EBCDIC, also look for the equivalent sequence 4C 6F A7 94 93. In general, if you have a file format that contains an encoding declaration, then look for that declaration rather than trying to guess the encoding. None of the above

Edit Unicode UTF-16 and UTF-8 text and files in UltraEdit, Everything is encoded in some way, including “plain text. UTF-8, UTF-16, and UTF-32 all exist as separate things and encode the Unicode standard. Even if I did run encoding detection constantly, I'd need to make sure all  la détection de l'encodage est difficile. mb_detect_encoding fonctionne en devinant, basé sur un certain nombre de candidats que vous réussissez. Dans certains codages, certaines séquences d'octets sont invalides, ce qui permet de distinguer différents candidats.

Character Encodings, If you have a repository with source files that do not have UTF-8, you have two options: Convert all files in the repository to ASCII or UTF-8 (see "Detecting and  Detect encoding and make everything UTF-8 (16) Ÿ is Mojibake for ß. In your database, you may have hex DF if the column is "latin1", C39F if the column is utf8 -- OR -- it is latin1, but "double-encoded" C383C5B8 if double-encoded into a utf8 column

Comments
  • "The "ß" in "Fußball" should look like this in my database: "Ÿ".". No it should look like ß. Make sure you collation and connection are set up correctly. Otherwise sorting and searching will be broken for you.
  • Your database is badly setup. If you want to store Unicode content, just configure it for that. So instead of trying to workaround the issue in your PHP code, you should first fix the database.
  • USE: $from=mb_detect_encoding($text); $text=mb_convert_encoding($text,'UTF-8',$from);
  • Well, if you look at the code, fixUTF8 simply calls forceUTF8 once and again until the string is returned unchanged. One call to fixUTF8() takes at least twice the time of a call to forceUTF8(), so it's a lot less performant. I made fixUTF8() just to create a command line program that would fix "encode-corrupted" files, but in a live environment is rarely needed.
  • How does this convert non-UTF8 characters to UTF8, without knowing what encoding the invalid characters are in to begin with?
  • It assumes ISO-8859-1, the answer already says this. The only difference between forceUTF8() and utf8_encode() is that forceUTF8() recognizes UTF8 characters and keeps them unchanged.
  • "You dont need to know what the encoding of your strings is." - I very much disagree. Guessing and trying may work, but you'll always sooner or later encounter edge cases where it doesn't.
  • I totally agree. In fact, I didn't mean to state that as a general rule, just explain that this class might help you if that's the situation you happen to find yourself in.
  • Thanks. This would be easy. But would it really work? There are often wrong encodings given in the HTTP headers or in the attributes of XML.
  • Again: That’s not your problem. Standards were established to avoid such troubles. If others don’t follow them, it’s their problem, not yours.