Encoding conversation UTF-8 to 1251 in javascript

I need to convert string in UTF-8 to byte array in 1251 codepage in JavaScript. Google says nothing useful. Help :)

EDIT: What is it for.

We have server and client. Server has userpasswords hased with SHA512 and 1251 codepage. Client (web browser) has to hash passwords (provided by user in plain text) with SHA512 and transmit it to the server. The problem is: browser works in UTF-8 encoding, and of course SHA512 in UFT-8 is not the same as SHA512 in 1251. Any Ideas?

EDIT2: got it! :)

//utf8 to 1251 converter (1 byte format, RU/EN support only + any other symbols) by drgluck
function utf8_decode (aa) {
    var bb = '', c = 0;
    for (var i = 0; i < aa.length; i++) {
        c = aa.charCodeAt(i);
        if (c > 127) {
            if (c > 1024) {
                if (c == 1025) {
                    c = 1016;
                } else if (c == 1105) {
                    c = 1032;
                }
                bb += String.fromCharCode(c - 848);
            }
        } else {
            bb += aa.charAt(i);
        }
    }
    return bb;
} 

Usage with SHA512 implementation by Paul Johnston (http://pajhome.org.uk/crypt/md5/sha512.html)

...
return rstr2hex(rstr_sha512(utf8_decode(input)));

This encodes the Unicode codepoints in JavaScript to windows-1251:

var DMap = {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48, 49: 49, 50: 50, 51: 51, 52: 52, 53: 53, 54: 54, 55: 55, 56: 56, 57: 57, 58: 58, 59: 59, 60: 60, 61: 61, 62: 62, 63: 63, 64: 64, 65: 65, 66: 66, 67: 67, 68: 68, 69: 69, 70: 70, 71: 71, 72: 72, 73: 73, 74: 74, 75: 75, 76: 76, 77: 77, 78: 78, 79: 79, 80: 80, 81: 81, 82: 82, 83: 83, 84: 84, 85: 85, 86: 86, 87: 87, 88: 88, 89: 89, 90: 90, 91: 91, 92: 92, 93: 93, 94: 94, 95: 95, 96: 96, 97: 97, 98: 98, 99: 99, 100: 100, 101: 101, 102: 102, 103: 103, 104: 104, 105: 105, 106: 106, 107: 107, 108: 108, 109: 109, 110: 110, 111: 111, 112: 112, 113: 113, 114: 114, 115: 115, 116: 116, 117: 117, 118: 118, 119: 119, 120: 120, 121: 121, 122: 122, 123: 123, 124: 124, 125: 125, 126: 126, 127: 127, 1027: 129, 8225: 135, 1046: 198, 8222: 132, 1047: 199, 1168: 165, 1048: 200, 1113: 154, 1049: 201, 1045: 197, 1050: 202, 1028: 170, 160: 160, 1040: 192, 1051: 203, 164: 164, 166: 166, 167: 167, 169: 169, 171: 171, 172: 172, 173: 173, 174: 174, 1053: 205, 176: 176, 177: 177, 1114: 156, 181: 181, 182: 182, 183: 183, 8221: 148, 187: 187, 1029: 189, 1056: 208, 1057: 209, 1058: 210, 8364: 136, 1112: 188, 1115: 158, 1059: 211, 1060: 212, 1030: 178, 1061: 213, 1062: 214, 1063: 215, 1116: 157, 1064: 216, 1065: 217, 1031: 175, 1066: 218, 1067: 219, 1068: 220, 1069: 221, 1070: 222, 1032: 163, 8226: 149, 1071: 223, 1072: 224, 8482: 153, 1073: 225, 8240: 137, 1118: 162, 1074: 226, 1110: 179, 8230: 133, 1075: 227, 1033: 138, 1076: 228, 1077: 229, 8211: 150, 1078: 230, 1119: 159, 1079: 231, 1042: 194, 1080: 232, 1034: 140, 1025: 168, 1081: 233, 1082: 234, 8212: 151, 1083: 235, 1169: 180, 1084: 236, 1052: 204, 1085: 237, 1035: 142, 1086: 238, 1087: 239, 1088: 240, 1089: 241, 1090: 242, 1036: 141, 1041: 193, 1091: 243, 1092: 244, 8224: 134, 1093: 245, 8470: 185, 1094: 246, 1054: 206, 1095: 247, 1096: 248, 8249: 139, 1097: 249, 1098: 250, 1044: 196, 1099: 251, 1111: 191, 1055: 207, 1100: 252, 1038: 161, 8220: 147, 1101: 253, 8250: 155, 1102: 254, 8216: 145, 1103: 255, 1043: 195, 1105: 184, 1039: 143, 1026: 128, 1106: 144, 8218: 130, 1107: 131, 8217: 146, 1108: 186, 1109: 190}

function UnicodeToWin1251(s) {
    var L = []
    for (var i=0; i<s.length; i++) {
        var ord = s.charCodeAt(i)
        if (!(ord in DMap))
            throw "Character "+s.charAt(i)+" isn't supported by win1251!"
        L.push(String.fromCharCode(DMap[ord]))
    }
    return L.join('')
}

JavaScript stores each character as raw Unicode codepoints, so the above should work before getting the hash using SHA512.

utf 8, You can get Url-encoded string (for win-1251) with help of simple modification of code from accepted answer in Encoding conversation UTF-8 to 1251 in  I need to convert a string from Windows-1251 to UTF-8. I tried to do this with iconv, but all I get is something like this: пїЅпїЅпїЅпїЅпїЅ

When you access string in javascript, its not even UTF-8 encoding anymore (browsers automatically decode it to built-in javascript encoding OR the one you specified when you load javascript or text, OR default browser encoding), so you need to convert it to UTF-8 first and then convert to cp1251.

Take a look at this unicode encoding converter, I hope you'll get some ideas from that.

How to convert encoding from windows-1251 to UTF-8 in node.js?, Here the problem is not with conversion, and output utf-8 text from nodes on Windows: https://github.com/nodejs/node-v0.x-archive/issues/2190 In reality, those are windows-1252 encoded string that were mis-interpreted as UTF-8, and as such they get mapped to the Unicode Latin-1 Supplement Block. Luckily, characters from 0080 to 009F, spanning the whole windows-1252 encoding, are non-printable in Unicode, so it's perfectly safe to assume those are

If you should guess encoding, for Russian text you can use this

[PDF] JS Character Encodings, Windows-1251 (Cyrillic) … … … … 0xD0. Р UTF-8. Variable-length encoding with single-byte code units: U+0000 - U+007F: 0xxxxxxx. U+0080 - U+07FF: JavaScript does not perform any conversion of strings into bytes. ○ The underlying  toUTF8Array: Javascript function for encoding a string in UTF8. - toUTF8Array.js

In case if someone needs to put encoded string to url modify code from previous answer to

var DMap = {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48, 49: 49, 50: 50, 51: 51, 52: 52, 53: 53, 54: 54, 55: 55, 56: 56, 57: 57, 58: 58, 59: 59, 60: 60, 61: 61, 62: 62, 63: 63, 64: 64, 65: 65, 66: 66, 67: 67, 68: 68, 69: 69, 70: 70, 71: 71, 72: 72, 73: 73, 74: 74, 75: 75, 76: 76, 77: 77, 78: 78, 79: 79, 80: 80, 81: 81, 82: 82, 83: 83, 84: 84, 85: 85, 86: 86, 87: 87, 88: 88, 89: 89, 90: 90, 91: 91, 92: 92, 93: 93, 94: 94, 95: 95, 96: 96, 97: 97, 98: 98, 99: 99, 100: 100, 101: 101, 102: 102, 103: 103, 104: 104, 105: 105, 106: 106, 107: 107, 108: 108, 109: 109, 110: 110, 111: 111, 112: 112, 113: 113, 114: 114, 115: 115, 116: 116, 117: 117, 118: 118, 119: 119, 120: 120, 121: 121, 122: 122, 123: 123, 124: 124, 125: 125, 126: 126, 127: 127, 1027: 129, 8225: 135, 1046: 198, 8222: 132, 1047: 199, 1168: 165, 1048: 200, 1113: 154, 1049: 201, 1045: 197, 1050: 202, 1028: 170, 160: 160, 1040: 192, 1051: 203, 164: 164, 166: 166, 167: 167, 169: 169, 171: 171, 172: 172, 173: 173, 174: 174, 1053: 205, 176: 176, 177: 177, 1114: 156, 181: 181, 182: 182, 183: 183, 8221: 148, 187: 187, 1029: 189, 1056: 208, 1057: 209, 1058: 210, 8364: 136, 1112: 188, 1115: 158, 1059: 211, 1060: 212, 1030: 178, 1061: 213, 1062: 214, 1063: 215, 1116: 157, 1064: 216, 1065: 217, 1031: 175, 1066: 218, 1067: 219, 1068: 220, 1069: 221, 1070: 222, 1032: 163, 8226: 149, 1071: 223, 1072: 224, 8482: 153, 1073: 225, 8240: 137, 1118: 162, 1074: 226, 1110: 179, 8230: 133, 1075: 227, 1033: 138, 1076: 228, 1077: 229, 8211: 150, 1078: 230, 1119: 159, 1079: 231, 1042: 194, 1080: 232, 1034: 140, 1025: 168, 1081: 233, 1082: 234, 8212: 151, 1083: 235, 1169: 180, 1084: 236, 1052: 204, 1085: 237, 1035: 142, 1086: 238, 1087: 239, 1088: 240, 1089: 241, 1090: 242, 1036: 141, 1041: 193, 1091: 243, 1092: 244, 8224: 134, 1093: 245, 8470: 185, 1094: 246, 1054: 206, 1095: 247, 1096: 248, 8249: 139, 1097: 249, 1098: 250, 1044: 196, 1099: 251, 1111: 191, 1055: 207, 1100: 252, 1038: 161, 8220: 147, 1101: 253, 8250: 155, 1102: 254, 8216: 145, 1103: 255, 1043: 195, 1105: 184, 1039: 143, 1026: 128, 1106: 144, 8218: 130, 1107: 131, 8217: 146, 1108: 186, 1109: 190}
function UnicodeToWin1251(s) {
    var L = []
    for (var i=0; i<s.length; i++) {
        var ord = s.charCodeAt(i)
        if (!(ord in DMap))
            throw "Character "+s.charAt(i)+" isn't supported by win1251!"
        L.push('%' + DMap[ord].toString(16));
    }
    return L.join('')
}

ashtuchkin/iconv-lite: Convert character encodings in pure , Pure JS character encoding conversion Intuitive encode/decode API; Streaming support for Node v0.10+; [Deprecated] Can extend Node.js primitives (​buffers,  When you access string in javascript, its not even UTF-8 encoding anymore (browsers automatically decode it to built-in javascript encoding OR the one you specified when you load javascript or text, OR default browser encoding), so you need to convert it to UTF-8 first and then convert to cp1251.

iconv-lite: Docs, Tutorials, Alternatives, Pure JS character encoding conversion. Doesn't need native streaming example fs.createReadStream('file-in-win1251.txt') .pipe(iconv. Supported encodings. All node.js native encodings: utf8, ucs2 / utf16-le, ascii, binary, base64, hex. Online charset / codepage converter. This page is currently viewed using utf-8 codepage. If you wish to directly copy-paste text into the below form please switch to the proper charset first. When uploading a file, this step is not needed. If you encounter any errors with this tool please drop me a line attaching the file that causes problems.

iconv, Text recoding in JavaScript for fun and profit! Encode from one character encoding to another: // convert from UTF-8 to ISO-8859-1 var Buffer  An online, on-the-fly UTF-8 encoder/decoder. About this tool. This tool uses utf8.js to UTF-8-encode any string you enter in the ‘decoded’ field, or to decode any UTF-8-encoded string you enter in the ‘encoded’ field.

Package, Pure JS character encoding conversion. Doesn't need native All node.js native encodings: utf8, ucs2 / utf16-le, ascii, binary, base64, hex. Additional unicode encode('win1251') ~96 Mb/s ~320 Mb/s decode('win1251') ~95 Mb/s ~246 Mb/s  Client browser handles the data from the source form as a string data encoded by document charset (windows-1251 in the case of this document) and sends the data as a binary http stream to a web server. You can choose another character set for the conversion of the source text data (the textarea).

Comments
  • You might want to make sure characters outside the 255 range of windows-1251 (e.g. Japanese characters etc) not be allowed in utf8_decode as it could corrupt the hash on the server side.
  • e.g. bb += aa.charAt(i);} if (bb.charCodeAt(i) > 255) throw "Character "+aa.charAt(i)+" can't be encoded using 1251!"
  • This is about what I was looking for. But I found another implementation. Anyway - thanks a lot :)
  • Do you have function to do reverse conversion? From windows-1251 to Unicode?