btoa('café') gives you the wrong bytes, and nothing tells you
Run this in your browser console: btoa('café'); // 'Y2Fm6Q==' Now encode the same word as UTF-8 first: btoa(String.fromCharCode(...new TextEncoder().encode('café'))); // 'Y2Fmw6k=' Same word, two different answers. Both are valid base64. Neither one throws. Only the second is what everyone else means by "café in base64". Why this happens btoa and atob are old. They work on "binary strings", where every character has to fit into one byte. Anything above U+00FF has no byte to go into, so btoa throws. Anything at or below U+00FF gets used as its own byte value. That is latin-1, not UTF-8. Take é. It is U+00E9. In latin-1 that is one byte, 0xE9. In UTF-8 it is two bytes, 0xC3 0xA9. btoa picks the latin-1 one. Everything else expects the UTF-8 one. See it yourself Paste this in once and you get all four cases: const show = (text) => { let naive; try { naive = btoa(text); } catch (e) { naive = 'THROWS ' + e.name; } const correct = btoa(String.fromCharCode(...new TextEncoder().encode(text))); console.log({ text, naive, correct, back: atob(correct) }); }; ['José 🎉', '日本語テキスト', 'Grüße aus München', 'café'].forEach(show); Here is what comes back. The console prints the broken text with the control characters escaped, like \x9f, which is a hint at the real problem. More on that below. Input btoa(text) atob(correct base64) José 🎉 throws InvalidCharacterError José ð\x9f\x8e\x89 日本語テキスト throws InvalidCharacterError æ\x97¥æ\x9c¬èª\x9eã\x83\x86... Grüße aus München R3L832UgYXVzIE38bmNoZW4= GrüÃ\x9fe aus München café Y2Fm6Q== café Three things happen, not one Encoding is loud. Anything above U+00FF throws. Emoji and CJK break straight away. That is the good case. You find out. Decoding is quiet. Run atob on proper UTF-8 base64 and you get mojibake back, with no error at all. This is the one that hurts, and it is the common one, because the base64 usually came from somewhere else. A backend, a JWT, another tool. Plain accented text survives. btoa('café') then atob gives you back café, exactly. é fits in one byte, so a broken tool talking to itself looks completely fine. That last one is why this sits in code for years. Encode and decode in the same place and the test passes. So does clicking around by hand. You need two implementations before anything looks wrong, and by then the bytes are already in a database. What the mojibake hides The broken output has C1 control characters in it, between 0x80 and 0x9f. They do not render at all. José 🎉 comes back as ten characters. You see seven. So if you copy that into a bug report to show what went wrong, the invisible ones get dropped along the way, and it looks less broken than it is. It hides from the exact thing you would use to find it. The fix Stop handing text to btoa. Turn it into bytes yourself, and be strict on the way back. const UTF8_ENCODE = new TextEncoder(); const UTF8_DECODE = new TextDecoder('utf-8', { fatal: true }); function encodeBase64(text) { const bytes = UTF8_ENCODE.encode(text); const CHUNK = 0x8000; let binary = ''; for (let i = 0; i < bytes.length; i += CHUNK) { binary += String.fromCharCode.apply(null, bytes.subarray(i, i + CHUNK)); } return btoa(binary); } function decodeBase64(b64) { const bytes = Uint8Array.from(atob(b64), (ch) => ch.charCodeAt(0)); return UTF8_DECODE.decode(bytes); } Three things in there matter. fatal: true. Without it, TextDecoder swaps bad bytes for U+FFFD and hands you a string anyway. That puts you back where you started: a wrong answer and no error. With it, bad input throws, so you can tell "this is not text" from "this is text and I broke it". The chunked loop. Do not write String.fromCharCode(...bytes). Spreading a big array into arguments blows the stack. I tested it: 100,000 was fine, 125,000 threw Maximum call stack size exceeded. The limit moves between engines, so do not tune to that number. Just do not spread. Lone surrogates. TextEncoder swaps an unpaired surrogate for U+FFFD instead of failing, so your input changes before you have encoded anything. Check first if you care: const LONE_SURROGATE = /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?
This is a summary aggregated from Dev.to. Read the complete article on the original site:
Read full article at Dev.to