Extended KotD #19: _UCHARPOS

[SMcNeill]

Okies guys, this is one of those keywords that I've kinda been dreading to get to have to cover.  

Why??

Because the *concept* behind this one and what it does for us is rather complex to describe fully.  I have a feeling this is going to be a long arse post, so I'll probably end up wrapping it into spoiler tags so folks can navigate it a little easier, but grab yourself a soda and a snack before delving into this topic. 

INTRO

Show Content

Spoiler
To start with, let me sidestep this whole command for a moment, and just step sideways and talk about Unicode itself.  (If you've paid attention to the last several keywords, you've probably noticed that they've all be leading with _U, and have been all related to unicode printing.  This command works in the same vein for us -- it's unicode related.

Now, all of you should be fairly familiar with ASCII and ANSI characters.  These are the CHR$(0) to CHR$(255) characters that have been built into the language forever and ever and ever.  Each value from 0 to 255 represents one single character which we can print to the screen.  It's a very simple system, easy to understand, and one which nobody ever has any issues with.

CHR$(65)  <--- this value represents the ASCII captial character "A".
CHR$(66)  <--- this value represents the ASCII captial character "B".

Easy to use and fairly easy to understand.   When looking at a string of data, each byte represents one single character which we'd print to the screen.

Yaaay  ASCII!!   The really is about as BASIC as one can get when needing to print to the screen!!

---

Unicode...  Unicode, unfortunately, doesn't fit this philosophy. 

Unicode wasn't created to just work with US Keyboards, in English, with more-or-less just "A"-to-"Z" represenentation. 

Unicode was created so that it could work with ALL keyboard layouts, ALL languages, and all character sets that someone might want to be able to display onto their screen.

About the closest to ASCII that one can come, is one specific subset of Unicode -- UTF32.  Unicode currently has something insane like about 150,000 characters to be used with it...  And a single byte?  That only holds values from 0 to 255.  Two bytes, such as for an integer?  That only holds values up to 65535...

UTF32, has to use a LONG-type to hold the value for all those possible combinations, resulting in 4-bytes per character.

ASCII uses 1-byte per character to allow you to have 0 to 255 possible characters to use and PRINT.  UTF32 uses 4-byte per character, and allows you to use the whole range of unicode characters which are currently recognized as being valid for use.

So, at it's basic underlying concept, that's all unicode is -- it's just a larger character set which holds all the possible characters that have been formally recognized for use with text.  All those accented characters.  All the Chinese, Greek, Hebrew, Japenese characters.  Even the Emoji that everyone seems to love to hate...  All of those characters are in that one single dataset.

UTF-8 vs UTF-16 vs UTF-32

Show Content

Spoiler
And with the above intro out of the way, let's talk about some of the various FORMATS which can be used with that extended unicode character set.  The three that I'll go over briefly here are:

UTF-8 format:  UTF8 is a lot like ASCII in that each byte has special meaning to it.  I'll go into detail about that below.

UTF-16 format:  UTF16 has two-byte formatting.  I'm going to more or less ignore it, to keep things from going too long here.

UTF-32 format:  UTF32 is 4-byte formatting.  It's actually the simplest to understand, as I explained above.  Each character takes 4-bytes to represent it.  Who can't understand that UnicodeChr$(12345) is the smiley face?  And that UnicdoeChr$(23456) is the sunglass face 8) ?  One value = one character.  Even a monkey can figure this one out. 

---

So, with that basically said, the only one I'm going to go into detail with is UTF-8 here.

UTF-8 is basically the bastard child of an insane scientist who had drunk too much coffee and was working too late into the night.  They basically said, "Man, it takes a long time to send text under the new unicode format on my 2600baud router!  And I don't want to have to compress it before sending it...  I'm going to come up with a way to use unicode, and still only send a single-byte at a time!!

So what they came up with was this brilliant concept:

First -- The vast maority of text is simple "A" to "Z", "0" to "9", ANSI-style characters.  This insane code standardization guru decided, "Hey, let's leave them alone.  IF some just needs to send simple text messages like, 'Honey, I'm going to be late coming coming tonight,' then they can just send those with one byte representing one character."

Then they thought, "Now, how am I going to get my other 149,872 chracters, without screwing up those special 128 characters?"

And, after adding a few more swigs of vodka to their coffee, they came up with:

"I know!  I'm going to make the ascii values greater than 127 'special page' references!!"

And so, that's what they did.  LOL!

The first byte you see is basically the control-byte of an UTF-8 character.

If the ASCII value of that byte is from 0 to 127, it corresponds to the ANSI/ASCII character from 0 to 127.
If the ASCII value of that byte is from 128 to 223, then you take the value of TWO bytes to make that character.
If the ASCII value of that byte is from 224 to 239, then you take the value of THREE bytes to make that character.
If the ASCII value of that byte is from 120 to 248, then you take the value of FOUR bytes to make that character.
If the ASCII value of that byte is from 249 to 253, then you take the value of FIVE bytes to make that character.
If the ASCII value of that byte is 254 or 255...

...you add more vodka to your coffee and cry.  As far as I know, those 2 byte values aren't defined at all...

So with UTF-8, your character can be represented by 1-byte.  Or 2-bytes.  Or 5-bytes...

A string of text, which is 25-bytes in size, may be 25 characters.  Or it may be 5 characters.  Or, it may even be less than that! 

And you say, "How the FLIP is that possible??  The control byte only allocates for a max of 5-bytes to a character!!"

And... I'll put that in the next spoiler. 

CODE POINTS vs GLYPHS

Show Content

Spoiler
Now, as I mentioned above, UTF-8 has variable-length data associated with it.  A character might be 1, 2, 3, 4, or 5 bytes in size...

Truth is, I lied to you guys, to keep it kind of simple up to this point. 

UTF-8 doesn't have 1, 2, 3, 4, 5 bytes PER CHARACTER.  Those are 1, 2, 3, 4, or 5-bytes PER CODEPOINT.

Now, each one of those code points is going to be something unique in our unicode dataset of characters.  By themselves, as far as I know, each of these code points should make their own character. 

UnicodeChr$(1234) might be the ":" colon symbol.  (It's not, but I'm lazy and don't want to look up actual values.  Pretend it is, for illustration's sake.

UnicodeChr$(2345) might be the "(" left-parenthesis symbol.  (It's not, but read above. )

Now, the point I'm trying to make here is:
":" + "(" =

Colon plue left-parenthesis = Sad Smiley Face

---

These two code points combine to form one GLYPH (character), which we print to the screen.

By itself UnicodeCHR$(1234) is the colon.  ":"
By itself UnicodeCHR$(2345) is the left-parenthesis.  "("

One following the other gives us the sad smiley face.  ""

And THAT gets us to the point where _UCharPos comes into existence for us!!

As I explained above, a string of unicode data may use 1, 2, 3, 4, 5 bytes to generate a codepoint.  And then it might take multiple codepoints and merge them together to make a final glyph/character, before printing it to the screen.

So a string of 20-bytes, in UTF-8 format, might be 20 characters.  Or 19 characters.  Or it might be only 2.   Heck, it might even be formatted wrong and not be any!!

UGHH!!!

So how the BLEEP would someone know how many characters an unicode string has??  How would you underline the word "FOO" in bight red, if you can't even know how many characters are before it, or after it??   

HOW THE FLIP DOES ANYONE DO ANYTHING MUCH AT ALL WITH UTF-8 FORMATTED CRAP???

UGGGHHHHHH!!!

---

Have no fear, _UCharPos is here!!    

(to be continued in the next post below this one, as the forum has text limits and I don't want to write and write and write, just to have it cut me off or lose my work)