Extended KotD #19: _UCHARPOS

[SMcNeill]

Okies guys, this is one of those keywords that I've kinda been dreading to get to have to cover.  

Why??

Because the *concept* behind this one and what it does for us is rather complex to describe fully.  I have a feeling this is going to be a long arse post, so I'll probably end up wrapping it into spoiler tags so folks can navigate it a little easier, but grab yourself a soda and a snack before delving into this topic. 

INTRO

Show Content

Spoiler
To start with, let me sidestep this whole command for a moment, and just step sideways and talk about Unicode itself.  (If you've paid attention to the last several keywords, you've probably noticed that they've all be leading with _U, and have been all related to unicode printing.  This command works in the same vein for us -- it's unicode related.

Now, all of you should be fairly familiar with ASCII and ANSI characters.  These are the CHR$(0) to CHR$(255) characters that have been built into the language forever and ever and ever.  Each value from 0 to 255 represents one single character which we can print to the screen.  It's a very simple system, easy to understand, and one which nobody ever has any issues with.

CHR$(65)  <--- this value represents the ASCII captial character "A".
CHR$(66)  <--- this value represents the ASCII captial character "B".

Easy to use and fairly easy to understand.   When looking at a string of data, each byte represents one single character which we'd print to the screen.

Yaaay  ASCII!!   The really is about as BASIC as one can get when needing to print to the screen!!

---

Unicode...  Unicode, unfortunately, doesn't fit this philosophy. 

Unicode wasn't created to just work with US Keyboards, in English, with more-or-less just "A"-to-"Z" represenentation. 

Unicode was created so that it could work with ALL keyboard layouts, ALL languages, and all character sets that someone might want to be able to display onto their screen.

About the closest to ASCII that one can come, is one specific subset of Unicode -- UTF32.  Unicode currently has something insane like about 150,000 characters to be used with it...  And a single byte?  That only holds values from 0 to 255.  Two bytes, such as for an integer?  That only holds values up to 65535...

UTF32, has to use a LONG-type to hold the value for all those possible combinations, resulting in 4-bytes per character.

ASCII uses 1-byte per character to allow you to have 0 to 255 possible characters to use and PRINT.  UTF32 uses 4-byte per character, and allows you to use the whole range of unicode characters which are currently recognized as being valid for use.

So, at it's basic underlying concept, that's all unicode is -- it's just a larger character set which holds all the possible characters that have been formally recognized for use with text.  All those accented characters.  All the Chinese, Greek, Hebrew, Japenese characters.  Even the Emoji that everyone seems to love to hate...  All of those characters are in that one single dataset.

UTF-8 vs UTF-16 vs UTF-32

Show Content

Spoiler
And with the above intro out of the way, let's talk about some of the various FORMATS which can be used with that extended unicode character set.  The three that I'll go over briefly here are:

UTF-8 format:  UTF8 is a lot like ASCII in that each byte has special meaning to it.  I'll go into detail about that below.

UTF-16 format:  UTF16 has two-byte formatting.  I'm going to more or less ignore it, to keep things from going too long here.

UTF-32 format:  UTF32 is 4-byte formatting.  It's actually the simplest to understand, as I explained above.  Each character takes 4-bytes to represent it.  Who can't understand that UnicodeChr$(12345) is the smiley face?  And that UnicdoeChr$(23456) is the sunglass face 8) ?  One value = one character.  Even a monkey can figure this one out. 

---

So, with that basically said, the only one I'm going to go into detail with is UTF-8 here.

UTF-8 is basically the bastard child of an insane scientist who had drunk too much coffee and was working too late into the night.  They basically said, "Man, it takes a long time to send text under the new unicode format on my 2600baud router!  And I don't want to have to compress it before sending it...  I'm going to come up with a way to use unicode, and still only send a single-byte at a time!!

So what they came up with was this brilliant concept:

First -- The vast maority of text is simple "A" to "Z", "0" to "9", ANSI-style characters.  This insane code standardization guru decided, "Hey, let's leave them alone.  IF some just needs to send simple text messages like, 'Honey, I'm going to be late coming coming tonight,' then they can just send those with one byte representing one character."

Then they thought, "Now, how am I going to get my other 149,872 chracters, without screwing up those special 128 characters?"

And, after adding a few more swigs of vodka to their coffee, they came up with:

"I know!  I'm going to make the ascii values greater than 127 'special page' references!!"

And so, that's what they did.  LOL!

The first byte you see is basically the control-byte of an UTF-8 character.

If the ASCII value of that byte is from 0 to 127, it corresponds to the ANSI/ASCII character from 0 to 127.
If the ASCII value of that byte is from 128 to 223, then you take the value of TWO bytes to make that character.
If the ASCII value of that byte is from 224 to 239, then you take the value of THREE bytes to make that character.
If the ASCII value of that byte is from 120 to 248, then you take the value of FOUR bytes to make that character.
If the ASCII value of that byte is from 249 to 253, then you take the value of FIVE bytes to make that character.
If the ASCII value of that byte is 254 or 255...

...you add more vodka to your coffee and cry.  As far as I know, those 2 byte values aren't defined at all...

So with UTF-8, your character can be represented by 1-byte.  Or 2-bytes.  Or 5-bytes...

A string of text, which is 25-bytes in size, may be 25 characters.  Or it may be 5 characters.  Or, it may even be less than that! 

And you say, "How the FLIP is that possible??  The control byte only allocates for a max of 5-bytes to a character!!"

And... I'll put that in the next spoiler. 

CODE POINTS vs GLYPHS

Show Content

Spoiler
Now, as I mentioned above, UTF-8 has variable-length data associated with it.  A character might be 1, 2, 3, 4, or 5 bytes in size...

Truth is, I lied to you guys, to keep it kind of simple up to this point. 

UTF-8 doesn't have 1, 2, 3, 4, 5 bytes PER CHARACTER.  Those are 1, 2, 3, 4, or 5-bytes PER CODEPOINT.

Now, each one of those code points is going to be something unique in our unicode dataset of characters.  By themselves, as far as I know, each of these code points should make their own character. 

UnicodeChr$(1234) might be the ":" colon symbol.  (It's not, but I'm lazy and don't want to look up actual values.  Pretend it is, for illustration's sake.

UnicodeChr$(2345) might be the "(" left-parenthesis symbol.  (It's not, but read above. )

Now, the point I'm trying to make here is:
":" + "(" =

Colon plue left-parenthesis = Sad Smiley Face

---

These two code points combine to form one GLYPH (character), which we print to the screen.

By itself UnicodeCHR$(1234) is the colon.  ":"
By itself UnicodeCHR$(2345) is the left-parenthesis.  "("

One following the other gives us the sad smiley face.  ""

And THAT gets us to the point where _UCharPos comes into existence for us!!

As I explained above, a string of unicode data may use 1, 2, 3, 4, 5 bytes to generate a codepoint.  And then it might take multiple codepoints and merge them together to make a final glyph/character, before printing it to the screen.

So a string of 20-bytes, in UTF-8 format, might be 20 characters.  Or 19 characters.  Or it might be only 2.   Heck, it might even be formatted wrong and not be any!!

UGHH!!!

So how the BLEEP would someone know how many characters an unicode string has??  How would you underline the word "FOO" in bight red, if you can't even know how many characters are before it, or after it??   

HOW THE FLIP DOES ANYONE DO ANYTHING MUCH AT ALL WITH UTF-8 FORMATTED CRAP???

UGGGHHHHHH!!!

---

Have no fear, _UCharPos is here!!    

(to be continued in the next post below this one, as the forum has text limits and I don't want to write and write and write, just to have it cut me off or lose my work)

[SMcNeill]

Now, with all the above covered as a sort of foreward, let's get into what _UCharPos actually does for us. 

As I mentioned previously, an UTF8 text string consisting of 25-bytes, may have anywhere from 1 to 25 characters associated with it.   So how the heck do we know how many bytes it *actually* has, and how many characters are we going to draw onto the screen?

https://qb64phoenix.com/qb64wiki/index.php/UCHARPOS

_UCharPos is the closest tool we have at the moment, and it's not 100% perfect.

WHY do I say that??

Because it only gives us the nuumber of CODE POINTS that we generate with our string -- and you remember what I mentioned about them and glyphs being different?   ":" plus ")" = ""    Two (or more) code points can be added together to generate ONE glyph/character.

From the top of my head I can think of several examples of this:
1) Emoji is probably the mose obvious for most folks -- thus my mentioning it.
2) Several of the country flag symbols are composite and made from multiple code points.
3) A whole bunch of the oriental/foreign words are composed of multiple code points.

So let me emphasize for everyone here:   _UCharPos isn't guaranteed to give you the combined GLYPH/CHARACTER count. At the moment, it only gives you the CODE POINT count.

---

And with that pointed out, let me now step back and also point out:

We don't currently do ANY code point combining in QB64PE (as of version 3.13).  

That's right...   As of now, you can't type a colon and a parenthesis and generate a smiley face in QB64PE.   

What we have is a basic TRUE TYPE FONT RENDERING LIBRARY that we use.  It simply draws the characters that we specify for it.  It doesn't do the combing of multiple code points to create composite glyphs.

So why am I even bothering to mention the difference and point them out for folks??

*Because we may eventually expand our functionality and add additional font functionality to our source.*

---

At this moment, _UCharPos will give you the number of characters thay you're printing to the screen -- but folks need to keep in mind that what it's REALLY giving you is the number of CODE POINTS that you're printing to the screen.   IF a libray is added later that does do the combining to make emojis and such, then the number of code points may not match the number of characters you print to the screen.

At the moment however, it does, as we don't have any combination code point libraries working for us.

---

So for now, _UCHARPOS can tell you how many code points (characters) your string holds.

This is the basic return value for the command, and is used as simply as:

Code: (Select All)

codepoints = _UCharPos(utf8$, , 8)

Notice that 8 in the code above?  That's *required*, if you want a correct answer, so you can tell _UCharPos how your string is formatted.

0 = ASCII
8 = UTF-8
16 = UTF-16
32 = UTF-32

If you leave that value out, the return value you're going to get is always going to be the LENGTH of the text.   May as well just use LEN(text$) for the same value there, as ASCII encoded text is always 1 code point per byte.

And with that basic return value out of the way (it returns the number of code points for us), and that 3rd paramater mentioned above (that's the 8 I was talking about in that little sample line of code), let's talk about the rest of the parameters:

codepoints& = _UCHARPOS(text$[, posArray&()][, utfEncoding&][, fontHandle&])

Now, the text$ is the text that you want to get this information back on.  I'd imagine it's going to be an unicode string -- if not, why aren't you just doing things the same way you always have, with LEN and _PRINTWIDTH or _FONTWIDTH??

That second parameter is posArray&(), and it's a LONG array.  Let's take a moment to talk about this parameter and what it does for us:

---

PosArray()

Since _UCharPos can tell us how many code points our string has, what do you guys think PosArray does here?

I'll be nice and go ahead and tell you! 

PosArray is a long array which we pass to the command, and it passes back the POSition of each character in that ARRAY.

Let me give you a simple make believe example:

ABCDEF

Now, if the above is in monospaced-font, it's simple enough to calculate these positions.   Let's say the monospace width is 10:
A starts at pixel 0 and goes to pixel 9
B starts at pixel 10 and goes to pixel 19
and so on, with each character being 10 pixels wide.  

But now, what if this wasn't a monospaced set of characters?

abcWiii

a , b, and c...  look close enough that each of them might be 10 pixels wide.
W ... this is obviously wider.  It's probably 15 pixels in width.
i  ...  these puny little i's are probably no more than 5 pixels in width.

And THAT's what the PosArray gives us --- It's an array of the starting points of those characters on the screen.

(And note, that these character positions have zilch to do with the byte-position of our text.   Remember, it can take 1 to a variable X-number bytes to make a character.)

---

Think of PosArray as basically running a for loop, and getting the _UPrintWidth for each character in your string of text.   Your text might be 20-bytes in length, only have 5 characters to it, and those characters might be 16 pixels, 8 pixels, 8 pixels, 11pixels and 13 pixels in width.

That's basically what _UCharPos can return back to you:

codepoints = _UCharPos <-- the function gives you the number of codepoints (which is currently also the number of characters, but may change if we ever expand the library to do glyph merging)

PosArray&()   <-- this LONG array that you pass it gives you the width of each actual character/codepoint in the text.

0/8/16/32 <--- this 3rd parameter tells the function what encoding it's looking for with your string.

fonthandle&   <--- and the last parameter is simply a shortcut to tell it which font you're calculating these values using.

---

Steve's honest thoughts on this command:

It's complicated.  It's messy.  BUT, unicode and UTF-8 isn't the simplest thing in the world to work with.  codepoints.  glyphs.  characters.   multiple encoding methods....  UGGHHH!!  It's enough to make your head spin!!

Unless you're just working with unicode specifically, and unless you NEED to know the number of code points and positional placement of each character in that string of text, I'd honestly just whistle merrily and forget this command even existed.

It's essential what it does for us, and we need it for working with unicode -- but only in extremely specific cases such as if you're writting unicode word wrapping routines.

For most folks, who don't tend to worry about such things, add this to your list of "Things I'm not worried about."

If you need it, it's here.  I just honestly don't think a lot of folks are going to need:

1) To know the number of unicode codepoints s they're printing to a line.
2) The x-position of each unicode codepoint that they print to that line.

And that's basically what this function gives to us.  Use it, or leave it, as your personally programming needs dictate.

(And my needs dictate that I now need a little more vodka for my coffee....)