[Feature Request] [stdlib] [proposal] Unicode Indexing for `String` and buffer idx for `StringSlice` in `__getitem__` and `__setitem__` about mojo HOT 3 OPEN

(As discussed internally with @connorg):

This is a bit tricky. I see and agree with the argument, but I do worry about being inconsistent between these two fundamental types. If folks get used to codepoint-based indexing with String and then start to use StringSlice, the could easily be confused and write incorrect code without realizing it.

For folks who want to work on raw bytes, they can use StringSlice.as_bytes_slice()[idx] to be explicit about it.

from mojo.

they can use StringSlice.as_bytes_slice()[idx] to be explicit about it

you're right, totally forgot that existed

I'll leave the issue open until we get the utf32 indexing done. If somebody else wants to tackle it be my guest. I was thinking if nobody takes it on by this weekend to give it a go and try to setup something faster for counting utf8 continuation bytes with SIMD since it is used in other places.

from mojo.

@JoeLoser opened PR #3295 but there are some issues with other functions such as String.format, String.rfind, and some others may have an issue but might not be well tested. I'm worried that too many places assume string is indexed by byte offset directly.

I think it might be worth it to rethink if we really want utf32 indexing in a utf8 encoded String. It might be better to just build a PythonString that uses utf32. Or to just let indexing be by byte offset and provide an explicit method for python style indexing (this is my favorite one tbh.).

from mojo.