Suggested by Roman K https://dfinity.slack.com/archives/CPL67E7MX/p1595078968092100?thread_ts=1595078968.092100&cid=CPL67E7MX

Our collection of Hash functions is incomplete or partial defined (#187).

We should provide decent hash functions for all primitive types.

It would be good for all of the base *.mo modules to follow a consistent pattern. I think it would be nice to use a format like this:

• Module-name

• Module-description: At least one sentence but maybe more to describe the module -- maybe even highlighting a few of the most common or unique use cases.

• Function-name

  • Function-description: At least one sentence but maybe more to describe the function purpose -- maybe even highlighting a few of the most common or unique use cases.

  • Function definition (I'm not sure what the proper terminology for this is)

  • Function example (I think this could be the unit test code snippet)

  • Additional remarks/notes, if applicable

I couldn't figure out how to add these additions to the existing code given the current script, but I'm thinking of something like this (warning: fictional content!):

/**
[#mod-Char]
= Char

The `Char` module provides functions for manipulating single character data types in your program.
The module includes functions to evaluate `char` data. 
*/

import Prim "mo:prim";
module {
  public let isDigit : Char -> Bool = func(char) {
    Prim.charToWord32(char) - Prim.charToWord32('0') <= (9 : Word32)
  };
/* 
Evaluates a character and returns True if the 
character is a numeric digit, or False if it not.
*/
}
/* 
Insert isDigit test / example code 
*/

I'm happy to take a stab at the first part with the module name and description, but I'm not sure how/where to put the additional sections that I think would make the library feel more complete.

Currently, range has type (Nat, Int) -> Iter. The Int was introduced by @chenyan-dfinity to safely support patterns like range(0, array.size()-1).

Instead, we should provide variants of range and revRange that interpret the bounds as an open interval on the max end. We probably shouldn't change range itself, though, since it is too natural to (mis)interpret range(0, 10) as a closed interval, which could lead to bugs if it wasn't.

Any ideas for a nice suggestive naming scheme for open range functions?

It just contains a bunch of trapping functions now. Call it Assert.mo? And change motoko doc to match. Or merge into Debug?

This tripped up @hansl during his livestream and he had to resort to using the prelude.

Language manual also needs updating since it says use charToText (sans qualification).

I see this is a limitation of the current version, but any idea when it will be fixed?

Text includes the following methods which present internationalization problems:

  /// Returns `t1 == t2`.
  public func equal(t1 : Text, t2 : Text) : Bool { t1 == t2 };

  /// Returns `t1 != t2`.
  public func notEqual(t1 : Text, t2 : Text) : Bool { t1 != t2 };

  /// Returns `t1 < t2`.
  public func less(t1 : Text, t2 : Text) : Bool { t1 < t2 };

  /// Returns `t1 <= t2`.
  public func lessOrEqual(t1 : Text, t2 : Text) : Bool { t1 <= t2 };

  /// Returns `t1 > t2`.
  public func greater(t1 : Text, t2 : Text) : Bool { t1 > t2 };

  /// Returns `t1 >= t2`.
  public func greaterOrEqual(t1 : Text, t2 : Text) : Bool { t1 >= t2 };

  /// Returns the order of `t1` and `t1`.
  public func compare(t1 : Text, t2 : Text) : { #less; #equal; #greater } {

The methods equal and notEqual only compare unicode values. Two strings may be equivalent after Unicode normalization, but these methods will not be able to tell.

The methods less, lessOrEqual, greater, and greaterOrEqual are locale-dependent. For example in English, n < y because n comes before y in the alphabet. But in Lithuanian, the y comes right after the i, so y < n. https://en.wikipedia.org/wiki/Lithuanian_orthography If you really want to get crazy, try comparing strings in Japanese or Chinese. Unicode order is completely unrelated to what they expect out of a sorted list of strings. And just to be complicated, there are different ways of doing sorting for the ideographic characters. (stroke-and-radical, pinyin, romanization, person names, etc.) Even in English, there are problems with it. Think accented characters like in the British spelling of "naïve". When sorting by Unicode value, "naïve" comes after "naysayer".

The very presence of any of these methods will encourage programmers to do the wrong thing and make sorting algorithms that only work based on Unicode value, which does not work in many languages. And then people like me have to come around years later and fix it.

I would recommend keeping equal and notEqual and making the documentation VERY clear that it is about Unicode equivalence only for comparing non-user-visible strings like URIs or font names or what have you. Then, there should be a Collator class that takes a locale and compares strings in a locale-dependent fashion. I would also recommend removing/deprecating all of the other comparison functions, because they don't work correctly for many languages, including English, and just encourage programmers to do the wrong thing.

Classes should preferably be named after their interface, not their implementation. That also helps programmers who don't necessarily have a background in data structures.

Not sure if this is the right place for this; I couldn't find dfx on github.

I've gotten the following error a few times when running dfx build. It would be nice if the error listed what files weren't found so I can more easily address the problem. As it is, unless you're building very often and happen to catch the missing file as you add a refrence somewhere, it's hard to track down exactly what's missing.

The error is as follows:

$ dfx build
Building canisters...
Building frontend...
Build failed. Reason:
  Postbuild step failed for canister cxeji-wacaa-aaaaa-aaaaa-aaaaa-aaaaa-aaaaa-q with error: An error occured:
Io(
    Os {
        code: 2,
        kind: NotFound,
        message: "No such file or directory",
    },
)

Finish writing at least one line of description for each method in the module or class below:

• Buf (rename to Buffer?)
• Array -- #138
• Deque -- #132
• Hashmap
• RBTree -- #140
• Trie -- #130
• TrieMap -- #128
• TrieSet -- #139
• AssocList -- #141

cc @crusso @kritzcreek @rossberg

This seems broken to me.

Most of this is already implemented in the prelude so its just a matter of exposing it properly, I think.

@rossberg, @kritzcreek sound sensible?

Based on questions from users, it is not clear when AssocList's should be used, e.g., over ordered maps. Arguably there are almost no cases where they should. Yet their existence may mistake programmers into thinking that their use is adequate when it isn't.

These almost certainly shouldn't appear in production code. @matthewhammer, can you take a look?

Building help using the generated adoc files issues a wanting.
asciidoctor: WARNING: Iter.adoc: line 72: id assigned to section already in use: value.next

It looks like there is a value.next for Nat and value.next for Int. I can modify the adoc to get rid of the error, but it would be nice to fix this in the source. It wasn't obvious to me what was generating these section anchors.

https://forum.dfinity.org/t/how-to-handle-optional-types-in-motoko/1213/3
from morrolan:
the documentation of the Option module seem to have an error, it advocates the following pattern:

let int1orZero : Int = switch(optionalInt1) {
case null 0;
case ?(int) int;
};
Ie ?(int) instead of (?int) - which causes a syntax error.

This should throw a compiler error:

var ok = false;
func f(): () {
  ok := true
};
if (false) { P.unreachable(); } // no semicolon
f(); // this never gets called
assert(ok)

Instead, f is not called and the assertion fails.

As the title says, the height and size methods for the tree return 0 regardless of its actual structure.

From Tungsten Hackathon:

Do we have some basic Text/String processing functions? Like split string by linebreak?

The combining function takes the accumulator in different positions.

I need a compare function for Text that is similar to what we have for Nat.
https://github.com/dfinity-lab/motoko-base/blob/6b6f7f217f245e3df17b9706ae07c737339db1f4/src/Nat.mo#L53

Currently go via Int.abs.

The Interface design doc mentions trapping briefly, but not directly. Currently, it gives this parenthetical, when describing what get should do to a container-like object:

get: read from container (usually returns option, except for arrays)

This seems clear enough at first, except that for Buffers, we would like two things at once, which conflict:

1. Use Buf as a drop-in replacement for a mutable array (that does not grow)
2. Follow the guide, and use optional return

Currently, we break from the guide in favor of rationale 1. Is that okay?

To answer such questions with more clarity and consistency, we should expand this parenthetical to a full (sub)section, where we prescribe trap behavior explicitly (either permitting it for no cases, for no cases except explicit, listed Array operations, or some other prescription that permits a non-optional get from a Buffer, etc.).

cc @crusso @kritzcreek @rossberg

Do we really want that inefficiency built in?@rossberg?

Hi all! Hope everything has been going well, I've recently started building stuff with the SDK again and had a quick question.

I have two files:

// B.mo
actor B {
  public func get(): async Blob { "Hello" };
}

// A.mo
import B "canister:B";

type A = actor {
  get: () -> async Blob;
};

actor {
  let a: A = B;
}

This produces an error:

type error, expression of type
  actor {get : shared () -> async [Nat8]}
cannot produce expected type
  actor {get : shared () -> async Blob}

Is this expected? I can change A to use [Nat8], but I'm trying to keep the types the same. Thanks!

to bitShiftRight 31

These happen frequently in the Random.mo code.

Note: this issue should be in motoko proper, but it ended up here by mistake. I cannot move it there.

see rts/float.c

but beware stackoverflow...

Special members like a.keys(), blob.bytes(), text.char() are not very discoverable unless you RTFM. Shall we add eponymous functions to the various libs?

If so, how do we distinguish a.keys() for mutable and immutable arrays. I'm tempted to separate Vector and and Array libs instead of naming each overloaded operation apart...

@dfinity-lab/languages thoughts?

Primitives (Motoko):

• array.len -> array.size
• array.set -> array.put
• text.len -> text.size

Array:

• equals -> equal
• apply: switch param order
• bind -> chain
• enumerate: remove in favour of entries
• foldl -> foldLeft, switch param order
• foldr -> foldRight, switch param order
• find: switch param order
• join -> flatten
• map -> transform, switch param order
• mapWithIndex -> mapEntries, switch param order
• pure -> make
• vals: add
• keys: add

Buf:

• len -> size
• iter -> vals
• set -> put

Deque:

• removeFront -> popFront
• removeBack -> popBack

Function:

• rename to Func

Hash:

• hashOfInt -> Int.hash
• hashOfIntAcc -> Int.hashAcc (is this general utility enough to be in the lib?)
• hashOfText -> Text.hash
• hashEq -> equal
• getHashBit -> bit
• bitsPrintRev -> debugPrintBitsRev
• hashPrintRev -> debugPrintBits
• hashWord8s -> Array.hashWord8

HashMap:

• count -> size
• set -> put
• swap -> replace
• iter -> entries
• map -> transform
• mapFilter -> transformFilter

Heap:

• add -> put
• removeMin -> deleteMin

Iter:

• forIn -> apply, switch param order
• length -> size
• map -> `transform, switch param order
• pure -> make
• toListWithLength -> toListWithSize (is this needed?)

List:

• len -> size
• lenIsEqLessThan: do we need this in the lib? (at least replace with compareLengthWith, cf OCaml)
• lenClamp: dito?
• nth -> get
• rev -> reverse
• iter -> apply
• map -> transform
• mapFilter -> transformFilter
• concat -> flatten
• revAppend -> reverseAppend
• exists -> some
• lessThanEq: replace with compare
• isEq -> equal
• singleton -> make
• split -> partition
• splitAt -> split
• chunksOf -> chunks
• fromArrayMut -> fromVarArray
• toArrayMut -> toVarArray

Nat:

• hash: add

None:

• absurd -> impossible

Option:

• unwrapOr -> get
• option -> getTransform, switch param order
• map -> transform, switch param order
• apply: switch param order
• bind -> chain
• join -> flatten
• pure -> make
• assertSome: remove?
• assertNull: remove?

Ord:

• rename to Order
• Ordering -> Order, also rename tags
• isLT -> isLess
• isEQ -> isEqual
• isGT -> isGreater

Prelude:

• printLn -> debugPrintLine

Principal:

• hash: should return Hash.Hash

RBTree:

• getTree -> share
• find -> get
• insert -> put
• delete: add function
• iter -> entries
• rev -> entriesRev
• toIter -> iter, change direction tags to #fwd, #bwd

Result:

• bind -> chain
• mapOk -> transformOk
• fromSomeMap -> fromSomeTransform (or remove?)
• joinArrayIfOk -> toArrayOk
• assertUnwrap -> unwrapOk
• assertUnwrapAny: remove (should be redundant)
• assertErrAs: replace with unwrapErr
• assertErrIs: remove

Trie:

• should we split this module, e.g, with Trie2D, Trie3D separate?
• keyEq -> equalKey
• count -> size
• copy -> clone
• insert -> put
• exists -> some
• forAll -> all
• mapFilter -> mapTransform
• insertFresh -> putFresh
• insert2D -> put2D
• insert3D -> put3D

Trie.Build:

• TrieBuild -> Build
• buildCount -> size
• buildSeq -> seq
• prodBuild -> prod
• buildNth -> nth
• projectInnerBuild -> projectInner
• buildToArray -> toArray
• buildtoArray2 -> toArray (do we need both?)

TrieMap:

• count -> size
• put: add
• set -> replace
• del -> delete
• iter -> entries
• map -> transform
• mapFilter -> transformFilter

TrieSet:

• insert -> put
• remove -> delete
• eq -> equal
• card -> size
• iter -> entries
• map -> transform
• mapFilter -> transformFilter
• unitEq -> remove (or create a Tuple module?)

Not that you say it: Absolutely, Int.range, Nat.range etc. make much more sense.

Originally posted by @nomeata in #51

• isPunctuation ( table driven, needs a prim)
• isNumeric (table driven, needs a prim)
• isAlphaNumeric (derived)
• isHexDigit (or, more generally) isRadixDigit(radix))
• multichar toLower
• multichar toUpper.

Perhaps some others (cf. Rust Char type).

To avoid cylic definitions, requires a cycle breaking Types.mo (useful anyway) or motoko's special mo:prim to export all primitive types under their own name.

We have List.mapFilter and Array.filterMap right now, which we should unify. Once we've decided on a name we should also add it to docs/design.md.

This issue is tracking the answer to the question: How does one write what could be an SML/OCaml-style functor in Motoko, using the current language features?

As one example (WIP): Adapton functor from the Adapton package.

I've done some research and can't find anything, but are there any plans for releasing an SDK for Windows?

I think we need a style guide for base library. There are already some inconsistencies in the existing modules.

Several items I would like to discuss here:

• Do we want to provide an OO style interface for all containers?
  List doesn't have a class interface. Trie provides a class interface in a separate file TrieMap.
• For the modules that have class interface, which methods goes inside the class, and which stays outside? Is there a cost for having too many methods inside a module/class?
• How do we handle containers that require "type class", e.g. Hash, Ord, Eq.
  We use public class Container<T>(ord : (T, T) -> Bool) for taking the ord relation. Does this mean that all the methods should stay inside the class. Because methods outside of the class will need to take both ord and Container as arguments, and this can lead to inconsistency.
• Module naming convention. Each module usually has a type and a class constructor, e.g. type Container<T> = ...; public class MakeContainer<T>(eq). The usual convention is to use t for value type and mkContainer for constructor. But t<T> looks odd, and we usually name the constructor as a type name.
  The code then looks like let buf : Buf.Buf<Int> = Buf.Buf<Int>(10). Does let buf : Buf.t<Int> = Buf.mkBuf<Int>(10) feel better?
• Method naming convention. We need a consistent naming convention and type signature for common ADT methods, e.g. add vs insert; remove vs delete; does add return a value; does remove trap for empty container; does remove return the removed elements; etc.
• Multiple implementation for the same interface. Do we have a preference for imperative vs functional implementation, or we may want both in some cases? How about different data structures for the same interface, e.g. red-black tree vs splay tree. Do we want an interface file for each container, i.e. an object type that lists all method signatures.
• Guidelines about testing and documentation. What needs to go in the comments/documentation, e.g. time/space complexity. Can we have a testing framework that works on all container modules.

In the Hackathon I repeatedly saw code like

for (x in List.toArray(xs).vals()) ...

which of course defeats the purpose of using lists. So we should have iterators for lists.

Another observation from the hackathon is that folks need to split and slice strings, and we do not provide any way to do that other than converting to an array.

I propose:

Text.slice(begin : Iter, end : ?Iter) : Text

returns the text consisting of all characters between the begin iterator (inclusively) and the end iterator (exclusively). If end is null then the end of the text is used.

For this purpose, the character pointed to by an iterator is the one last returned by next. If that is null, it points to the end of the string.

Throws if begin or end are not text iterators, not iterators on equal text values, or iterators for which next has not yet been invoked.

The initial limbo state of an iterator is rather unfortunate in this context. Should we reconsider our iterator type?

A module to handle decimals would be super useful for financial or scientific applications. I'm currently representing decimals as n * 1e18 but that's not really ideal.

TIL Null implements == and !=, but why bother? (o == null does not work for options o : ? T, only for o : null.