deshaw / ksmm Goto Github PK

we need to polish the UI for objects of type {'string': 'string', 'string2':'string2', ...}

currently, we can only display the predefined variables.

Package the extension for download on pypi and conda-forge.

Sometime we use vanilla javascript alert, sometimes we use react-bootstrap messages, which could also be better.

We need to align and polish those messages.

For all the reason mentioned here, can we move the python code to be formatted with black?

Also the current screenshot have typos, IPython is uppercase I.

For now, we hide the standard react-json-form button to only display the JupyterLab buttons

https://github.com/Quansight/ksmm/blob/9d92b59f12f432dd5c78c6b9d5e511e128b01f0a/src/widget.tsx#L96-L107

The standard react-json-form button ensures validation, but the JupyterLab buttons does not.

There should be a way to call the validation and not accept the creation if the form is not valid. The validation ways are documented on https://react-jsonschema-form.readthedocs.io/en/latest/usage/validation/

I have quickly tried liveValidate attribute but it does not seem to work.

I'm guessing this comes from copy/past or examples ?

Kernel Cards that are selected need to show specific qualities to the user.

The kernel <> environment namespace is extremely confusing, and we are adding a layer of abstraction on top of this with iPython Kernels, Jupyter Kernels, and conda environments.

Therefore, the card specification will strive to simplify this by showing the key things that need to be noted about the environment:

{
jupyter_display_name : String  /* The display name that  is used from within Jupyter; a Friendly name for the kernel. */
kernel_name: String  /* The name of the kernel itself; in this case, we assume it is the conda env name. */
/*
 * Machine Spec is the object containing the attributes for machine specifications 
 */
machine_spec : {
          cores: Integer 
          mem: Integer
}
}

Then, we will also have two options on the card; "Duplicate" which will use the copy file icon,
and we will also have an edit button (the edit button discussion pending.

Above is essentially the schema for the cards to be generated properly. Further documentation to follow.

It looks like some code is left over in the python handlers and should be removed to avoid confusion.

I am going to add a custom tab for @Carreau's work that is currently open at PR #41

For this task, essentially, we need a form to set the parameters to define a user's custom template.

@mlucool Is testing the version published currently. Looking for feedback here.

We would like to support everything 3.7+, so build time deps should be changed.

Ideally, environment variables can be set by IPython before launching the instance. This would ideally be passed in a KernelSpec configuration as one of the key values or as part of the metadata.

Since this is a new feature, barring tracking the implementation details here, the bulk of the conversation surrounding this format will take place at: jupyterlab/jupyterlab/issues/9374

I just get a JS alert saying "undefined".

This has been created during quansight work hours, it is the property of quansight.

People who should likely have access.

• mbussonn
• @ericdatakelly (not sure what you PyPI username is)
• datalayer (I belive that's Eric Charles username on PyPI.)

@echarles
I'm going to write some dockerfiles to test the application out in an isolated environment. I think this is the best way to mitigate issues during testing.

There are still a bunch of historical methods and block code in the python extension that are not used and not applicable.

These should be removed.

Thanks @echarles for pushing this forward, I've tried the new version and it was relatively straightforward.

A couple of notes:

1. we might want some instructions on how to create a template in the readme
   • or maybe even directly in the launcher.
   • or even a way to create one from a form I dont' know.
2. I like the pipette to mean pick new parameters.
3. I'm not sure what "Launch" is the right name. Maybe "create kernelspec" or something else.
4. uncaught error if one submit the forms without filling all the values.
5. deleting a kernelspec still uses "alert".

$ jupyter labextension develop --overwrite .
setup.py:35: DeprecatedWarning: create_cmdclass is deprecated as of 0.8 and will be removed in 1.0. "
Use `wrap_installers` to handle prebuild steps in cmdclass.
Use `get_data_files` to handle data files.
Use `include_package_data=True` and `MANIFEST.in` for package data.

  "jsdeps", package_data_spec=package_data_spec, data_files_spec=data_files_spec
setup.py:39: DeprecatedWarning: install_npm is deprecated as of 0.8 and will be removed in 1.0. Use `npm_builder` and `wrap_installers`
  jupyter_packaging.install_npm(HERE, build_cmd="build:prod", npm=["jlpm"]),

Those should likely be updated.

Like #7, barring the implementation details here, there is a larger discussion at hand that is happening. I will update this issue periodically, but we need to have a larger discussion surrounding this issue: jupyterlab/jupyterlab/issues/9374

When running jlpm run build:lib the following errors are logged in the console and I can't bring up a dev setup.

$ tsc
node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:3:1 - error TS6200: Definitions of the following identifiers conflict with those in another file: ErrorSchema, Form, UiSchema, FieldId, IdSchema, FieldPath, PathSchema, Widget, Field, FieldTemplateProps, ArrayFieldTemplateProps, ObjectFieldTemplateProps, AjvError, ErrorListProps, ISubmitEvent, FieldError, FieldValidation, FormValidation, FormSubmit, ThemeProps, AddButtonProps, ADDITIONAL_PROPERTY_FLAG, SchemaFieldProps, SchemaField

3 declare module '@rjsf/core' {
  ~~~~~~~

  node_modules/@rjsf/core/index.d.ts:3:1
    3 declare module '@rjsf/core' {
      ~~~~~~~
    Conflicts are in this file.

node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:132:22 - error TS2430: Interface 'FieldProps<T>' incorrectly extends interface 'Pick<HTMLAttributes<HTMLElement>, "id" | "dir" | "draggable" | "hidden" | "lang" | "placeholder" | "title" | "aria-activedescendant" | "aria-atomic" | ... 243 more ... | "onTransitionEndCapture">'.
  Types of property 'onFocus' are incompatible.
    Type '(id: string, value: any) => void' is not assignable to type '(event: FocusEvent<HTMLElement>) => void'.

132     export interface FieldProps<T = any>
                         ~~~~~~~~~~

node_modules/@rjsf/core/index.d.ts:3:1 - error TS6200: Definitions of the following identifiers conflict with those in another file: ErrorSchema, Form, UiSchema, FieldId, IdSchema, FieldPath, PathSchema, Widget, Field, FieldTemplateProps, ArrayFieldTemplateProps, ObjectFieldTemplateProps, AjvError, ErrorListProps, ISubmitEvent, FieldError, FieldValidation, FormValidation, FormSubmit, ThemeProps, AddButtonProps, ADDITIONAL_PROPERTY_FLAG, SchemaFieldProps, SchemaField

3 declare module '@rjsf/core' {
  ~~~~~~~

  node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:3:1
    3 declare module '@rjsf/core' {
      ~~~~~~~
    Conflicts are in this file.

node_modules/@rjsf/core/index.d.ts:39:9 - error TS2717: Subsequent property declarations must have the same type.  Property 'onBlur' must be of type '(id: string, value: any) => void', but here has type '(id: string, value: JSONPrimitive) => void'.

39         onBlur?: (id: string, value: boolean | number | string | null) => void;
           ~~~~~~

  node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:40:9
    40         onBlur?: (id: string, value: any) => void;
               ~~~~~~
    'onBlur' was also declared here.

node_modules/@rjsf/core/index.d.ts:42:9 - error TS2717: Subsequent property declarations must have the same type.  Property 'onFocus' must be of type '(id: string, value: any) => void', but here has type '(id: string, value: JSONPrimitive) => void'.

42         onFocus?: (id: string, value: boolean | number | string | null) => void;
           ~~~~~~~

  node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:43:9
    43         onFocus?: (id: string, value: any) => void;
               ~~~~~~~
    'onFocus' was also declared here.

node_modules/@rjsf/core/index.d.ts:43:9 - error TS2717: Subsequent property declarations must have the same type.  Property 'onSubmit' must be of type '(e: ISubmitEvent<T>, nativeEvent: FormEvent<HTMLFormElement>) => any', but here has type '(e: ISubmitEvent<T>) => any'.

43         onSubmit?: (e: ISubmitEvent<T>) => any;
           ~~~~~~~~

  node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:44:9
    44         onSubmit?: (e: ISubmitEvent<T>, nativeEvent: React.FormEvent<HTMLFormElement>) => any;
               ~~~~~~~~
    'onSubmit' was also declared here.

node_modules/@rjsf/core/index.d.ts:111:9 - error TS2717: Subsequent property declarations must have the same type.  Property 'onBlur' must be of type '(id: string, value: any) => void', but here has type '(id: string, value: JSONPrimitive) => void'.

111         onBlur: (id: string, value: boolean | number | string | null) => void;
            ~~~~~~

  node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:113:9
    113         onBlur: (id: string, value: any) => void;
                ~~~~~~
    'onBlur' was also declared here.

node_modules/@rjsf/core/index.d.ts:112:9 - error TS2717: Subsequent property declarations must have the same type.  Property 'onFocus' must be of type '(id: string, value: any) => void', but here has type '(id: string, value: JSONPrimitive) => void'.

112         onFocus: (id: string, value: boolean | number | string | null) => void;
            ~~~~~~~

  node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:114:9
    114         onFocus: (id: string, value: any) => void;
                ~~~~~~~
    'onFocus' was also declared here.

node_modules/@rjsf/core/index.d.ts:128:9 - error TS2717: Subsequent property declarations must have the same type.  Property 'onBlur' must be of type '(id: string, value: any) => void', but here has type '(id: string, value: JSONPrimitive) => void'.

128         onBlur: (id: string, value: boolean | number | string | null) => void;
            ~~~~~~

  node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:140:9
    140         onBlur: (id: string, value: any) => void;
                ~~~~~~
    'onBlur' was also declared here.

node_modules/@rjsf/core/index.d.ts:129:9 - error TS2717: Subsequent property declarations must have the same type.  Property 'registry' must be of type 'Registry', but here has type '{ fields: { [name: string]: Field; }; widgets: { [name: string]: Widget; }; definitions: { [name: string]: any; }; formContext: any; }'.

129         registry: {
            ~~~~~~~~

  node_modules/@jupyterlab/ui-components/node_modules/@rjsf/core/index.d.ts:142:9
    142         registry: Registry;
                ~~~~~~~~
    'registry' was also declared here.

node_modules/@rjsf/core/index.d.ts:141:9 - error TS2374: Duplicate string index signature.

141         [prop: string]: any;
            ~~~~~~~~~~~~~~~~~~~~


Found 11 errors.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
npm ERR! code 1
npm ERR! path /codemill/choudhdi/ksmm
npm ERR! command failed
npm ERR! command sh -c jlpm run clean && jlpm run build:prod

npm ERR! A complete log of this run can be found in:
npm ERR!     /u/choudhdi/.npm/_logs/2022-03-16T10_10_40_942Z-debug.log

Purely cosmetic, very low priority.

I would like to add pagination of some sort for the kernel cards that are rendered on the home screen.

We need to package the schema/kernelSchema.json in the pypi release, relying on the latest jupyter-packaging.

I have defined in MANIFEST.in include schema/kernelSchema.json without luck

Do we need to add that path in the setup.pyunder data_files_spec key?

The kernelspec schema is loaded from local file which works in dev env, but fails when installed from pypi unless you are located in the repo folder...

Let's load that from source.

Following a quick discussion that happen yesterday at the end of the meeting, I think there needs to be clarification of the goal of KSMM, potentially via users stories. I thought some of the requirements/end goals were clear, but that does not appear to be the case.

I don't particularly like user stories, but let's see if it's clearer.

Alice and Bob work at ACME Inc.

Alice is part of research IT that deploys jupyterHub backed on a multi-tenant and multi-cloud cluster. Alice has deep understanding of resources available, and how to configure a performant kernel deployment, by setting multiple parameters across many software in her stack.

Bob is a data scientist at ACME Inc, bob is good at analysing data, and know his models and the math behind them really well, even if Bob could spend the time to understand the dozen of parameter he could tweak using Jupyter on the cluster, he may not have the time to do so, and trust Alice to provide a number of premade choices with limited configurability.

For example Bob wish to have the following kernel:

• "Python3 - with GPU - 200Gb – 12 core - 1 node - staging"
  • with the options to switch from 12 core to 48 core, 1 nodes to 10 nodes and staging to production.
• "R - No GPU - 20Gb – 1 core - 1 node - production"
  • with the options to switch from 200Gb to 1Tb

Bob may want to also set some env variable as part of the kernel, to specify some extra configuration options, like OMP_NUM_THREADS, but those env variable are not know to alice, and might be part of BOB code. For the sake of simplicity we are going to assume those env variable are not Secrets.

Bob want a relatively simple UI. Click on the Python 3 kernel and change parameters, possibly see dropdown only for relevant parameters. For example R is never used with GPU, so no dropdown to switch GPU.

Alice is responsible for preparing the kernel "template" for Bob, she know the same options might be in multiple places, and want it to be both readable for Bob, and flexible to be given to her software.

For example the scheduler only takes memory amounts in KBytes, so she does not want bob to have to choose an option saying "209,715,093" (kb). So she needs a way to create a kernel spec with "templates":

Template 1:

• A Python Kernel.
• Can have GPU or not. If GPU, pass the GPU=True env variable, otherwise pass nothing.
• Memory can be from 100Gb to 1TB by step of 50Gb, option for the scheduler is --mem <value in kb>
• Core can be integers from 1 to 48.
• Number of nodes, integer from 1 to 100
• queues are strings passed as flags to the scheduler.

Template 2:

• A Python Kernel.
• Never needs a GPU parameter.
• Memory can be from 100Gb to 1TB by step of 50Gb, option for the scheduler is --mem <value in kb>
• Core is always 1.
• Number of nodes is always 1
• queues are strings passed as flags to the scheduler.
• user are not aware, but R is always ran on --tag=ssd machines.

User requirements and changes in machine configuration varies rarely, so it's ok if creating those files is time consuming, it can be automatized later.

Alice is concerned that a high numbers of options would be too confusing for user and would prefer to only have variables parameters to be shown to the user.

While not everything can be tested, we should at least make sure that the basic building and a couple of lints are run on github action.

As mentioned on #57 (comment), we could have a UI create a template from a form.

This is low priority feature.

 export const fetchMachineParameters = () => {
  const url = "http://localhost:8888/parameters"

Refers to a non existing handler.

Is that related to KSSchemaHandler ?

It would be good to migrate the release process to a more automated system like https://github.com/jupyter-server/jupyter_releaser