Giter Club home page Giter Club logo

ctuning / ck-crowdnode Goto Github PK

View Code? Open in Web Editor NEW
28.0 5.0 3.0 2.78 MB

Standalone, thin and portable server to let users participate in experiment crowdsourcing using Windows devices via Collective Knowledge Framework:

Home Page: http://cKnowledge.org

License: Other

C 87.29% CMake 1.30% Shell 0.16% Batchfile 1.01% C++ 0.30% Python 9.94%
collaborative-optimization collective-knowledge workload-automation windows server crowdsourcing-experiments json-api adb-like-server

ck-crowdnode's Introduction

Linux/MacOS: Build Status Windows: Windows Build status

Collective Knowledge Node for experiment crowdsourcing (on Windows devices)

Standalone, thin and portable server to let users participate in experiment crowdsourcing via Collective Knowledge. It unifies remote execution on Windows similar to Android ADB (experiment crowdsourcing via Linux and Android platforms is already supported by CK). It can also be used to create farms of machines for collaborative benchmarking and tuning (crowd-benchmarking).

Note that both server and client should run Windows.

Project homepage:

License

  • Permissive 3-clause BSD license. (See LICENSE.txt for more details).

Status

Relatively stable - testing phase

Usage: server side

On Windows:

  1. Download the installer from Appveyor

  2. Install and start "CK crowd-node server".

  3. Write down "[INFO for CK client]" - you will require this info to configure this target machine on a client.

Alternatively, you can build it manually (you'll need CMake and Visual Studio):

mkdir build
cd build
cmake ..
cmake --build . --config Release --target PACKAGE

Now, you should be able to run build/Release/ck-crowdnode-server.exe.

Usage: client side

Install CK framework. If you have PIP, you can install it simply as following:

 $ pip install ck

Pull ck-autotuning repository (including dependencies):

 $ ck pull repo:ck-autotuning

Prepare local file with a secret key (see [INFO for CK client]), for example in C:\secret-key.txt

Register target machine with ck-crowdnode-server via (substitute ''my-remote-target'' with any other user-friendly name)

 $ ck add machine:my-remote-target

Select 4) CK: remote Windows machine accessed via CK crowd node. Then 4) windows-64

Then enter hostname, port, path to public key (C:\secret-key.txt), and full path to files on a target machine (all info is available via [INFO for CK client] - we later plan to automate this process).

Now you can check that you machine is connected and online via

 $ ck show machine

or

 $ ck browse machine

Now you should be able to compile and run sample program using this target. You need to have Microsoft C compilers and Microsoft SDK installed (there is a free edition available). You can also download and install LLVM for Windows, but remember that it also requires Visual C compiler and Microsoft SDK.

Try to compile susan benchmark (during first compilation, CK will attempt to automatically detect installed compilers and SDK while asking you extra questions, if needed):

 $ ck compile program:cbench-automotive-susan --speed --target=my-remote-target

Finally, you can try to run it:

 $ ck run program:cbench-automotive-susan --target=my-remote-target

If everything is configured correctly, this code will be executed several times on a required target and execution time will be reported!

If you have any problems, questions or comments, do not hesitate to get in touch with the CK community via [our public mailing list](https://groups.google.com/forum/#!forum/collective-knowledge open CK mailing list)!

Further details

Public discussions

logo

ck-crowdnode's People

Contributors

dsavenko avatar gfursin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

ck-crowdnode's Issues

Adding secret_key file ...

One more thing - we should add some basic security for now (I expect that if prototype is accepted, we or someone else can provide much stronger security).

Whenever ANY command is passed to server, you need to check if a 'secret_key' string is present in config file (#5), and if it's !='', compare it with the 'secret_key' in the input JSON, and quit if they are not the same:
{
'return':3,
'error':'secret keys do not match'
}

Thanks a lot!

making ck-crowdnode-server available via Windows app store

I have registered ctuning in Windows app store. However, it requires .xap, .appx, .appxbundle, .appxupload package to be deployed. Is it possible to check how to make it (along with ck-crowdnode-server.exe), please? I can then deploy it there and check on a few other devices (maybe it will even work on Windows mobile?)

server crashes after push command

Hi,
I tested new server on Linux and it accepts push from CK, records file properly, but then crashes with the log (attached).
log.txt
I think it's something with memory allocation/deallocation.
Is it possible to check it please.
To reproduce:
$ ck add repo:remote-ck-node --url=http://localhost:3333 --remote --quiet
$ ./ck-crowdnode-server
$ cat > test.txt
HELLO
$ ck push remote-ck-node:: --filename=test.txt

Thanks!

P.S. By the way, pull worked and server didn't crash:
$ ck pull remote-ck-node:: --filename=test.txt

big file was not recorded properly during push

Tried to push relatively large binary file (700Kb). I could see that server prints lots of incoming stuff (seems correct, by the way, it's better probably to put a limit on print of long key, i.e. < 256 bytes), but recorded only 90 bytes ... Interestingly, server didn't crash this time at the end (see #1) and even returned 'ok'...

Is is possible to test it too, please?

Thanks!

problem compiling with VS2017 - not urgent

When trying to compile this project using VS2017 and _compile.bat, there is an error
(_compile_raw.bat works fine) :
C:\Users\ckdnn\CK-MISC\ck-crowdnode>C:\Users\ckdnn\CK-MISC\ck-crowdnode_compile.bat
C:\Users\ckdnn\CK-MISC\ck-crowdnode>call ck set env tags=compiler,lang-c --local --bat_file=tmp-ck-env.bat --bat_new --print && call tmp-ck-env.bat && del /Q tmp-ck-env.bat
C:\Users\ckdnn\Anaconda3\lib\runpy.py:125: RuntimeWarning: 'ck.kernel' found in sys.modules after import of package 'ck', but prior to execution of 'ck.kernel'; this may result in unpredictab
le behaviour
warn(RuntimeWarning(msg))
More than one environment found for required software with tags="compiler,lang-c" and setup={"host_os_uoa": "windows-64", "target_os_uoa": "windows-64", "target_os_bits": "64"}:
Microsoft C compiler - v19.10.25017 (64bits,compiler,host-os-windows-64,lang-c,lang-cpp,mcl,microsoft,target-os-windows-64,v19,v19.10,v19.10.25017 (fb528ca2021d4c9b))

  • Depends on "msdk" (env UOA=1777d4bd6890515b, tags="microsoft,sdk", version=10.0A)
    LLVM compiler - v4.0.0 (64bits,compiler,host-os-windows-64,lang-c,lang-cpp,llvm,target-os-windows-64,v4,v4.0,v4.0.0 (7048a5075ccd4687))
  • Depends on "mvsc" (env UOA=fb528ca2021d4c9b, tags="microsoft,compiler", version=19.10.25017)
    GNU C compiler - v6.3.0 (64bits,compiler,gcc,host-os-windows-64,lang-c,lang-cpp,target-os-windows-64,v6,v6.3,v6.3.0 (7cab70c651968ea8))
    Select one of the options for required software with tags="compiler,lang-c" and setup={"host_os_uoa": "windows-64", "target_os_uoa": "windows-64", "target_os_bits": "64"} or press Enter for 0
    :
    CK environment found using tags "compiler,lang-c" : fb528ca2021d4c9b

** Visual Studio 2017 Developer Command Prompt v15.0.26228.9
** Copyright (c) 2017 Microsoft Corporation

[vcvarsall.bat] Environment initialized for: 'x64'

** Visual Studio 2017 Developer Command Prompt v15.0.26228.9
** Copyright (c) 2017 Microsoft Corporation

[vcvarsall.bat] Environment initialized for: 'x64'
C:\Users\ckdnn\Anaconda3\lib\runpy.py:125: RuntimeWarning: 'ck.kernel' found in sys.modules after import of package 'ck', but prior to execution of 'ck.kernel'; this may result in unpredictab
le behaviour
warn(RuntimeWarning(msg))
CK environment found using tags "tool,cmake" : 23ac3bf54fbf90bb
A subdirectory or file build already exists.
CMake Warning at C:/Program Files/CMake/share/cmake-3.7/Modules/InstallRequiredSystemLibraries.cmake:453 (message):
system runtime library file does not exist:
'MSVC14_REDIST_DIR-NOTFOUND/x64/Microsoft.VC140.CRT/msvcp140.dll'
Call Stack (most recent call first):
CMakeLists.txt:26 (include)
CMake Warning at C:/Program Files/CMake/share/cmake-3.7/Modules/InstallRequiredSystemLibraries.cmake:453 (message):
system runtime library file does not exist:
'MSVC14_REDIST_DIR-NOTFOUND/x64/Microsoft.VC140.CRT/vcruntime140.dll'
Call Stack (most recent call first):
CMakeLists.txt:26 (include)
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/ckdnn/CK-MISC/ck-crowdnode/build
Microsoft (R) Build Engine version 15.1.548.43366
Copyright (C) Microsoft Corporation. All rights reserved.
MSBUILD : error MSB1009: Project file does not exist.
Switch: Project.sln

adding configuration file

It will be useful to add reading of the configuration file 'ck-crowdnode-config.json' when server starts.

On Linux we can keep it in '$HOME/.ck-crowdnode/ck-crowdnode-config.json'.
On Windows we may keep it in %LOCALAPPDATA%.ck-crowdnode\ck-crowdnode-config.json (see https://blogs.msdn.microsoft.com/patricka/2010/03/18/where-should-i-store-my-data-and-configuration-files-if-i-target-multiple-os-versions/)

If it doesn't exist at first run, we can create it with the following default config:

Current config should have:
{
'port':3333,
'path_to_files':'/tmp/ck-crowdnode'
}

Or on Windows, it will likely be %Temp%\ck-crowdnode

Also at run we should check that this directory exists and create it otherwise.

path_to_files is where we push/pull/execute files

We can then add a new command 'status' which returns
{ 'return':0,
'path_to_files":...
}

This will allow us to query server and check that it works fine ...

extra improvements

I managed to create a distro (setup.exe) and run it on a remote machine. A few small things that needs to be added to the server:

  • We need to print real IP (i.e. not 127.0.0.1, but something like 192.168.1.64 on local or 186.xx.yy.zz on real network)...
    [INFO for CK client]: real IP: xyz
    This is needed to set up CK machine (otherwise I need to check IP via ipconfig)
  • When executing cmd (shell), we should change current directory to "path to files", before executing it!

Thanks!

make this node compile and run on Windows

I tried to make it compatible with Windows build system (using my 10-years old scripts when I did some client/servers on Windows via socket), but it's not yet working. Need help to finalize it! Thanks!

query CK crowdnode status from CK machine module

We should obtain various parameters automatically (including full path, etc) when registering CK crowdnode in the CK machine module (to be more user-friendly) using new status function. It is not critical though at this moment.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.