Giter Club home page Giter Club logo

yagna's Introduction

Golem

Official Rust implementation of Golem. Golem is a network of nodes that implement the Golem Network protocol. We provide the default implementation of such a node in the form of the Golem daemon, Yagna.


A flexible, open-source platform for democratised access to digital resources.

Golem Network has officially gone on Ethereum Mainnet with the Beta I release in March 2021.

Golem democratizes society’s access to computing power by creating a decentralized platform where anyone can build a variety of applications, request computational resources and/or offer their idle systems in exchange for cryptocurrency tokens (GLM). The actors in this decentralized network can assume one of the three non-exclusive roles:

  • Requestor Has a need to use IT resources such as computation hardware. Those resources are purchased in the decentralized market. The actual usage of the resources is backed by Golem's decentralized infrastructure.

  • Provider Has IT resources available that can be shared with other actors in the network. Those resources are sold in the decentralized market.

  • Developer Builds applications to run for requestors on the network. Golem's potential goes much beyond a singular application. See Awesome Golem for just a taste of the various types of applications that can be built and run on Golem!

Documentation

For a more in-depth look at how Golem works, head over to our documentation.

Project Layout

  • agent/provider - provider agent implementation based on core services.
  • core - core services for the open computation marketplace.
  • exe-unit - ExeUnit Supervisor - a common part of all runtimes for yagna.
  • test-utils - some helpers for testing purposes
  • utils - trash bin for all other stuff ;)
  • docs - project documentation including analysis and specifications.

Public API

The public API rust binding with data model is in the ya-client repo.

High Level APIs

The public high-level API for Python is in yapapi repo and the JS/TS port is contained in the @golem-sdk/golem-js repo.

Runtimes

We call our runtime ExeUnit. As for now we support

Other ExeUnit types are to come (see below).

Golem Beta Release(s)

Important milestones for Golem development were Beta I and most recent Beta II. With those releases we have delivered:

  • MVP (minimum viable product), though not feature rich yet, it is usable for early adopters
  • Clean and easy experience for new and existing users.
  • Support for GLM payments (both L1 & L2 on Ethereum Mainnet)
  • Production-ready and easy to maintain code base.
  • Modular architecture with all the building blocks being replaceable.
  • Small binaries (under 30Mb).
  • Documentation and SDK for Golem app developers.

List of implemented and planned functionality

  1. Distributed computations
    • Batching
    • Services (PoC stage)
  2. Computational environment (aka ExeUnit)
    • Wasm computation
    • Light vm-s
    • Docker on Linux (optional)
    • SGX on Graphene (PoC stage)
  3. Payment platform
    • Payments with GLM
    • ERC20 token
    • Layer 1 & Layer 2 transactions
    • Payment matching (optional) (Ability for the invoice issuer to match the payment with Debit Note(s)/Invoice(s)).
  4. Transaction system
    • Pay as you go(lem) (see more)
    • Pay per task
    • Pay for dev (optional)
  5. Network
    • P2P (Hybrid P2P; in progress)
    • Ability to work behind NAT (Relays; in progress)
  6. Verification
    • Verification by redundancy (see also)
    • No verification
    • Verification by humans (optional)

Road ahead

We are actively working on improving Yagna and extending its functionality, check upcoming releases and other news on our blog.

yagna's People

Contributors

azawlocki avatar bidzyyys avatar biryukovmaxim avatar cryptobench avatar dependabot[bot] avatar etam avatar evik42 avatar filipgolem avatar jalas167 avatar jiivan avatar kamirr avatar kmazurek avatar kubkon avatar maaktweluit avatar mat7ias avatar mateuszsrebrny avatar mfranciszkiewicz avatar mrdarthshoe avatar nieznanysprawiciel avatar omeg avatar pnowosie avatar prekucki avatar pwalski avatar scx1332 avatar staszek-krotki avatar stranger80 avatar szymek156 avatar tworec avatar wiezzel avatar wkargul avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

somerandomusername73 developerfred mat7ias cduverne 1crazymoney figurestudios marijnstevens paulperegud root-servers polymath-is tomerze qingshang990 murataslan1 1ndevelopment mariszo biryukovmaxim orabhan pwalski akudamarlp alishahzadsaab hboisgibault mlliarm icodein jedbrooke 00mjk deltav-deltaverse evanrichter mayhemheroes adaptivefailure pinkdiamond1 william3johnson inferno-inc openssl-sg-insights 3122365857 dmgolembiowski prekucki samkenxstream imgbit iq-scm bilalb84 atlas16a jorgecuenca1 gu7758521 msiqe983 jmgomezl cafercangundogdu omahs theinventor17 cameronraysmith mohanish2504 cnkumar20 azmisahincom armandojff akalnengn muhtasimtanmoy scx1332 lsygitowicz

yagna's Issues

Payment API: Implement payer signing keys associated with an Identity

We want to have a ability to detach the "Node Id" (identity) from the "Signer Address" - the key used to sign payment transactions (which is correlated with the wallet from which payments are made).

For an Identity (NodeId) we need to be able to specify Signer Addresses (multiple) with ability to indicate the "current" sender (so when a payment is scheduled via AcceptDebitNote/AcceptInvoice).

The Signer Addresses should be managed via Yagna CLI:

  • AddSignerAddress(identity, signer)
  • SetActiveSignerAddress(identity, signer)
  • LockSIgnerAddress(signer)
  • UnlockSignerAddress(signer)
  • DeleteSignerAddress(signer)

This logic should be embedded in Payment API Implementation, ie for an appKey associated with AcceptInvoice request, the Node Id is determined, and this determines the active signer address to be selected by the Payment Driver for transaction signing. The signer address is passed to Identity API when passing transaction data for signing.

Requestor lib for batch tasks

Provide simple API for different requestor implementations.

let tasks = vec!["1","2","3","4","5"];
let image_spec = ImageSpec::from_github("prekucki/[email protected]")
  .runtime(WasmRuntime::Wasi(1));

let progress = TaskSession::new("simple wasm app")
  .with_timeout(Duration::from_secs(60))
  .demand(WasmDemand::with_image(image_spec).min_ram_gib(0.5).min_storage_gib(1))
  .tasks(tasks.into_iter().map(|arg| commands! {
     deploy;
     start;
     run("test-wasi", arg)
     stop;
  })
  .run();

tui_progress_monitor(progress).await;

TODO:

  • Hide configuration in enviroment variables.
  • Write simple negotiation
  • Write helpers for WasmDemand
  • Write simple scheduler
  • Write helpers to build Commands (macro commands!)
  • Wrtie simple tui reporter

AuthMiddlewar: already borrowed: BorrowMutError 

[2020-02-27T10:19:36Z INFO  actix_web::middleware::logger] 127.0.0.1:63314 "GET /activity-api/v1/events?timeout=3 HTTP/1.1" 200 134 "-" "-" 3.767851
[2020-02-27T10:19:36Z DEBUG ya_activity::provider] set_activity_state_web Json: ActivityState { state: StatePair(Terminated, None), reason: None, error_message: None }
[2020-02-27T10:19:36Z DEBUG ya_client::web] doing GET on http://127.0.0.1:5001/market-api/v1/agreements/958495eb-9b71-4e2f-80d4-2ae8980ed94c
[2020-02-27T10:19:36Z DEBUG ya_activity::common] checking caller: 0xee1dbafa491577c55fd2892073340d0f3967acca vs expected: 0xee1dbafa491577c55fd2892073340d0f3967acca
[2020-02-27T10:19:36Z INFO  actix_web::middleware::logger] 127.0.0.1:63314 "PUT /activity-api/v1/activity/c3bdd047aa464dc69927ff6a4b5c3353/state HTTP/1.1" 200 4 "-" "-" 0.005542
[2020-02-27T10:19:36Z DEBUG ya_activity::dao::activity_state] getting activity state
[2020-02-27T10:19:36Z DEBUG ya_activity::provider] getting events
[2020-02-27T10:19:40Z DEBUG ya_activity::provider] getting events
[2020-02-27T10:19:44Z DEBUG ya_activity::provider] getting events
[2020-02-27T10:19:48Z DEBUG ya_activity::provider] getting events
[2020-02-27T10:19:52Z DEBUG ya_activity::provider] getting events
[2020-02-27T10:19:56Z DEBUG ya_activity::provider] getting events
[2020-02-27T10:20:00Z DEBUG ya_activity::provider] getting events
[2020-02-27T10:20:04Z DEBUG ya_activity::provider] getting events
thread 'actix-rt:worker:0' panicked at 'already borrowed: BorrowMutError', src/libcore/result.rs:1188:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:84
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1025
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:193
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:210
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:471
  11: rust_begin_unwind
             at src/libstd/panicking.rs:375
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:84
  13: core::result::unwrap_failed
             at src/libcore/result.rs:1188
  14: core::result::Result<T,E>::expect
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/result.rs:983
  15: core::cell::RefCell<T>::borrow_mut
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/cell.rs:879
  16: <alloc::rc::Rc<core::cell::RefCell<S>> as actix_service::Service>::poll_ready
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-service-1.0.5/src/lib.rs:276
  17: <ya_service_api_web::middleware::auth::AuthMiddleware<S> as actix_service::Service>::poll_ready
             at /Users/tworec/git/yagna/core/serv-api/web/src/middleware/auth/mod.rs:70
  18: <actix_web::app_service::AppInitService<T,B> as actix_service::Service>::poll_ready
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-web-2.0.0/src/app_service.rs:227
  19: <actix_http::cloneable::CloneableService<T> as actix_service::Service>::poll_ready
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-http-1.0.1/src/cloneable.rs:36
  20: <actix_http::service::HttpServiceHandler<T,S,B,X,U> as actix_service::Service>::poll_ready
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-http-1.0.1/src/service.rs:514
  21: <actix_service::and_then::AndThenService<A,B> as actix_service::Service>::poll_ready
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-service-1.0.5/src/and_then.rs:45
  22: <actix_server::service::StreamService<T> as actix_service::Service>::poll_ready
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-server-1.0.1/src/service.rs:72
  23: actix_server::worker::Worker::check_readiness
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-server-1.0.1/src/worker.rs:260
  24: <actix_server::worker::Worker as core::future::future::Future>::poll
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-server-1.0.1/src/worker.rs:453
  25: std::future::poll_with_tls_context
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/future.rs:99
  26: actix_server::worker::Worker::start::{{closure}}::{{closure}}
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-server-1.0.1/src/worker.rs:217
  27: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/future.rs:43
  28: tokio::task::core::Core<T>::poll
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/core.rs:128
  29: tokio::task::harness::Harness<T,S>::poll::{{closure}}::{{closure}}
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/harness.rs:120
  30: core::ops::function::FnOnce::call_once
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/ops/function.rs:232
  31: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/panic.rs:318
  32: std::panicking::try::do_call
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/panicking.rs:292
  33: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  34: std::panicking::try
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/panicking.rs:270
  35: std::panic::catch_unwind
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/panic.rs:394
  36: tokio::task::harness::Harness<T,S>::poll::{{closure}}
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/harness.rs:101
  37: tokio::loom::std::causal_cell::CausalCell<T>::with_mut
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/loom/std/causal_cell.rs:41
  38: tokio::task::harness::Harness<T,S>::poll
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/harness.rs:100
  39: tokio::task::raw::poll
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/raw.rs:162
  40: tokio::task::raw::RawTask::poll
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/raw.rs:113
  41: tokio::task::Task<S>::run
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/mod.rs:381
  42: tokio::task::local::Scheduler::tick
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/local.rs:513
  43: <tokio::task::local::LocalFuture<F> as core::future::future::Future>::poll::{{closure}}
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/local.rs:407
  44: tokio::task::local::Scheduler::with::{{closure}}
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/local.rs:478
  45: std::thread::local::LocalKey<T>::try_with
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/thread/local.rs:262
  46: std::thread::local::LocalKey<T>::with
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/thread/local.rs:239
  47: tokio::task::local::Scheduler::with
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/local.rs:474
  48: <tokio::task::local::LocalFuture<F> as core::future::future::Future>::poll
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/local.rs:402
  49: std::future::poll_with_tls_context
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/future.rs:99
  50: tokio::task::local::LocalSet::run_until::{{closure}}
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/local.rs:358
  51: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/future.rs:43
  52: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/basic_scheduler.rs:138
  53: tokio::runtime::Runtime::block_on::{{closure}}
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/mod.rs:411
  54: tokio::runtime::context::enter
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/context.rs:72
  55: tokio::runtime::handle::Handle::enter
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/handle.rs:34
  56: tokio::runtime::Runtime::block_on
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/mod.rs:408
  57: tokio::task::local::LocalSet::block_on
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/task/local.rs:321
  58: actix_rt::runtime::Runtime::block_on
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-1.0.0/src/runtime.rs:89
  59: actix_rt::arbiter::Arbiter::new::{{closure}}
             at /Users/tworec/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-1.0.0/src/arbiter.rs:125
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

ExeUnit CPU accounting

Features

  • Reports CPU*s usage to given GSB endpoint (from cmdline).
  • Reporting every given n seconds since ExeUnit start.
  • Reporting every n CPU*s exceded.
  • Reporting on demand

Implementation notes

TODO

Allocation managment

Allocation does not have an association with identity.

Rest GET /payment-api/v1/requestor/allocations returns all allocations. not only allocation associated with token identity.

Cli yagna payment status returns sum for all reservations not only reservations associated with given identity.

Closing file in gftp upload

Related to changes introduced in: #137
File descriptor exist after finished upload.
We need gsb unbind function to remove all file references

Market API negotiation

Current implementation of MarketApi Mk1 (aka testbed) is not delivering Proposal event to the Provider after the Requestor sends CounterProposal.

datadir initalization

yagna server fails if datadir doas not exists.

2020-02-05T09:22:03Z INFO  yagna] Using data dir: "/home/reqc/.local/share/yagna" 
[2020-02-05T09:22:03Z INFO  yagna] Starting yagna service!
[2020-02-05T09:22:03Z INFO  ya_sb_router] Router listening on: 127.0.0.1:7464
[2020-02-05T09:22:03Z INFO  ya_persistence::executor] using database at: /home/reqc/.local/share/yagna/yagna.db
[2020-02-05T09:22:03Z ERROR r2d2] Unable to open the database file
[2020-02-05T09:22:03Z ERROR r2d2] Unable to open the database file

Payment API: Allocation 

[
  {
    "paymentId": "82a449cd-579c-4eec-8fd9-7c79a968b6b4",
    "payerId": "0x4563845390a720f93f57dfef1f4f24fc66729ff9",
    "payeeId": "0x0a92dd80ed3d4bcb7448e7f0a5e70acfb8662c1f",
    "amount": "10",
    "timestamp": "2020-03-02T10:27:45Z",
    "allocationId": "012e0acb-e137-4fbf-9127-5f2c9e961b3d",
    "debitNoteIds": [],
    "invoiceIds": [
      "cec4c8c1-6658-4918-b2f1-45a0461414fa"
    ],
    "details": "eyJyZWNpcGllbnQiOiIweDBhOTJkZDgwZWQzZDRiY2I3NDQ4ZTdmMGE1ZTcwYWNmYjg2NjJjMWYiLCJzZW5kZXIiOiIweDQ1NjM4NDUzOTBhNzIwZjkzZjU3ZGZlZjFmNGYyNGZjNjY3MjlmZjkiLCJhbW91bnQiOiIweDhhYzcyMzA0ODllODAwMDAiLCJkYXRlIjoiMjAyMC0wMy0wMlQxMDoyNzo0NS4yODY0NDk1NDhaIn0="
  }
]

curl http://127.0.0.1:2001/payment-api/v1/requestor/allocations/012e0acb-e137-4fbf-9127-5f2c9e961b3d'

{
  "allocationId": "012e0acb-e137-4fbf-9127-5f2c9e961b3d",
  "totalAmount": "50",
  "spentAmount": "10",
  "remainingAmount": "40",
  "makeDeposit": false
}
curl 'http://127.0.0.1:2001/payment-api/v1/requestor/allocations' | jq

[
  {
    "allocationId": "012e0acb-e137-4fbf-9127-5f2c9e961b3d",
    "totalAmount": "50",
    "spentAmount": "0",
    "remainingAmount": "50",
    "makeDeposit": false
  }
]

Market API: missing demand/offer properties on agreement

After negotaitions:

$ curl -s -H "Authorization: Bearer 49acf3568f5542a29ad37f75adcecdc6" -H "Accept: 
application/json" http://127.0.0.1:5001/market-api/v1/agreements/b0e1258f-0253-48ad-a41a-cc4186725046 

{
  "agreementId": "b0e1258f-0253-48ad-a41a-cc4186725046",
  "demand": {
    "demandId": "6cd04f1e-69be-4b04-9056-78b8dd813159",
    "requestorId": "0x4563845390a720f93f57dfef1f4f24fc66729ff9",
    "properties": {},
    "constraints": "(&\n  (golem.runtime.wasm.wasi.version@v=0.0.0)\n  (golem.inf.mem.gib>0.1)\n  (golem.srv.comp.wasm.task_package=https://github.com/golemfactory/mandelbrot/releases/download/0.2.0/mandelbrot-0.2.0.yimg)\n  (golem.node.id.name=6136a9c6-58a4-11ea-ad3e-dba860f04160)\n)\n"
  },
  "offer": {
    "offerId": "b0e1258f-0253-48ad-a41a-cc4186725046",
    "providerId": "0x0a92dd80ed3d4bcb7448e7f0a5e70acfb8662c1f",
    "properties": {},
    "constraints": "()\n"
  },
  "validTo": "2021-02-25T10:06:08.8272057Z",
  "approvedDate": null,
  "state": "Approved",
  "proposedSignature": null,
  "approvedSignature": null,
  "committedSignature": null
}

Scanario

curl -X POST -s -H "Authorization: Bearer 3f86aa74c75e40af9d11e1dc63d3aff6" -H "Accept: application/json" -H "Content-type: application/json" --data '{
  "constraints": "()\n",
  "properties": {
    "golem": {
      "activity.caps": {
        "transfer.protocol": [
          "http",
          "https",
          "container"
        ]
      },
      "com": {
        "pricing": {
          "model": "linear",
          "model.linear": {
            "coeffs": [
              0,
              0.0001,
              0.0016
            ]
          }
        },
        "scheme": "payu",
        "scheme.payu": {
          "interval_sec": 60
        }
      },
      "inf": {
        "mem.gib": 0.5,
        "storage.gib": 5.0
      },
      "node": {
        "id.name": "6136a9c6-58a4-11ea-ad3e-dba860f04160"
      },
      "runtime.wasm": {
        "wasi.version@v": "0.0.0"
      },
      "srv.comp.wasm.task_package": [
        "https://github.com/golemfactory/mandelbrot/releases/download/0.2.0/mandelbrot-0.2.0.yimg"
      ],
      "usage": {
        "vector": [
          "golem.usage.duration_sec",
          "golem.usage.cpu_sec"
        ]
      }
    }
  }
}' http://127.0.0.1:5001/market-api/v1/offers
curl -X POST -s -H "Authorization: Bearer 49acf3568f5542a29ad37f75adcecdc6" -H "Accept: application/json" -H "Content-type: application/json" --data '{
  "constraints": "(&\n  (golem.runtime.wasm.wasi.version@v=0.0.0)\n  (golem.inf.mem.gib>0.1)\n  (golem.srv.comp.wasm.task_package=https://github.com/golemfactory/mandelbrot/releases/download/0.2.0/mandelbrot-0.2.0.yimg)\n  (golem.node.id.name=6136a9c6-58a4-11ea-ad3e-dba860f04160)\n)\n",
  "properties": {
    "golem.node": {
      "id.name": "test requestor"
    },
    "golem.srv.comp": {
      "wasm.task_package": "https://github.com/golemfactory/mandelbrot/releases/download/0.2.0/mandelbrot-0.2.0.yimg"
    }
  }
}' http://127.0.0.1:5001/market-api/v1/demands
curl -X GET -s -H 'Authorization: Bearer 49acf3568f5542a29ad37f75adcecdc6' -H 'Accept: application/json' 'http://127.0.0.1:5001/market-api/v1/demands/816a54a4-101e-4c40-b35c-714b61568b27/events?maxEvents=1&timeout=30'
curl -X POST -s -H "Authorization: Bearer 49acf3568f5542a29ad37f75adcecdc6" -H "Accept: application/json" -H "Content-type: application/json" --data '{
  "constraints": "(&\n  (golem.runtime.wasm.wasi.version@v=0.0.0)\n  (golem.inf.mem.gib>0.1)\n  (golem.srv.comp.wasm.task_package=https://github.com/golemfactory/mandelbrot/releases/download/0.2.0/mandelbrot-0.2.0.yimg)\n  (golem.node.id.name=6136a9c6-58a4-11ea-ad3e-dba860f04160)\n)\n",
  "properties": {
    "golem.node": {
      "id.name": "test requestor"
    },
    "golem.srv.comp": {
      "wasm.task_package": "https://github.com/golemfactory/mandelbrot/releases/download/0.2.0/mandelbrot-0.2.0.yimg"
    }
  }
}' http://127.0.0.1:5001/market-api/v1/demands/816a54a4-101e-4c40-b35c-714b61568b27/proposals/b9283a22-61d0-4778-be29-130b7a24273f
curl -X GET -s -H 'Authorization: Bearer 3f86aa74c75e40af9d11e1dc63d3aff6' -H 'Accept: application/json' 'http://127.0.0.1:5001/market-api/v1/offers/6ce3751b-aff1-4765-b0aa-14af9b64ef44/events?maxEvents=1&timeout=30'
offer=6ce3751b-aff1-4765-b0aa-14af9b64ef44, arg=
curl -X POST -s -H "Authorization: Bearer 3f86aa74c75e40af9d11e1dc63d3aff6" -H "Accept: application/json" -H "Content-type: application/json" --data '{
  "constraints": "()\n",
  "properties": {
    "golem": {
      "activity.caps": {
        "transfer.protocol": [
          "http",
          "https",
          "container"
        ]
      },
      "com": {
        "pricing": {
          "model": "linear",
          "model.linear": {
            "coeffs": [
              0,
              0.0001,
              0.0016
            ]
          }
        },
        "scheme": "payu",
        "scheme.payu": {
          "interval_sec": 60
        }
      },
      "inf": {
        "mem.gib": 0.5,
        "storage.gib": 5.0
      },
      "node": {
        "id.name": "6136a9c6-58a4-11ea-ad3e-dba860f04160"
      },
      "runtime.wasm": {
        "wasi.version@v": "0.0.0"
      },
      "srv.comp.wasm.task_package": [
        "https://github.com/golemfactory/mandelbrot/releases/download/0.2.0/mandelbrot-0.2.0.yimg"
      ],
      "usage": {
        "vector": [
          "golem.usage.duration_sec",
          "golem.usage.cpu_sec"
        ]
      }
    }
  }
}' http://127.0.0.1:5001/market-api/v1/offers/6ce3751b-aff1-4765-b0aa-14af9b64ef44/proposals/6cd04f1e-69be-4b04-9056-78b8dd813159
curl -X POST -s -H "Authorization: Bearer 49acf3568f5542a29ad37f75adcecdc6" -H "Accept: application/json" -H "Content-type: application/json" --data '{"proposalId": "b0e1258f-0253-48ad-a41a-cc4186725046", "validTo": "2021-02-25T10:06:08.827205733Z"}' http://127.0.0.1:5001/market-api/v1/agreements
curl -X POST -s -H "Authorization: Bearer 49acf3568f5542a29ad37f75adcecdc6" -H "Accept: application/json" -H "Content-type: application/json" --data '{}' http://127.0.0.1:5001/market-api/v1/agreements/b0e1258f-0253-48ad-a41a-cc4186725046/confirm
curl -X GET -s -H 'Authorization: Bearer 3f86aa74c75e40af9d11e1dc63d3aff6' -H 'Accept: application/json' 'http://127.0.0.1:5001/market-api/v1/offers/6ce3751b-aff1-4765-b0aa-14af9b64ef44/events?maxEvents=1&timeout=30'
curl -X POST -s -H "Authorization: Bearer 3f86aa74c75e40af9d11e1dc63d3aff6" -H "Accept: application/json" -H "Content-type: application/json" --data '{}' http://127.0.0.1:5001/market-api/v1/agreements/b0e1258f-0253-48ad-a41a-cc4186725046/approve
curl -X POST -s -H "Authorization: Bearer 49acf3568f5542a29ad37f75adcecdc6" -H "Accept: application/json" -H "Content-type: application/json" --data '{}' http://127.0.0.1:5001/market-api/v1/agreements/b0e1258f-0253-48ad-a41a-cc4186725046/wait

WASM-Time example app

Goals:

  • Prove that WASMTime is usable for our purposes.
  • Check if memory limitation is supported
  • Check if is it possible to add diffrent VFS implementation.

Write a simple application:
wasm-test-app [wasm-code] [in-dir] [out-dir]

where:

  • wasm-code is path to wasm file
  • in-dir is a path for read-only mounted directory accessible inside wasm as in/..
  • 'out-dir' is a path for directory accessible inside wasm as out/..

Market API - HTTP Error 500

Steps to reproduce:
POST Demand object in JSON format to /market-api/v1/demands/ with invalid token/app_key.

Result:
500 Internal Server Error.

Expected result:
401 or 403.

Gwasm-runner on wasi

  • Replace emscripten with wasmtime.
  • in WASI there is no current directory. only mapping will be curdir to the current directory.
  • fix examples to run with it.

Redundant environment variable ("GSB_URL") used in ya-service-bus module

The remote_router.rs file in ya-service-bus module refers to GSB_URL environment variable, which seems to be redundant to what already is assembled in YAGNA_BUS_ADDR_STR constant in ya-serv-api (constants.rs) module.

Instead of yet another GSB_URL env variable, the ya-service-bus should receive the URL as a dependency from its creator (eg. if yagna process collects it from environment variables, it should pass the url into the ya-service-bus module).

Deme step: Provider Create Offer

ya-provider change offer creation mechanism:

  • usage vector definition. (golem.com.ussage.vector)
  • configuration for golem.com.pricing.* linear only.
  • confiuration createtion/update from ya-provider cli.

[CLI] Payment Status

Expected:

yagna payment status

---
amount: 100 GNT
reserved: 0 GNT
outgoing:
   requested: 0 GNT
   accepted: 0 GNT
   confirmed: 0 GNT
   rejected: 0 GNT
incoming:
   requested: 0 GNT
   accepted: 0 GNT
   confirmed: 0 GNT
   rejected: 0 GNT

Duplicated activity id in database

Issue: Can't run requestor and provider on the same machine.
It causes conflict in database, while adding activity and changing it's state, because activity entity is duplicated.

Activity table should have unique key consisting of activity_id and owner_id.

Demo

  1. REQUESTOR and PROVIDER

    • Install compiled binary (deb), which consists of:
      • yagna core
      • requestor agent
      • provider agent
      • exe unit
    • yagna service run
    • yagna appkey

  2. PROVIDER

    • Create Offer (default all ram, all disk)

  3. REQUESTOR

    • PAYMENT INIT (CLI)
    • ya-requestor github golem/mandelbrot@latest 4000 4000 10 --timeout 180
    • [ progress bar in command line ]
    • allocation (1GNT)
    • Create Demand
    • Receive offer proposals with provider IDs
    • Request Proposal Acceptances

  4. PROVIDER

    • accepts proposal

  5. REQUESTOR // this is potentially in the outer loop (an agreement per subtask) -- for PoC we have only one subtask

    • Create Agreements
      • and confirms it
    • Request Agreement Acceptances

  6. PROVIDER

    • accepts agreement

  7. REQUESTOR
    (Agreement is accepted synchronously)
    In queue / "loop"

    • Create Activity

    1. PROVIDER

      • SPAWNS EXE UNIT

    2. REQUESTOR

      • Send Exec Command

    3. EXE UNIT

    4. REQUESTOR

      • Gets results
      • Destroy data
      • Destroy activity

    5. PROVIDER

      • DESTROY EXEUNITS

  8. REQUESTOR

    • Terminate Agreement
    • PAYMENT SCENARIO
    • Image appears in the folder
    • We open the image on the demo to show it

  9. PROVIDER

    • We Open Etherscan to show providers' balances

replace random generator with cryptographically secure

currently we use plain uuid v4 for activity_id's

yagna/core/activity/src/common.rs

Lines 48 to 49 in 1c415fc

// TODO: replace with a cryptographically secure generator
Uuid::new_v4().to_simple().to_string()

and even better for identities

yagna/core/identity/src/id_key.rs

Lines 105 to 107 in 1c415fc

fn generate_new_secret(password: Protected) -> (KeyFile, SecretKey) {
let random_bytes: [u8; 32] = rand::thread_rng().gen();
let secret = SecretKey::from_raw(random_bytes.as_ref()).unwrap();

but we probably should use some market as CryptoRng
https://rust-random.github.io/rand/rand/trait.CryptoRng.html

ExeUnit

  1. Spawn ExeUnit // Process has been started
  2. Connect to GSB // Registered on GSB
  3. Receive list of commands from GSB and execute:
    1. Deploy (downloads WASM binary)
      1. Compare Hash
      2. Load into Cache
      3. Return Error if any of above action fails
    2. Start (Validates image)
    3. Transfer (HTTP GET)
      1. Compare Hash
      2. Return Error if any of above action fails
    4. Run
    5. Execute Meta Calls when task is running
      1. Get State
      2. Get Usage
      3. Get Running Command
      4. Get Exec Batch Results
    6. Transfer (HTTP PUT)
    7. Destroy

Payment

  • Driver init (via testnet faucet)
  • Requestor sends an allocation

(Agreement is being created, task is being executed)

  • Provider issues an invoice on task finish
  • Provider sends invoice to Requestor
  • Requestor receives NewInvoice event
  • Requestor reads invoice
  • Requestor accepts invoice and sends InvoiceAccepted
  • Provider receives InvoiceAccepted event
  • Requestor's payment driver makes payment for invoice using allocation, waits for network confirmation and sends "NewPayment"
  • Provider receives NewPayment event
  • Provider verifies payment detials
  • Provider queries account balance (CLI)

Screenshot 2020-02-12 at 12 18 14

Market API 1.5 - GetDemands/GetOffers

Implement GetDemands/GetOffers in Market API.
The said methods are expected to return all active subscriptions raised by Requestor/Provider respectively.
Consider this Market API 1.5.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.