Giter Club home page Giter Club logo

buckyos / cyfs Goto Github PK

View Code? Open in Web Editor NEW
2.0K 215.0 276.0 13.23 MB

CYFS is the next-generation technology to build real Web3 by upgrading the basic protocol of Web (TCP/IP+DNS+HTTP),is short for CYberFileSystem. https://www.cyfs.com/, cyfs://cyfs/index_en.html.

Home Page: https://www.cyfs.com/

License: BSD 2-Clause "Simplified" License

JavaScript 0.27% Rust 99.53% C 0.01% Makefile 0.01% Python 0.02% Shell 0.04% RenderScript 0.01% Objective-C 0.04% Java 0.08%
blockchain cryptography named-data-networking p2p rust web3 named-object-networking protocol ndn non

cyfs's Introduction

CYFS Logo cyfs-lib-ver rust liscence Downloads

CYFS:Next Generation Protocol Family to Build Web3

Introduce

CYFS is the next-generation technology to build real Web3 by upgrading the basic protocol of Web (TCP/IP+DNS+HTTP). It has a subversive architectural design that everyone brings their own OOD (Owner Online Device) to form a truly decentralized network.

We define Web3 as follows:

  • Web3 allows everyone to have rights to save data on the Internet, publish data, and get income from data.
  • Web3's Internet applications are completely decentralized, developers have the right to publish and update applications, and users have the right to use them permanently.
  • Web3's data has been determined ownership when generated, and use Trust Link to link together to form the 'data-ownership network', which is the future semantic web that allows data to show its true value.

After 6 years of research and development, we have achieved the above goals on the CYFS testnet. For the detailed design of the theory and practice of CYFS, please refer to the relevant documents. Open source of CYFS is an important milestone for CYFS, marking that its design has been basically stable and has entered an open community-driven R&D stage. We look forward to having more people join us. We need your passion and strength. Let’s build Web3 that belongs to ourselves together.

This is the repository of the CYFS protocol stack on Github. Although we have built a decentralized GitHub (CodeDAO) on the CYFS testnet to realize the decentralized bootstrapping of CYFS. But before the CYFS mainnet online, we still decided to stay a little longer in WEB2.0 and use this repository as the main community-driven R&D base. After the CYFS mainnet goes online, the entire Repo will move to CodeDAO.

This repository only contains the implementation of the CYFS core system. It does not include the important basic software of the CYFS ecosystem: CyberChat and CYFS Browser, nor does it include SDK implementations for developers of different languages. The following is a list of other important open source projects in the CYFS ecosystem (continuously updated)

  • cyfs-ts-sdk: cyfs-sdk based on typescript to help typescript developers build Web3 DEC apps.
  • CYFS Transmission Lab: A system for the network lab for testing the CYFS protocol (Coming soon)
  • CyberChat: CYFS wallet, manage DID, OOD, DECApp and data assets (Coming soon)
  • CYFS Browser: A browser based on the Chrome kernel that supports the cyfs:// protocol (https://github.com/buckyos/cyfs-browser)
  • CodeDAO: Decentralized Github with CYFS (Coming soon)

Background

CYFS Architecture

The above is the overall architecture diagram of CYFS, which can help to establish a macro understanding of the design of CYFS. The core design of CYFS can be decomposed into the following parts

  • Design of NamedObject and double-section construction of cyfs://o/$ownerid/$objid (Object Link).
  • The BDT protocol improves TCP/IP: the network protocol evolves from address-oriented to trusted identity-oriented.
  • MetaChain upgrades DNS, realizes decentralized addressing, and realizes Data is Account.
  • The Owner deploys its own server OOD (Owner Online Device) on the network, which supports the decentralized accessibility of cyfs://
  • Design proofs of storage and proofs of service based on the game-based consensus theory of dispute before going on the chain, and realize the decentralized high reliability and high availability of cyfs://
  • The off-chain delivery of computing resources is separated from on-chain matching, and an evolvable economic model is used to build decentralized shared cloud computing
  • Data property rights analysis based on the input and results of trusted computing, to realize the evolution from on-chain smart contracts to off-chain Data Exchage Contract (DEC).
  • The Service of Web3 DEC App is installed on everyone's OOD, realizing the decentralization of application services

What key problems are these designs designed to solve? (We believe that "finding the right problem is half done~") You can read "CYFS Architecture" and "CYFS Whitepaper" (coming soon), the amount of content comparison Great, you can understand it with practice~

Start your Web3 journey with “Hello CYFS”

We strongly recommend that you read the "Hello CYFS" series of articles in full to get a complete initial experience of CYFS (the whole process can be completed within 1 hour):

  1. "Compile and deploy DIY-OOD"
  2. "Create Your DID"
  3. "Publish File and Download"
  4. "Tips your friend"
  5. "Publish Website and View"

You can also use the following fast progress to have a first experience of "doing nothing without understand and save time"~

Advance CYFS DEC App develpe tutorial is online

"Decentralized message board development"

Quick Start

The following will build your own Web3 environment by compiling the CYFS source code, and complete the basic experience of Web3 in this environment:

  • Publish a folder to the Web3 world and get a permanent address of cyfs://o/$ownerid/$dirid/index.html which can be accessed by CYFS Browser.
  • Use the cyfs get command to download the files in the folder you just published.

Note that this environment is normally not isolated, but can be added to the CYFS network (we have planned 3 networks: nightly, beta, relase, currently only nightly is available), so the following tutorial does not include compiling MetaChain with source code +SN part. If you plan to build a complete system in an independent environment and need to compile and deploy MetaChain super nodes, please read the document "MetaChain Construction" (Coming soon).

Compile

Prepare an idle Linux host, preferably a VPS, with more than 2G of memory. Preparation:

  • Node.js 16 or above
  • rustc 1.63 or above

Run the following command to compile the OOD System.

cd ${cyfs_sources_dir}/src
npm i
node ../scripts/build-standalone.js

During the compilation process of the script, the ${cyfs_root} folder needs to be created. If the current system or current user does not have permission to create this folder, you need to manually create the ${cyfs_root} folder in advance, and assign read and write permissions to the current user. (In the official environment, all components and data of the OOD System will be installed to ${cyfs_root})

${cyfs_root} path:

  • Windows: c:\cyfs
  • MacOS: ~/Library/cyfs
  • Other systems: /cyfs

Compile successfully generates ood-installer

Install the newly compiled OOD System

Before installation, you need to prepare the depends:

  • Node.js 16 and above
  • MongoDB 4.4 version, configured to boot, use the default port, no authentication (using SQLite as the object storage engine can not rely on MongoDB, the subsequent installation script will support the selection of the storage engine)
  • The latest version of docker-ce, configured to start at boot

Find the newly compiled ood-installer in the src directory and execute

./ood-installer --target solo

After a minintes , the installation is complete.

Activate OOD

In order to experience the process, the cli tool is used here to complete the D.I.D creation process. **D.I.D created based on the cli tool can only be used for testing purposes! **

  1. Install cyfs-tool: use the command line npm i -g cyfs-tool-nightly to install the nightly version of the cyfs-tool tool
  2. Generating sets of identities

    Use the command cyfs desc -s <save_path> to generate a matching identity file and save it in the save_path directory. If save_path is not specified, it defaults to ~/.cyfs_profile

  3. Bind OOD

    After the identity is generated, copy the two files <save_path>/ood.desc and <save_path>/ood.sec to ${cyfs_root}/etc/desc on the OOD machine and rename it to device. desc and device.sec

  4. Bind CYFS-Runtime

    After the identity is generated, copy the two files <save_path>/runtime.desc and <save_path>/runtime.sec to ${cyfs_runtime_root}/etc/desc on the CYFS browser machine and rename them to device.desc and device.sec

${cyfs_runtime_root} specific path:

  • Windows: %appdata%/cyfs
  • Mac OS: ~/Library/Application Support/cyfs

Publish your first Web3 website

First prepare the www directory of your website, let's first experience publishing static websites to Web3, and follow-up documents for the construction of dynamic websites will be introduced.

use command

cyfs upload <dir_path> -e ood -t ood

Add the file pointed to by the local <dir_path> to the OOD. The command execution is complete, the local www directory has been uploaded to OOD and the unique URL to the Web3.0 website has been generated (the end of the command execution). The link is cyfs O-Link, which looks like this cyfs://o/$ownerid/$objid, where $objid is the ContentId of the directory.

Browse the website just released

Use the command on any machine with cyfs-tool installed

cyfs get cyfs://o/$ownerid/$objid/index.html

You can download the just-released official website.

Any machine with a cyfs browser installed can use the cyfs browser to open cyfs://o/$ownerid/$objid/index.html and browse the website just released. For the download of cyfs browser, see here

Code guide

Through the above process, you have a basic understanding of the design and use of CYFS. Although the design of CYFS is basically stable, we still have a lot of code to write. We are very eager for your help, but certainly not too much energy to write documentation (details are in the source code~). Here we do a minimalist code introduction, hoping to help you understand the implementation of CYFS faster.

According to the architecture, we know that the core of cyfs:// is the construction and acquisition of Object Linke, and the premise of acquisition is to upload data to OOD at least. The process is as follows:

  1. Start the local protocol stack (cyfs-rutnime)
  2. Calculate the hash of all files in the directory locally, and construct a FileObject with the current PeopleId as the Owner
  3. Generate a Map structure (directory structure) locally, and construct a MapObject with the current PeopleId as the Owner, At this time, cyfs:// has been constructed, but the cyfs:// cannot be accessed at this time
  4. Add the above named objects and named data to the local stack
  5. Initiate CYFS PUT operation to OOD: save the MapObject to OOD and set the access permission to public
  6. Let OOD start MapObject Prepare and save a named data on OOD
  7. MapObject Prepare is completed on OOD, cyfs:// can be accessed

Then the process of using cyfs get to obtain is as follows:

  1. Start the local protocol stack cyfs-runtime
  2. Use the HTTP protocol to initiate an HTTP GET request to cyfs-runtime
  3. cyfs-runtime checks whether the object exists in the local cache
  4. cyfs-runtime initiates a NamedObject query request (the following behaviors are not serial)
    • 4.1 Query NamedObject from OOD
    • 4.2 OOD query local, whether NamedObject exists
    • 4.3 OOD queries MetaChain, whether NamedObject exists
    • 4.4 OOD queries whether the NamedObject exists on the previous hop device according to the Reference information in get
    • 4.5 OOD queries the configuration of Object's Owner Zone through MetaChain
    • 4.6 OOD is configured through Zone, connected to NamedObject's OOD, or connected to NamedObject' Cache, and query NamedObject
  5. After getting the ChunkId, cyfs-runtime calls the Channel interface (NDN semantic interface) of the BDT to request the Chunk
    • 5.1 For the first, small Chunk, get it directly from the associated OOD
    • 5.2 For the second Chunk, it will try to get it from the previous hop (Reference OOD)
    • 5.3 BDT will try to perform multi-source search and multi-source download based on fountain code based on the context information of the application layer
    • 5.4 The router can identify the Chunk request packets sent by the BDT, intercept and forward them, and further optimize the overall load of the network
    • 5.5 Only OOD will upload Chunk
  6. When the first Chunk of the FileObject is ready (passed authentication), the HTTP GET request in step 1 starts to return data

(The above process is shown in the figure below) get-from-cyfs

For the overall design of the CYFS protocol, please refer to Introduction to CYFS Protocol.md

After understanding the logic of the above process, you can read the relevant code according to the following guidelines.

Upload

  1. Start the local protocol stack: util.ts: create_stack()
  2. Construct FileObject: file_recorder.rs: FileRecorder.add_file()
  3. Construct ObjectMap: publish_manager.rs: PublishLocalDirTask.publish()
  4. Add the above Named-Objects and Named-Data to the local protocol stack: file_recorder.rs: FileRecorder.record_file_chunk_list()
  5. Initiate a CYFS PUT operation to the OOD: save the MapObject to the OOD and set the access permission to public: upload.ts: upload_obj()
  6. Let OOD start MapObject Prepare and save Named-data(Chunks) on OOD: upload.ts: run()

Get

  1. Start the local protocol stack: util.ts: create_stack()
  2. Initiate an HTTP GET request using the HTTP protocol
  3. cyfs-runtime checks whether the object exists in the local cache
  4. cyfs-runtime initiates NamedObject query requirements (the following behaviors are usually not serial)
    • 4.1 Query NamedObject from OOD
    • 4.2 OOD query local, whether NamedObject exists
    • 4.3 OOD queries MetaChain, whether NamedObject exists
    • 4.4 OOD queries whether the NamedObject exists on the previous hop device according to the Reference information in GET
    • 4.5 OOD queries the configuration of Object's Owner Zone through MetaChain
    • 4.6 OOD is configured through Zone, connected to NamedObject's OOD, or connected to NamedObject' Cache, and query NamedObject
  5. After get the ChunkId, cyfs-runtime calls the Channel interface (NDN semantic interface) of the BDT to request the Chunk
    • 5.1 For the first and small Chunk, get it directly from the associated OOD
    • 5.2 For the second Chunk, it will try to get it from the previous-jump (Reference OOD)
    • 5.3 BDT will try to perform multi-source search and multi-source download based on fountain code based on the context information of the application layer
    • 5.4 The router can identify the Chunk request packets sent by the BDT, intercept and forward them, and further optimize the overall load of the network
    • 5.5 Only OOD will upload Chunk
  6. When the first chunk of the FileObject is ready and verified, the HTTP GET request in step 1 starts to return data

DecApp development practice

The following series of articles describes how to implement a message board DecApp based on CYFS. this is a long series of articles detailing the general process of developing a DecApp

  1. Introduction
  2. DEC App basic principle
  3. Implement your own message service
  4. Run and debug your own message service
  5. Extended reading: Further explanation of the principles and introduction of some tools
  6. Implement the front end
  7. Send it to your friends and understand the CYFS permission system
  8. Use the emulator to test the app across zones
  9. Extended reading: Decentralized, trusted data, semantic data, data property rights
  10. Add comments
  11. DEC App data security and high availability
  12. Extended reading: Learn more about data exchange contracts and DAOs
  13. Summary

Directory Structure

You have learned from the architecture diagram that the project implementation of CYFS is not small. To prevent you from getting lost, this chapter is a small map that will help you get a basic understanding of the CYFS code structure and understand where the key code will be.

  • src //source directory, many development commands require this directory to be the current directory for operations
    • service // contains the code of a series of important executable files, which is a good starting point for reading the code
      • gateway //nginx of CYFS system, the most important basic service on OOD
      • cyfs-runtime //CYFS user mode protocol stack implementation, the most common process in development and debugging
      • ood-daemon //The basic daemon of OOD, keep alive and automatically update other basic services
      • app-manager //Installation manager of DEC App
      • chunk-manager // The most primitive and simple thinking and implementation of the concept of CYFS NDN, currently used for the lowest basic services
      • file-manager // The most primitive and simple thinking and implementation of the concept of CYFS NON, currently used for the lowest-level basic services
      • cyfs-runtime // The backend of the CYFS protocol stack used by the CYFS browser, based on the standard CYFS protocol stack, provides some functions that are convenient for the browser to use
    • component
      • cyfs-base //The basic component shared by all projects, here you can see the implementation details of NamedObject
      • cyfs-base-meta // MetaChain's base object definition
      • cyfs-bdt //The implementation of the BDT protocol, be careful of the huge scale when expanding
      • cyfs-core // Definition of core objects in the CYFS system
      • cyfs-chunk // CYFS protocol stack supports NDN protocol
      • cyfs-stack //The implementation of the CYFS protocol should be the largest component in the system
      • cyfs-meta-lib //Related implementation of MetaChain Client
      • cyfs-mobile-stack //Encapsulation of CYFS mobile protocol stack
      • cyfs-noc //NamedObject storage engine implementation (SQLite+MongoDB), which is the component that has the greatest impact on system IO performance
    • test //Test project, only includes the parts that have been sorted out, the code for a large number of network tests of CYFS is in the CYFS Transmission Lab. -tools //cyfs tools. The design idea of ​​cyfs toolchain is a top-level script and a series of independent gadgets, taking into account the consistency of cli experience and the independence of project construction
  • scripts //Compile script directory, we use js script to replace makefile
  • doc //document directory

Roadmap

CYFS is currently in the state of dev_testnet (nightly), in this state, all components will be affected by the daily build results by default, and we do not consider how the data structure is backward compatible, and the protection of data security is not enough. whole. This version should only be used for development testing and should not be used in production production.

Our next goal is to complete the launch of the testnet (beta). Relatively nightly, beta will specify strict release standards, and will make the data structure as backward compatible as possible, and can confidently protect users' data security. The stability of beta will be able to support the release of beta products~ The main difference from the release version is assets is fake assets, and there are gaps in stability and performance.

We hope to launch the CYFS mainnet within 18 months. There is no definite launch time for the CYFS mainnet. As long as the asset security, stability and performance of the testnet reach the preset goals, we will enter the release channel of the mainnet.

Welcome to follow CYFS-Stack & Services to learn more about CYFS Roadmap~

Contributing

The dev_testnet of CYFS is designed and implemented by Shenzhen BuckyCloud. The core team of Shenzhen BuckyCloud is inherited from the original Xunlei(NASQ:XNET) infrastructure R&D team. We have rich experience in network protocol and infrastructure development, as well as super-large-scale P2P network design and tuning. experience. After CYFS is open source, BuckyCloud has completed its key historical mission. Now we prefer to call ourselves CYFS CoreDevTeam. In the future, it will be an open team organized based on DAO. The main responsibility is to promote the continuous research and development of CYFS. We welcome all engineers to join:

  • If you have rich experience in network protocol development, at this moment we need you to help us improve the core protocol together
  • If you are very familiar with open source culture and open source community, you can join decentralized github (CodeDAO) with us
  • If you have rich experience in blockchain development, MetaChain, which is waiting to be rewritten at the moment, urgently needs you~ We already have an ambitious design, but over the years we have focused on off-chain protocol development, block-chain development resource is very limited
  • If your main language does not include typescript and rust, you can help us build SDK for other languages
  • If you are very good at the product, you can help us improve the basic experience of CYFS, or create a real Web3 Startup on CodeDAO
  • Of course, everyone is welcome to submit bugs, suggestions for product improvement, and help us revise documents.... We guarantee that your contributions will be remembered

We are designing a DAO-NFT-based contribution record system. When this Repo is move to CodeDAO as a whole, CYFS DAO Tokens can be issued according to the share size of contributors (fairness is difficult)

License

BSD-2-Clause License
Copyright (c) 2023, CYFS Core Dev Team.

cyfs's People

Contributors

alexsunxl avatar dependabot[bot] avatar glen0125 avatar jing-git avatar jinquantianxia avatar lurenpluto avatar photosssa avatar song0125 avatar streetycat avatar tracy101 avatar waterflier avatar weiqiushi avatar winevery avatar zwong91 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cyfs's Issues

Improve Perf module

Improve Perf module

Goal

  • Multi report target, Default local_cache of the local cyfs stack
  • Use local cache to store stat information
  • Easy to use
  • Start using the Perf module inside OOD service and CYFS stack

Super frame datagram

  • support super frame datagram: send/receive datagrams beyound UDP MTU;
  • support plaintext datagram;

StateStorage implementation

Add the implementation of the basic data structure based on ObjectMap to realize the support of data storage and access such as FriendList

Standby ood failed to sync chunk data

Describe the bug

the standby ood just sync the object_map ,not sync chunk data
To Reproduce

(1) use SharedCyfsStack ndn_service put a chunk to master ood
(2)master ood's root_state add a ObjectMap of chunkId
(3) waitting for standby ood sync data complete from master OOD
(4) check standby ood can use SharedCyfsStack ndn_service get the chunk
(5)the standby ood root_state revision is sync ,but get chunk data failed, ndn get_chunk from local but not found!
Err(err: (4, ndn get_chunk from local but not found! id=7EYaFPU2ry4McmHUJtxz1iDKEyPfkSaoDz7AFuxiDFJv, undefined))
Expected behavior

(1) the standby ood root_state revision is sync , the standby ood should get chunk from NOC

[Congestion Control] The performance problem of BDT TCP file transmission in high latency and high packet loss

【Problem Description】
Multinational network Shanghai, China DCFS <-> US VOOD BDT only uses TCP protocol and the transmission fails
Network environment: 20-30% packet loss, 150ms delay, 30MB bandwidth, BDT transmission rate of 20-50KB/s, there is a small probability of transmission failure

【Modify system configuration optimization plan】

Linux kernel configuration to enable TCP congestion control can definitely improve performance
vim /etc/sysctl.conf

net.core.default_qdisc = fq 
net.ipv4.tcp_congestion_control = bbr

【Lab Verification】
0% packet loss 0ms delay bandwidth 30MB speed 3MB/s
0% packet loss 150ms delay bandwidth 30MB speed 1MB/s
20% packet loss 150ms delay bandwidth 30MB speed 500KB/s

Add additional sync support to DirObject

Due to the particularity and complexity of DirObject, there are multiple layers of Object nesting and Chunk interleaving. Therefore, the synchronization mechanism needs to be considered additionally and corresponding implementations must be added.

App-Manager support for OOD standby mode

In the standby mode of OOD, App-Manager should not start the app service, and should not modify the root-state (because it is read-only mode), and when the OOD working mode is switched, all app services need to be stopped

A more concise solution is that when the mode is switched, the OOD-daemon is responsible for stopping the App-Manager, and the App-Manager needs to stop all currently running app services under the --stop command.

NDN get_data improvements and optimizations for local data access

  1. Integrate the existing get_data support with ChunkManager, ndc and tracker, which is consistent with the achievements of the BDT layer
  2. Optimize the reading mechanism of local file chunks, improve the performance of delayed reading of multiple blocks and the reading part with a range

Improve the event mechanism of zone role switching

At present, zone role switching is a built-in event of role manager, which can only be used in the same process of CyfsStack. It needs to be switched to rely on the standard RouterEvent mechanism to provide cross-process access mode. For example, ood-daemon/app-manager will also pay attention to this event.

Window ood-installer 重新启动导致的gateway.toml被覆盖的bug

Describe the bug

CYFSOOD-x86-64-1.0.0.505-nightly安装包。WIN10专业版,安装配置完成后,再次重新启动机器,不能正确启动服务导致CYFS_browser中,info不能正确的显示OOD的信息。同时会每隔30秒启动一次服务(应该是cyfs-drive-server.exe)
To Reproduce

1,确保本机没有安装mogodb
2,下载 diy-ood-window版CYFSOOD-x86-64-1.0.0.505-nightly安装包,安装
3,使用https://youtrack.buckycloud.com/articles/SHARES-A-2/CYFS开发环境搭建 (注意host修改)中提供的 cyfs身份创建工具 中的,gateway.toml,替换 c:\cyfs\etc\gateway\gateway.toml 这个文件。此文件中,数据库配置为sqlite
4,关联账号
5,打开browser,可以看到ood的info信息是正确的
6,关机重新启动,ood-installer 又被自动执行。
7,c:\cyfs\etc\gateway\gateway.toml 这个文件被重新覆盖,其中的数据库配置又被修改为mongodb。
8,启动browser,不能正确显示ood 信息,同时会不停的启动某个服务(一闪而过)。

Expected behavior

再次重启启动,运行ood-installer ,不应该覆盖被修改的配置文件
System information

ftp 服务上下载 的CYFSOOD-x86-64-1.0.0.505-nightly安装包

OS:win10

Additional context

BDT-Stream Direct failure between BDT LAN devices

【Test Node EP】
PC_0015_0 L4tcp192.168.1.145:38549
PC_0016_0 L4tcp192.168.1.142:45531
【Steps】

  1. PC_0015_0, PC_0016_0 do not use SN
  2. PC_0015_0 -> PC_0016_0 ,remote_desc contain EP L4tcp192.168.1.142:45531
  3. PC_0016_0 -> PC_0015_0 , remote_desc contain EP L4tcp192.168.1.145:38549
    【expected outcome】
  4. The connection is successful
    【actual results】
    1.The connection is failure . if not SN,Direct just can use WAN address

GlobalState op-env adds new methods support

On the basis of the existing interface, the following new interfaces and features are added

update

Unlike commit, update can be called multiple times. If something changes internally, it will try to commit and update the global state, returning the latest root
update does not affect the holding status of op-env on lock, so the lock on the specified path can continue to be valid (until commit or abort)

get_current_root

Get the root state of the current op-env, SingleOpEnv and PathOpEnv will return the corresponding values

SingleOpEnv -> root object

PathOpEnv -> (root, revision, global_root)

It should be noted that for PathOpEnv, calling get_current_root will make the op-env immediately bind the corresponding global-state snapshot (but lock will not)

reset

For SingleOpEnv, reset the current iterator

Datagram broadcasting

  • enable DHT routing map on BDT stack;
  • implement datagram broadcasting with DHT;
  • filter broadcating with area code in device object;

NDN refactor for new ACL and load logic

Loading logic of dir+inner-path

dir is the same as objectmap, the loading is changed to the full local protocol stack, that is, dir+inner_path is loaded from the target protocol stack, and the subdir, chunk and file that a dir depends on must be completely in the same protocol stack, and the internal nodes are no longer support complicated search across protocol stacks (consistent with objectmap loading and operation behavior)

NDN

ndn loads data, supports three types

  • chunk_id
  • file_id
  • dir_id+inner-path

The new ACL needs to match the following fields

  • referer_object
  • req_path

Several key designs

  • The minimum granularity of permission control is the file_id/dir_id corresponding to the referer_object, or the chunk_id/file_id/dir_id in the direct request (if the referer_object is empty), finer granularity is not supported
  • A request supports at most one referer_object to specify the association with the request target; or if no referer_object is specified, multiple referer_objects are no longer supported
  • Bridge target_object_id and req_path through referer_object to determine the permission of target_object_id, so the permission determination is basically the following two modes
    • referer_object + target_object_id + req_path triple
    • target_object_id + req_path 2-tuple

for chunk-id

Load a chunk data directly, check the permissions as follows

  1. referer_object is empty
    Then use req_path+chunk_id directly for verification, that is, require chunk_id to be linked to root_state
  2. referer_object = file_id
    First check if chunk_id exists in file.chunklist
    Then use req_path+file_id for verification, that is, the referenced file_id is required to be linked to root_state
  3. referer_object = dir_id + inner_path
    First get the corresponding file_id/dir_id through dir_id + inner_path
    Check if chunk_id is in file.chunklist or embedded chunk in dir
    Then use req_path+file_id or req_path + dir_id for verification, that is, the referenced file_id/dir_id is required to be linked to root_state

for file_id

Load a file data directly, verify the permissions as follows

  1. referer_object is empty
    Then use req_path+file_id directly for verification, that is, file_id is required to be linked to root_state
  2. referer_object = dir_id + inner_path
    First get the corresponding file_id through dir_id + inner_path
    Check if chunk_id is in file.chunklist
    Then use req_path + dir_id for verification, that is, the referenced dir_id is required to be linked to root_state and meet the permission requirements

For dir_id+inner_path

Load a file corresponding to the internal path of dir, and the permission check is as follows
The referer_object parameter is not accepted in this mode, and the dir_id itself is used directly
Use req_path + dir_id for verification, that is, the referenced dir_id is required to be linked to root_state and meet the permission requirements

SN Online Timeout

Describe the bug
cyfs stack wait sn online 30s timeout
run open bdt satck 166498 times success , timeout 237 times
To Reproduce

let stack = Stack::open(
            local, 
            key, 
            params).await?;

        let acceptor = stack.stream_manager().listen(0).unwrap();

        let peer_impl = unsafe {
            &mut *(Arc::as_ptr(&self.0) as *mut PeerImpl)
        };
        peer_impl.lazy_components = Some(LazyComponents {
            stack, 
            acceptor
        });
let begin_time = system_time_to_bucky_time(&std::time::SystemTime::now());
                    let mut local = peer.get_stack().local();
                    log::info!("on create succ, local: {}", local.desc().device_id());
                    let mut result = 0 ;
                    if sn.len()>0{
                        log::info!("peer sn list len={},wait for sn online",sn.len());
                        let result = match future::timeout(Duration::from_secs(30), peer.get_stack().net_manager().listener().wait_online()).await {
                            Err(_) => {
                                log::error!("sn online timeout");
                                1000
                            },
                            Ok(_) => {
                                log::info!("sn online success");
                                0
                            }
                        };
                    }
                    let online_time = system_time_to_bucky_time(&std::time::SystemTime::now()) - begin_time;

The parameters of A link should be maintained after transform.

Describe the bug
The parameters will be removed when I open an A link html file.

To Reproduce
For example, I will open the following link:
cyfs://a/9tGpLNna8UVtPYCfV1LbRN2Bqa5G9vRBKhDhZiWjd7wA/index.html?hello=world
once I open it, the link where transform to like this :
http://127.0.0.1:38090/o/5hLXAcNtoFKLkvPEmdMFDJ9NjpCWJxdSZU6uKXPkR6m9/95RvaS5cWCTCKejHojmHtauJTQwL7FZ1AMeLLba1Dk1w/index.html?dec_id=9tGpLNna8UVtPYCfV1LbRN2Bqa5G9vRBKhDhZiWjd7wA
which means I've lost my parameter.

Expected behavior
In fact, the result should be like this:
http://127.0.0.1:38090/o/5hLXAcNtoFKLkvPEmdMFDJ9NjpCWJxdSZU6uKXPkR6m9/95RvaS5cWCTCKejHojmHtauJTQwL7FZ1AMeLLba1Dk1w/index.html?dec_id=9tGpLNna8UVtPYCfV1LbRN2Bqa5G9vRBKhDhZiWjd7wA?hello=world

Windows bash environment cross compile cyfs_raptorq error

After compiling the linux version of the cyfs project using windows bash environment , and then using windows cmd environment to compile the cyfs project, an error will appear in the compilation of the cyfs_raptorq module

error[E0786]: found invalid metadata files for crate `cyfs_raptorq`
  --> CYFS\src\component\cyfs-bdt\src\ndn\chunk\encode\raptor.rs:10:5
   |
10 | use cyfs_raptorq::{
   |     ^^^^^^^^^^^^
   |
   = note: invalid metadata version found: \\?  CYFS\src\target\release\deps\libcyfs_raptorq.rlib

After deleting the "target" directory, windows cmd compile pass

NOC/NON/ACL redesign and refactoring

Mainly include the following two core changes

  1. Refactor the underlying structure and implementation of NOC, introduce permission design and GC mechanism support, etc.
  2. Refactor the NON interface to work with the new NOC and authority mechanism

CYFS DSG Cache Service

DSG Contract support cache service:

  • Cache storage fields;
  • Global cache storage option for cache node;

Prepare cached chunk list:

  • DSG cache aggregation objects: traffic proof, chunk traffic;
  • Update chunk list;

Stream send data content is incorrect

Describe the bug

(1)LN connect RN second success
(2)LN use stream send 1M data to RN
(3)RN resp 1M data to LN,
LN recv data content is incorrect, the content [0..8] is remark data length ,RN send [0..8] length is 1048576 ,LN recv length is 5941175098377794146
To Reproduce

    pub async fn send_file(&mut self, size: u64) -> Result<HashValue, BuckyError> {
        let mut hashs = Vec::<HashValue>::new();
        let mut send_buffer = Vec::new();
        send_buffer.resize(PIECE_SIZE, 0u8);
        let mut gen_count = PIECE_SIZE;
        let mut size_need_to_send = size + 8;
        if gen_count as u64 > size_need_to_send {
            gen_count = size_need_to_send as usize;
        }
        send_buffer[0..8].copy_from_slice(&size_need_to_send.to_be_bytes());
        Self::random_data(send_buffer[8..].as_mut());

        loop {
            let hash = hash_data(&send_buffer[0..gen_count]);
            hashs.push(hash);

            let _ = self.stream.write_all(&send_buffer[0..gen_count]).await.map_err(|e| {
                log::error!("send file failed, e={}",&e);
                e
            });
            size_need_to_send -= gen_count as u64;

            if size_need_to_send == 0 {
                break;
            }

            gen_count = PIECE_SIZE;
            if gen_count as u64 > size_need_to_send {
                gen_count = size_need_to_send as usize;
            }
            Self::random_data(send_buffer[0..].as_mut());
        }

        let mut total_hash = Vec::new();
        for h in hashs.iter() {
            total_hash.extend_from_slice(h.as_slice());
        }
        let hash = hash_data(total_hash.as_slice());

        log::info!("send file finish, hash={:?}", &hash);

        Ok(hash)
    }
pub async fn recv_file(&mut self) -> Result<(u64, HashValue), BuckyError> {
        let mut hashs = Vec::<HashValue>::new();
        let mut recv_buffer = Vec::new();
        recv_buffer.resize(PIECE_SIZE, 0u8);
        let mut piece_recv: usize = 0;
        let mut file_size: u64 = 0;
        let mut total_recv: u64 = 0;
        loop {
            let len = self.stream.read(recv_buffer[piece_recv..].as_mut()).await.map_err(|e| {
                log::error!("recv failed, e={}", &e);
                e
            })?;
            if len == 0 {
                log::error!("remote close");
                return Err(BuckyError::new(BuckyErrorCode::ConnectionReset, "remote close"));
            }
            piece_recv += len;
            total_recv += len as u64;

            if file_size == 0 {
                if piece_recv < FILE_SIZE_LEN {
                    continue;
                }
                let mut b = [0u8; FILE_SIZE_LEN];
                b.copy_from_slice(&recv_buffer[0..FILE_SIZE_LEN]);
                file_size = u64::from_be_bytes(b);
                log::info!("=====================================file_size={}", file_size);
            }

            if file_size > 0 {
                if total_recv == file_size || piece_recv == PIECE_SIZE {
                    let recv_hash = hash_data(&recv_buffer[0..piece_recv].as_ref());
                    hashs.push(recv_hash);
                }

                if total_recv == file_size {
                    log::info!("=====================================recv finish");
                    break;
                }
            }

            if piece_recv == PIECE_SIZE {
                piece_recv = 0;
            }
        }

        let mut total_hash = Vec::new();
        for h in hashs.iter() {
            total_hash.extend_from_slice(h.as_slice());
        }
        let hash = hash_data(total_hash.as_slice());
        log::info!("recv file finish, hash={:?}", &hash);
        Ok((file_size, hash))
    }

文檔建議

強迫症提示:建議所有關鍵字統一規範大小寫書寫風格,中文和英文字符串之間增加空格來達到更有好的可讀性,典型的案例如:

舊:订阅基于VPS实现的立即可用的VirtualOOD, VirtualOOD是门槛最低开通速度最快的OOD方式,相信会是大部分普通用户第一次的选择。

新:订阅基于 VPS 实现的立即可用的 Virtual OOD,Virtual OOD 是门槛最低开通速度最快的 OOD 方式,相信会是大部分普通用户第一次的选择。

另外,很多操作步驟沒有命令行實例,如:身份生成后,复制<save_path>/ood.desc和<save_path>/ood.sec两个文件到OOD机器的${cyfs_root}/etc/desc下,并重命名为device.desc和device.sec,這裡可以考慮用 cp 命令或者 cp 後再使用 mv 命令的完整提示。

Add mechanism to clear handlers based on dec

Due to the persistence mechanism of the handler, after dec modifies the handler, if you forget to remove the old handler, there will be residues that will affect the handler mechanism. So consider adding an interface for clearing the handler based on dec granularity. After dec starts or when necessary, you can call the clearing history version once.

Optimized compilation speed

Speed up the compilation of ood service

channel.rs moved from cyfs-base to new project cyfs-version
Move all logging-related code from cyfs-debug to the new project cyfs-log
keep the old dead-related code in cyfs-debug

This way, every time you change the version number, channel, etc. and recompile, you don't need to compile cyfs-base, cyfs-bdt, cyfs-lib and other libraries that take a long time, but only the final binary library, which can greatly speed up the compilation speed.


加快ood service的编译速度

channel.rs从cyfs-base里移出到新建工程cyfs-version
所有log相关的代码从cyfs-debug里移出到新建工程cyfs-log
cyfs-debug里保留原有的check相关代码

这样,每次修改版本号重编译的时候,就不需要再编cyfs-base, cyfs-bdt, cyfs-lib等需要很长时间的库了,而只需要编译最终的二进制库,可以大大加快编译速度

BDT packet version campatible

  • Version field in handshake packets;
  • Version property on tunnel;
  • Semantic session implemention differs from versioned tunnel;

Proxy node service proof

implement service proof to support a OOD running as proxy node, to help devices behind NAT which failed to play hold-punching, or speed up link between devices

Refactor cyfs-stack front-end support for protocol layer

The purpose of the task
In order to better and systematically improve the runtime layer's support for the protocol, solve some leftover problems and inconsistent errors in the past historical evolution, and better support the o/r/a protocol

The core changes are as follows

  • Add the Front module, deconstruct the existing translator processing mechanism, and uniformly handle the o/r/a protocol
  • The non/ndn/global-state processing mechanism of cyfs-stack strips the support for runtime protocol layers such as mode and format, and the function is more explicit and pure

Spend time
About one to two weeks, including testing

NDN Transfer Group basis

NDN Transfer Group is designed for named data transfer sessions, which may devices request some same named datas at same time; each transfer group could have different strategy for requested named data feature differs, BDT has to design transfer group basis to support making these strategies:

  • more fields on existing NDN package(e.g. interest package), or new defined NDN package to support transfer group member communication;
  • design basic roles in transfer group: tracker, hub, seeder etc.
  • design basic actions in transfer group member.

BDT support scale out for NDN request

  • create download session for data piece returned from a channel which is different from where interest is sent;
  • add a from field to Interest packet: if interest is transmit from one device to another device, this field points to orign device where interest if sent from;

NDN Cache Node support in BDT

interest package routing semantic

  • foward interest package to another device;
  • reply interest with another source;
  • reply interest with waiting;

scheduling upload tasks

  • resource usage statistics for upload tasks
  • resource quota for upload tasks, when reaches, reply waiting to all incoming interests

service proof of traffic for cache node

  • embed service proof fields in NDN packages

Hello CYFS Guide Feedback

Describe
This issue is to discuss the Hello CYFS Guide. If you have any issues with or feedback on the document, please leave a comment here.

Note: this is for feedback on the document, not on CYFS or on the CYFS changes.

Thank you for your efforts!

Device add bdt_version field

For the sake of bdt protocol version compatibility, an additional field needs to be added to indicate the bdt protocol version of the protocol stack used by the current device.

Device body content needs to add a field as follows:
bdt_version: Option<u8>

Compatibility between old and new binary data layers needs to be considered

danger for `import-people`

Exporting identities to the developer's computer is dangerous, and we may need a tool that authorizes publishing; exporting identities is only an option that is convenient for the development phase.

Need ood-daemon-stop.sh on DIYOOD

The purpose of the feature
I want stop the ood service and backup the /cyfs/data/, now I killed service process manually

Describe the solutions to the feature you'd like
I need start/stop scripts~

SharedCyfsStack.util().build_dir_from_object_map parameter input req_path will cause the request to fail

Describe the bug

build_dir_from_object_map filling in req_path will cause the request to fail

To Reproduce

let dir_obj_resp1 = await stack.util().build_dir_from_object_map({
common: {
req_path: "qaTest",
target : stack.local_device_id().object_id,
dec_id: ZoneSimulator.APPID,
flags: 0,
},
object_map_id: get_ret1.unwrap().object.object_id,
dir_type: cyfs.BuildDirType.Zip,
});

result:

Error: err: (4, request not handled: method=POST, path=/util/build_dir_from_object_map/qaTest, undefined)
let dir_obj_resp1 = await stack.util().build_dir_from_object_map({
common: {
req_path: "",
target : stack.local_device_id().object_id,
dec_id: ZoneSimulator.APPID,
flags: 0,
},
object_map_id: get_ret1.unwrap().object.object_id,
dir_type: cyfs.BuildDirType.Zip,
});

result:

{"ok":true,"err":false,"val":{"file_id":"95RvaS5aUeMomNs3LBR89YwU6b3Aj7Tm5aQJPsJfH8SC"}}

Expected behavior
req_path is an optional parameter in UtilOutputRequestCommon, it should be able to be filled in successfully

Add list method to op-env

Now op-env supports iterator mode, which has better support for the data structure of large mode, but for ObjectMap in most cases, it is more convenient and efficient to use list directly

OOD-daemon supports read-only mode of gateway service

OOD-daemon needs to deal with the ood working mode of the gateway, mainly in the case of readonly in the current standby-ood mode, the app-manager service needs to be stopped, including the monitoring mechanism of dynamic switching, etc.

brief information in object-id

Now to save any information, we need to construct a special object, which is too complicated when the amount of information is small. We can design an object, and the information is directly encapsulated in "Object-Id", such as: number, short byte stream, short string etc.

gateway unwrap

Problem
[2022-09-08 12:42:08.567990 +00:00] INFO [component/cyfs-debug/src/panic/panic.rs:32] thread 'async-std/runtime' panicked at 'called Option::unwrap() on a None value': component/cyfs-stack/src/interface/http_ws_listener.rs:88

To Reproduce
vood batch and concurrent op_env

Expected behavior
gateway work well

System information
Intel Xeon Processor (Skylake, IBRS) 2 Core
RAM: 4GB
OS: Ubuntu20.4

Additional context

[2022-09-08 12:42:08.567524 +00:00] INFO [component/cyfs-debug/src/panic/panic.rs:31] thread 'async-std/runtime' panicked at 'called Option::unwrap() on a None value': component/cyfs-stack/src/interface/http_ws_listener.rs:88
0: 0x0000564ec338d390
1: 0x0000564ec337b690
2: 0x0000564ec31ea950
3: 0x0000564ec3926000
4: 0x0000564ec3925f30
5: 0x0000564ec3924710
6: 0x0000564ec3925cd0
7: 0x0000564ec1dfa110
8: 0x0000564ec1df9fc0
9: 0x0000564ec23da310
10: 0x0000564ec2e52ed0
11: 0x0000564ec2e97c80
12: 0x0000564ec2e4d2c0
13: 0x0000564ec2ed9a30
14: 0x0000564ec38ba180
15: 0x0000564ec38babc0
16: 0x0000564ec38b88f0
17: 0x0000564ec38af430
18: 0x0000564ec38ae980
19: 0x0000564ec38b02d0
20: 0x0000564ec392b840
21: 0x00007f152b671530
22: 0x00007f152b44112d
23: 0x0000000000000000
[2022-09-08 12:42:08.567990 +00:00] INFO [component/cyfs-debug/src/panic/panic.rs:32] thread 'async-std/runtime' panicked at 'called Option::unwrap() on a None value': component/cyfs-stack/src/interface/http_ws_listener.rs:88
0: cyfs_debug::panic::manager::PanicManager::start::{{closure}}
1: std::panicking::rust_panic_with_hook
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:702:17
2: std::panicking::begin_panic_handler::{{closure}}
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:586:13
3: std::sys_common::backtrace::__rust_end_short_backtrace
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:138:18
4: rust_begin_unwind
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5
5: core::panicking::panic_fmt
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14
6: core::panicking::panic
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:48:5
7: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
8: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
9: std::thread::local::LocalKey::with
10: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
11: async_task::raw::RawTask<F,T,S>::run
12: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
14: async_io::driver::block_on
15: async_global_executor::threading::thread_main_loop
16: std::sys_common::backtrace::__rust_begin_short_backtrace
17: core::ops::function::FnOnce::call_once{{vtable.shim}}
18: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
std::sys::unix::thread::Thread::new::thread_start
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys/unix/thread.rs:108:17
19: start_thread
20: clone

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.