Giter Club home page Giter Club logo

joularjx's Introduction

Joular Project JoularJX πŸ”¬

License: GPL v3 Java

JoularJX Logo

JoularJX is a Java-based agent for software power monitoring at the source code level.

Detailed documentation (including user and reference guides) are available at: https://joular.github.io/joularjx/.

πŸš€ Features

  • Monitor power consumption of each method at runtime
  • Uses a Java agent, no source code instrumentation needed
  • Uses Intel RAPL (powercap interface) for getting accurate power reading on GNU/Linux, our research-based regression models on Raspberry Pi devices, and a custom program monitor (using a RAPL driver) for accurate power readings on Windows
  • Provides real-time power consumption of every method in the monitored program
  • Provides total energy for every method on program exit

πŸ“¦ Installation

Just run the installation script in the install/ folder:

  • On Windows, run in a command line: windows-install.bat. This will install JoularJX jar and ProgramMonitor to C:\joularjx.
  • On GNU/Linux, run in a terminal: sh linux-install.sh. This will install JoularJX to /opt/joularjx.

You can also just use the compiled jar package for JoularJX.

JoularJX requires a minimum version of Java 11+.

JoularJX depend on the following software or packages in order to get power reading:

  • On Windows, JoularJX uses a custom power monitor program that uses the Windows RAPL driver by Hubblo, and therefore require installing the driver first, and runs on Intel or AMD CPUs (since Ryzen).
  • On PC/server GNU/Linux, JoularJX uses Intel RAPL interface through powercap, and therefore requires running on an Intel CPU or an AMD CPU (since Ryzen).
  • On macOS, JoularJX uses powermetrics, a tool bundled with macOS which requires running with sudo access. It is recommended to authorize the current users to run /usr/bin/powermetrics without requiring a password by making the proper modification to the sudoers file.
  • On Raspberry Pi devices on GNU/Linux, JoularJX uses our own research-based regression models to estimate CPU power consumption with support for the following device models (we support all revisions of each model lineup. However, the model is generated and trained on a specific revision, listed between brackets, and the accuracy is best on this particular revision):
    • Model Zero W (rev 1.1), for 32-bit OS
    • Model 1 B (rev 2), for 32-bit OS
    • Model 1 B+ (rev 1.2), for 32-bit OS
    • Model 2 B (rev 1.1), for 32-bit OS
    • Model 3 B (rev 1.2), for 32-bit OS
    • Model 3 B+ (rev 1.3), for 32-bit OS
    • Model 4 B (rev 1.1, and rev 1.2), for both 32 bits and 64-bit OS
    • Model 400 (rev 1.0), for 64-bit OS
    • Model 5 B (rev 1.0), for 64 bits OS

We also support Asus Tinker Board (S).

πŸ’‘ Usage

JoularJX is a Java agent where you can simply hook it to the Java Virtual Machine when starting your Java program's main class:

java -javaagent:joularjx-$version.jar YourProgramMainClass

If your program is a JAR file, then just run it as usual while adding JoularJX:

java -javaagent:joularjx-$version.jar -jar yourProgram.jar

JoularJX will generate multiple CSV files according to the configuration settings (in config.properties), and will create these files in a joularjx-resultsfolder.

The generated files are available under the following folder structure:

  • joularjx-results
    • appName-PID-start_timestamp
      • all (power/energy data for all methods, including the JDK ones)
        • runtime (power consumption every second)
          • calltree (consumption for each call tree branch)
          • methods (consumption for each methods)
        • total (total energy consumption, generated at the program's end)
          • calltree
          • methods
        • evolution (power consumption evolution of every method, throughout the execution of the application)
      • app (power/energy data for methods of the monitored application, according to the filter-method-names setting)
        • runtime
          • calltree
          • methods
        • total
          • calltree
          • methods
        • evolution

JoularJX can be configured by modifying the config.properties files:

  • filter-method-names: list of strings which will be used to filter the monitored methods (see Generated files below for explanations).
  • save-runtime-data: write runtime methods power consumption in a CSV file.
  • overwrite-runtime-data: overwrite runtime power data files, or if set to false, it will write new files for each monitoring cycle.
  • logger-level: set the level of information (by logger) given by JoularJX in the terminal (allowed values: OFF, INFO, WARNING, SEVERE).
  • powermonitor-path: Full path to the power monitor program (only for Windows).
  • track-consumption-evolution: generate CSV files for each method containing details of the method's consumption over the time. Each consumption value is mapped to an Unix timestamp.
  • hide-agent-consumption: if set to true, the energy consumption of the agent threads will not be reported.
  • enable-call-trees-consumption: compute methods call trees energy consumption. A CSV file will be generated at the end of the agent's execution, associating to each call tree it's total energy consumption.
  • save-call-trees-runtime-data: write runtime call trees power consumption in a CSV file. For each monitoring cycle (1 second), a new CSV file will be generated, containing the runtime power consumption of the call trees. The generated files will include timestamps in their names.
  • overwrite-call-trees-runtime-data: overwrite runtime call trees power data file, or if set to false, it will write new file for each monitoring cycle.
  • application-server: properly handles application servers and frameworks (Sprig Boot, Tomcat, etc.). Set true when running on application servers. If false, the monitoring loop will check if the JVM is destroyed, hence closing JoularJX when the application ends (in regular Java application). If true, JoularJX will continue to monitor correctly as the JVM isn't destroyed in a application server.

You can install the jar package (and the PowerMonitor.exe on Windows) wherever you want, and call it in the javaagent with the full path. However, config.properties must be copied to the same folder as where you run the Java command.

πŸ’Ύ Compilation

To build JoularJX, you need Java 11+ and Maven, then just build:

mvn clean install -DskipTests

Alternatively, you can use the Maven wrappen shipped with the project with the command:

Linux: ./mvnw clean install -DskipTests
Windows: ./mvnw.cmd clean install -DskipTests

To compile the Windows power monitor tool, required by JoularJX on Windows, open the project in Visual Studio and compile there. Or open, Developer Command Prompt for VS (or Developer PowerShell for VS), and compile with this command:

msbuild.exe PowerMonitor.sln /property:Configuration=Release

Generated files

For real-time power data or the total energy at the program exit, JoularJX generated two CSV files:

  • A file which contains power or energy data for all methods, which include the JDK's ones.
  • A filtered file which only includes the power or energy data of those filtered methods (can be configured in config.properties). This data is not just a subset of the first data file, but rather a recalculation done by JoularJX to provide accurate data: methods that start with the filtered keyword, will be allocated the power or energy consumed by the JDK methods that it calls.

For example, if Package1.MethodA calls java.io.PrintStream.println to print some text to a terminal, then we calculate:

  • In the first file, the power or energy consumed by println separately from MethodA. The latter power consumption won't include those consumed by println.
  • In the second file, if we filter methods from Package1, then the power consumption of println will be added to MethodA power consumption, and the file will only provide power or energy of Package1 methods.

We manage to do this by analyzing the stacktrace of all running threads on runtime.

JoularJX Reader

JoularJX Reader is a GUI to process, analyze and visualize JoularJX generated energy files. It is available at its own repository here.

πŸ“‘ Cite this work

To cite our work in a research paper, please cite our paper in the 18th International Conference on Intelligent Environments (IE2022).

  • PowerJoular and JoularJX: Multi-Platform Software Power Monitoring Tools. Adel Noureddine. In the 18th International Conference on Intelligent Environments (IE2022). Biarritz, France, 2022.
@inproceedings{noureddine-ie-2022,
  title = {PowerJoular and JoularJX: Multi-Platform Software Power Monitoring Tools},
  author = {Noureddine, Adel},
  booktitle = {18th International Conference on Intelligent Environments (IE2022)},
  address = {Biarritz, France},
  year = {2022},
  month = {Jun},
  keywords = {Power Monitoring; Measurement; Power Consumption; Energy Analysis}
}

πŸ“° License

JoularJX is licensed under the GNU GPL 3 license only (GPL-3.0-only).

Copyright (c) 2021-2024, Adel Noureddine, UniversitΓ© de Pau et des Pays de l'Adour. All rights reserved. This program and the accompanying materials are made available under the terms of the GNU General Public License v3.0 only (GPL-3.0-only) which accompanies this distribution, and is available at: https://www.gnu.org/licenses/gpl-3.0.en.html

Author : Adel Noureddine

joularjx's People

Contributors

adelnoureddine avatar chickenpowerrr avatar jeremynison avatar kifetew avatar magielbruntink avatar metacosm avatar oumarkb avatar theobisproject avatar vivekvr1 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

joularjx's Issues

Add/calculate power consumption by methods call tree

Calculate power consumption for methods hierarchy (for each call tree), and export them in a tree-like format.

The goal is to identify power consumption for individual call trees and thus provide developers with insights about power consumption of code branches.

Better organize generated power files

With the upcoming version 2.0, we will have many files generated (if the user activates all features). JoularJX should organize these files in a better way.

For instance, in a tree hierarchy: results/{app,all}/{runtime,total}/{calltree,methods}

Joularjx does not profile other Java methods other main() method in Spring Boot application

Hi, great work on Joularjx - an important piece in Green Software hence we are keen to make full use of it.

We are profiling the sample Pet Clinic Spring Boot app and noticed only the main() method gets profiled but other app specific methods are not. This happens even though we have specified the top most package name for the pet clinic app.

  1. Please see the config.properties used.
  2. And we built the PetClinic jar based on instructions found on https://github.com/spring-projects/spring-petclinic.
    -- Are you able to provide me a location to upload the jar? it is 52MB and does not pass the Github upload restrictions of 25MB.
  3. This is all done on my local Windows laptop running Windows 10 Enterprise, 12th Gen Intel Core i7.

In the meantime we will profile a non spring boot app to see and another internal spring boot app to compare.

This issue might be related to this same issue reported: #64

Thank you!
TC

ARM Support

Hi there,

Are there plans to support the Aarch64 (Arm) platform?

JoularJX does not save the energy consumption data for the java code running for JMH becnchmark.

I have been running some test on java code with JMH and getting the details of energy consumption with JoularJX but for a specifc example it does not record the energy consumption details for anything instead it creates the similar folder structure as it creates for any test running with Joular, and the total-method energy file is empty even though the number of forks are set to 5 and the measurement iterations are 15.
The sample I am using:

  • It uses vectorized operations to add and multiply two arrays of integers.
  • Uses the jdk.incubator.vector package to do vector operations.

Is it because of vector operations used in the sample code? or something else.

Sample:
@benchmark
@WarmUp(iterations = 4)
@measurement(iterations = 6)
@BenchmarkMode({ Mode.AverageTime })
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@fork(jvmArgs = "--add-modules=jdk.incubator.vector", value = 2)
public void testVectorMultiply(ArraysState state, Blackhole blackhole) {
int[] result2 = multiply(state.a, state.b);
blackhole.consume(result2);
}

private static int[] multiply(int[] array1, int[] array2) {
    int[] multiplication = new int[array1.length];
    int vectorSize = SPECIES.length();
    int vectorCount = array1.length / vectorSize;
    for (int i = 0; i < vectorCount; i++) {
        IntVector vector1 = IntVector.fromArray(SPECIES, array1, i * vectorSize);
        IntVector vector2 = IntVector.fromArray(SPECIES, array2, i * vectorSize);
        IntVector resultVector = vector1.mul(vector2);
        resultVector.intoArray(multiplication, i * vectorSize);
    }
    // Perform remaining scalar multiplication
    for (int i = vectorCount * vectorSize; i < array1.length; i++) {
        multiplication[i] = array1[i] * array2[i];
    }
    return multiplication;
}

Negative consumption values reported

Negative consumption values are sometimes reported in the generated .csv files. The occurences of this kind of values is variable and likly dependait of the monitored application. Particularly, executing JoularJX with the "avrora" benchmark form the DaCapo-9.12-bach benchmark suite will likely generate a bunch of negative values.

Executing the command java -javaagent:<path to joular jx jar> -jar <path to dacapo jar> avrora --size large will generate this kind of reports joularJX-11032-all-methods-energy.csv where several methods shows a negative energy consumption.

Monitoring other benchmarks of the suite, such as jython, will not generate any negative value.

Monitoring loop exits prematurely

The issue comes from the fact that the DestroyJavaVM Thread is actually present before the VM is scheduled to be destroyed in a lot of contexts (in particular, for daemon-like processes) where the VM isn't actually shutting down.

Make JoularJX work in VM

Currently, JoularJX only works on host systems directly, as it requires access to RAPL on PC/servers.
RAPL is read through powercap that is only available on hosts and not in guest OS in virtual machines.

Proposed solutions:

  • Develop a communication between guest-host to calculate the energy consumption of the VM and send this data to the guest.
  • Or integrate the approach already deployed by Scaphandre or PowerAPI for this communication guest-host.

Create installer for GUI

Create installer to install the GUI, and potentially the JoularJX agent and its dependencies together.

JoularJX doesn't monitor the program consumption, displays 0 Joules (AMD Processor)

I would like to report a problem related to the fact that JoularJX does not monitor the correct value of the consumption of my Java program.

I installed the newest version of the program in repo. JoularJX starts at the same time as the program as desired, but when it finishes, it shows 0 Joules as energy consumed, which is clearly incorrect.

Furthermore, the PowerMonitor does not work: when you press the executable, a cmd window opens for an instant and then closes the next instant.

My specs:
Processor: AMD Ryzen 3 4300U with Radeon Graphics 2.70 GHz
RAM: 8GB
OS: Win11
I'm working with IntelliJ IDEA Community Edition v2022.3
I've got the Java JDK 21.0.1

0 joules

Thank you for your attention and everything that could help for this problem,
best regards.

Command line instructions to build PowerMonitor

Not a bug but a question: what would be required command to build PowerMonitor from the command line instead of VS.Code?
I'm currently looking at automating the whole release process using GitHub actions. I've got the Java part all solved out, but missing the CPP part.

Improve documentation

Improve readme and documentation on:

  • Installation and usage of the tool
  • How the tool works and calculations are made

Improve performance and overhead of JoularJX

Although JoularJX doesn't impact the software it is measuring, the tool itself have non negligible impact on system resources and energy consumption.
We want to think of ways to optimize current code, and also for new approach of monitoring that have less overhead.

Agent 1.5 no longer works on Wildfly Application Server

(Issue initially posted by Robin Schimpf on our old gitlab repo).

I finally had time to test the new agent version on our main application which runs on the wildfly application server. But as soon as I attach the agent wildfly will fail to start with the following error. I already have set the -Djava.util.logging.manager=org.jboss.logmanager.LogManager as suggested in various places.

java.lang.RuntimeException: WFLYCTL0079: Failed initializing module org.jboss.as.logging
	at [email protected]//org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:115)
	at [email protected]//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1045)
	at [email protected]//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:777)
	at [email protected]//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:466)
	at [email protected]//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1427)
	at [email protected]//org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:522)
	at [email protected]//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:554)
	at [email protected]//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:516)
	at [email protected]//org.jboss.as.server.ServerService.boot(ServerService.java:461)
	at [email protected]//org.jboss.as.server.ServerService.boot(ServerService.java:414)
	at [email protected]//org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:455)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at [email protected]//org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:107)
	... 11 more
Caused by: java.lang.IllegalStateException: WFLYLOG0078: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
	at [email protected]//org.jboss.as.logging.LoggingExtension.initialize(LoggingExtension.java:184)
	at [email protected]//org.jboss.as.controller.extension.ExtensionAddHandler.initializeExtension(ExtensionAddHandler.java:131)
	at [email protected]//org.jboss.as.controller.extension.ExtensionAddHandler.initializeExtension(ExtensionAddHandler.java:103)
	at [email protected]//org.jboss.as.controller.extension.ParallelExtensionAddHandler$ExtensionInitializeTask.call(ParallelExtensionAddHandler.java:144)
	at [email protected]//org.jboss.as.controller.extension.ParallelExtensionAddHandler$ExtensionInitializeTask.call(ParallelExtensionAddHandler.java:127)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at [email protected]//org.jboss.threads.JBossThread.run(JBossThread.java:513)

I know that the elastic-apm agent had the same problem in elastic/apm-agent-java#99 and they implemented something for that.

Ideas for improvement

I am presently exploring the utilization of JoularJX as an energy profiling tool for Java-based applications. Having meticulously delved into the project's source code after forking it, I have identified several crucial aspects that warrant discussion before implementing any modifications.

  • An imperative consideration revolves around establishing a mechanism for consumers to construct their extensions seamlessly integrating with the agent. Consequently, it becomes imperative to introduce Service Provider Interface (SPI) as a distinct entity. This will empower users to fashion extensions that seamlessly plug into the agent, such as developing an extension for data transmission to Prometheus or Grafana.
  • A corollary to the introduction of SPI is the necessitated modification of the existing code layout. This entails the creation of discrete Maven modules, each dedicated to distinct functionalities like SPI and the core, ensuring a modular and organized codebase.
  • Unifying the code formatting process is paramount to maintain a consistent and cohesive codebase. To this end, it is imperative that all contributors adhere to a shared code formatter, fostering a standardized approach to coding practices.
  • Observing the presence of numerous warnings within the workspace, it is prudent to address and eliminate these warnings, thereby enhancing the overall code quality and maintainability.
  • Proposing a noteworthy extension, the integration of OpenTelemetry/Micrometer or the introduction of an extension to facilitate data transmission to Grafana would undoubtedly augment the project's capabilities and relevance in a broader context.
  • A notable observation is the absence of the agent in Maven Central for wider accessibility. To rectify this, I suggest deploying both the agent and any prospective SPI extensions to Maven Central, ensuring seamless consumption by a broader audience.

I eagerly await your thoughts and feedback on these proposed ideas, as they aim to elevate the functionality, modularity, and accessibility of JoularJX.

Cloud support?

Hi,

Does JoularJx have a support for containered applications running on cloud platforms such as Kubernetes or Cloud Foundry?

Building JoularJX with Java 8

Hello,

Currently, JoularJX is built on Java 11, limiting its usage to applications developed with Java 11 and above. To make it compatible with Java 8, it would be important to rebuild JoularJX based on Java 8.

Thanks for your contributions.

Best regards

Mac support: Program consumed 0 joules

Hi!

I was trying out JoularJX on a Spring Boot application, on a macbook pro (intel) and I don't seem to be able to get it working. All measured values are 0.

I'm running the app as root, just to make sure I don't have any permission problems:

sudo java -javaagent:/Users/arjanl/tmp/joularjx/target/joularjx-2.8.1.jar --add-opens java.base/java.lang=ALL-UNNAMED -jar target/about-payments-web-6.0.0-SNAPSHOT.jar com.aboutpayments.AboutPaymentsApplication --spring.profiles.active=local

JoularJX seems to start up fine:

15/01/2024 10:21:02.449 - [INFO] - +---------------------------------+
15/01/2024 10:21:02.450 - [INFO] - | JoularJX Agent Version 2.8.1    |
15/01/2024 10:21:02.450 - [INFO] - +---------------------------------+
15/01/2024 10:21:02.474 - [INFO] - Results will be stored in joularjx-result/51590-1705310462470/
15/01/2024 10:21:02.486 - [INFO] - Initializing for platform: 'mac os x' running on architecture: 'x86_64'
15/01/2024 10:21:02.489 - [INFO] - Please wait while initializing JoularJX...
15/01/2024 10:21:03.543 - [INFO] - Initialization finished
15/01/2024 10:21:03.543 - [INFO] - Started monitoring application with ID 51590

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::               (v2.7.14)

2024-01-15 10:21:05,632 [JoularJX Agent Thread][:] INFO  c.a.AboutPaymentsApplication Starting AboutPaymentsApplication v6.0.0-SNAPSHOT using Java 17.0.7 on sune.local with PID 51590 (/Users/arjanl/workspaces/workspace-aboutpayments/about-payments/about-payments-web/target/about-payments-web-6.0.0-SNAPSHOT.jar started by root in /Users/arjanl/workspaces/workspace-aboutpayments/about-payments/about-payments-web)
2024-01-15 10:21:05,638 [JoularJX Agent Thread][:] DEBUG c.a.AboutPaymentsApplication Running with Spring Boot v2.7.14, Spring v5.3.29

But when I terminate the program:

15/01/2024 10:22:22.026 - [INFO] - JoularJX finished monitoring application with ID 51590
15/01/2024 10:22:22.026 - [INFO] - Program consumed 0 joules

Needless to say, all output files are empty or contain 0 values.

Am I missing something?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.