mozillasecurity / laniakea Goto Github PK
View Code? Open in Web Editor NEWLaniakea is a utility for managing instances at various cloud providers and aids in setting up a fuzzing cluster.
License: Mozilla Public License 2.0
Laniakea is a utility for managing instances at various cloud providers and aids in setting up a fuzzing cluster.
License: Mozilla Public License 2.0
The Laniakea.images
dictionary assumes one block device per machine and it is named /dev/sda1
. This is not a universal name, and prevents laniakea from launching AMIs with other requirements. For example the official AMIs for CoreOS use /dev/xvda1
.
Currently, launching a spot request for CoreOS-stable AMI with laniakea gives the following:
State: failed: Either snapshot ID or volume size must be provided for: /dev/sda1
Status: bad-parameters: Your Spot request failed due to bad parameters.
This line located at
causes the returned list to be huge - it times out often, but when it succeeds, it gives a huge result.
We seem to be retrieving a list that is too big - some folks have had success switching to Python 3 - but Python 3 only worked once for me.
I worked around the issue by removing aws-marketplace
to simplify the list retrieved, but it seems a solution might be to use paginator
or some sort of filter
as discussed.
GCE labels are closer to EC2 tags than GCE tags are.
$ python -m laniakea ec2 -create-spot -userdata default.sh -tags Name=test -max-spot-price 0.6 -region us-east-1 -zone us-east-1e -profile default
[Laniakea] 2018-08-24 10:25:49 INFO: Loading Laniakea configuration from /home/user/.config/laniakea/laniakea.json
[Laniakea] 2018-08-24 10:25:49 INFO: Reading user data script content from /home/user/src/m/laniakea/laniakea/userdata/ec2/default.sh
[Laniakea] 2018-08-24 10:25:49 INFO: Processing "import" of common.sh
[Laniakea] 2018-08-24 10:25:49 INFO: Using image definition "default" from /home/user/.config/laniakea/amazon.json
[Laniakea] 2018-08-24 10:25:49 INFO: Using Boto configuration profile "default"
[Laniakea] 2018-08-24 10:25:50 INFO: Waiting on fulfillment of requested spot instances.
$
The state of the spot request is 'failed' and status is 'bad-parameters' due to invalid availability zone (EC2 issue).
I would expect to see this logged as an error.
As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:
If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].
(Message COC001)
Laniakea is used in projects like FuzzManager which would benefit from being able to specify laniakea directly in the requirements.txt
Briefly discussed in #30
A possible idea would be:
Add a new directory called requirements
and extract module dependencies out of the requirements.txt and put those into their own requirement files: i.e azure.txt, amazon.txt. A user then would run:
pip install -r requirements.txt -r azure.txt -r amazon.txt
Where requirements.txt would consist of the base requirements for Laniakea. Each module would then call sys.exit if the module is not available and point with an error message to the required requirements file.
Libs supporting DigitalOcean:
https://developers.digitalocean.com/libraries/
http://doto.readthedocs.io/en/latest/
A proper approach could be: @macro_name|default_value@
If |macro_name| was not provided with the -userdata-macros flag then fall back to |default_value|.
We should support boto3 and remove boto2. In particular, the spot request API was simplified significantly (same API for spot/demand instances), but we need to use boto3 to take advantage of that..
Operations like create()
and terminate()
block execution until completed. There should be a way of polling for state changes as we do with EC2. Not sure if there is a precedent for this in libcloud or if we would have to abandon libcloud.
On its way.
Story: Outsource common routines in UserData scripts to importable UserData scripts.
Suggestion: @import@(filepath)
The following line throws if timeout is not specified:
https://github.com/MozillaSecurity/laniakea/blob/59536ad/core/manager.py#L158
Traceback (most recent call last):
File "/Users/skywalker/trees/laniakea/laniakea.py", line 299, in <module>
sys.exit(LaniakeaCommandLine().main())
File "/Users/skywalker/trees/laniakea/laniakea.py", line 223, in main
args.ebs_volume_type, args.ebs_volume_delete_on_termination)
File "/Users/skywalker/trees/laniakea/core/manager.py", line 158, in create_spot
elif time_exceeded:
UnboundLocalError: local variable 'time_exceeded' referenced before assignment
Add a way to quickly add and kill instances.
Suggestion:
-scale-up N
-scale-down N
The scale-down function should probably kill instances which are the oldest ones.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.