netflix / titus-control-plane Goto Github PK

Titus is the Netflix Container Management Platform that manages containers and provides integrations to the infrastructure ecosystem.

Home Page: http://netflix.github.io/titus/

License: Apache License 2.0

Shell 0.08% Java 99.90% HTML 0.01% Makefile 0.01%

titus containers

titus-control-plane's Introduction

Titus Control Plane (ARCHIVED)

This repo has been archived and is no longer in active development.

Overview

Titus is the Netflix Container Management Platform that manages containers and provides integrations to the infrastructure ecosystem. This repository contains the control plane components which are responsible for accepting job requests and scheduling containers on agents.

Documentation & Getting Started

netflix.github.io/titus

Building and Testing

For reference, check some of the scripts commonly used by committers in the scripts/dev/ folder.

Building

./gradlew build

Run Tests

./gradlew test

Run All Tests (including integration)

./gradlew testAll

Test databases

By default, the database tests are executed with an embedded Postgres service. This can be modified by setting database profiles in ~/.titus-jooq.yaml file:

profiles:
  relocation:
    name: relocation
    databaseUrl: jdbc:postgresql://localhost:5432/postgres
    user: titus
    password: 123
  jobactivity:
    name: jobActivityJooqContext
    databaseUrl: jdbc:postgresql://localhost:5432/jobactivity
    user: postgres
    password: postgres
  activity:
    name: producerJooqContext
    databaseUrl: jdbc:postgresql://localhost:5432/jobactivity
    user: postgres
    password: postgres

Extensions

There are several extensions in the titus-ext folder for integrations with various systems. In order to use these extensions, a wrapper project that reconfigures the guice bindings is needed. A tutorial project for binding the different implementations is coming soon.

LICENSE

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

titus-control-plane's People

Contributors

Stargazers

Watchers

titus-control-plane's Issues

Unreleased Resource: Streams

titus-control-plane/titus-common/src/main/java/com/netflix/titus/common/util/PropertiesExt.java

Line 47 in f7a582d

 InputStream is = Thread.currentThread().getContextClassLoader().getResourceAsStream(resourceName); 

The program can potentially fail to release a system resource.

Unreleased Resource: Streams

titus-control-plane/titus-ext/cassandra/src/main/java/com/netflix/titus/ext/cassandra/tool/command/CreateKeyspaceCommand.java

Line 106 in cc89c53

 InputStream fileResourceAsStream = IOExt.getFileResourceAsStream(TABLES_CQL_FILE); 

The program can potentially fail to release a system resource.

There is a vulnerability in Simple Logging Facade for Java (SLF4J) 1.7.0,upgrade recommended

titus-control-plane/build.gradle

Line 82 in f7a582d

slf4jVersion = '1.7.0'

CVE-2018-8088

Recommended upgrade version：1.7.2.jbossorg-1

There is a vulnerability in Postgresql JDBC Driver 42.2.8,upgrade recommended

titus-control-plane/titus-ext/jooq-common/build.gradle

Line 19 in f7a582d

ext.psql = '42.2.8'

CVE-2020-13692

Recommended upgrade version：42.2.13

Could not find a suitable constructor in com.netflix.titus.gateway.endpoint.v3.SupplementaryServiceLocationConfiguration

When running the current master branch, this error occurs in the gateway service:

Could not find a suitable constructor in com.netflix.titus.gateway.endpoint.v3.SupplementaryServiceLocationConfiguration. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
gateway_1       |   at com.netflix.titus.gateway.endpoint.v3.SupplementaryServiceLocationConfiguration.class(SupplementaryServiceLocationConfiguration.java:17)
gateway_1       |   while locating com.netflix.titus.gateway.endpoint.v3.SupplementaryServiceLocationConfiguration
gateway_1       |     for the 1st parameter of com.netflix.titus.gateway.endpoint.v3.ConfigurableReactorHttpClientFactory.<init>(ConfigurableReactorHttpClientFactory.java:18)
gateway_1       |   at com.netflix.titus.gateway.endpoint.GatewayEndpointModule.configure(GatewayEndpointModule.java:26) (via modules: com.google.inject.util.Modules$CombinedModule -> com.netflix.titus.gateway.startup.TitusGatewayModule -> com.netflix.titus.gateway.endpoint.GatewayEndpointModule)
gateway_1       | 
gateway_1       | 1 error
gateway_1       | 	at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
gateway_1       | 	at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
gateway_1       | 	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
gateway_1       | 	at com.google.inject.Guice.createInjector(Guice.java:99)
gateway_1       | 	at com.google.inject.Guice.createInjector(Guice.java:84)
gateway_1       | 	at com.netflix.governator.LifecycleInjectorCreator.createInjector(LifecycleInjectorCreator.java:100)
gateway_1       | 	at com.netflix.governator.LifecycleInjectorCreator.createInjector(LifecycleInjectorCreator.java:38)
gateway_1       | 	at com.netflix.governator.InjectorBuilder.createInjector(InjectorBuilder.java:209)
gateway_1       | 	at com.netflix.governator.InjectorBuilder.createInjector(InjectorBuilder.java:230)
gateway_1       | 	at com.netflix.titus.gateway.startup.TitusGateway.main(TitusGateway.java:60)

It is probably due to the inject of SupplementaryServiceLocationConfiguration in class ConfigurableReactorHttpClientFactory while SupplementaryServiceLocationConfiguration has a constructor with a Config parameter but without an inject

ConfigurableReactorHttpClientFactory >

titus-control-plane/titus-server-gateway/src/main/java/com/netflix/titus/gateway/endpoint/v3/ConfigurableReactorHttpClientFactory.java

Line 18 in 670719b

 public ConfigurableReactorHttpClientFactory(SupplementaryServiceLocationConfiguration configuration) { 

SupplementaryServiceLocationConfiguration.java >

titus-control-plane/titus-server-gateway/src/main/java/com/netflix/titus/gateway/endpoint/v3/SupplementaryServiceLocationConfiguration.java

Line 17 in 0c74f92

public SupplementaryServiceLocationConfiguration(Config config) {

Remove any kind of casting or data viewing

titus-control-plane/titus-common/src/main/java/com/netflix/titus/common/model/sanitizer/internal/CollectionValidator.java

Line 55 in 0ac45cb

 private boolean isValid(Collection<?> value, Function<String, ConstraintValidatorContext.ConstraintViolationBuilder> constraintViolationBuilderFunction) { 

Unused Field

titus-control-plane/titus-api/src/main/java/com/netflix/titus/api/agent/model/AgentInstance.java

Line 141 in cc89c53

private long timestamp;

This field is never used.

Double-Checked Locking

titus-control-plane/titus-ext/aws/src/main/java/com/netflix/titus/ext/aws/AmazonClientProvider.java

Lines 33 to 35 in cc89c53

 if (client == null) { 

 synchronized (this) { 

 client = loadBalancerClients.get(accountId);

titus-control-plane/titus-ext/aws/src/main/java/com/netflix/titus/ext/aws/AmazonClientProvider.java

Lines 52 to 55 in cc89c53

 if (credentialsProvider == null) { 

 synchronized (this) { 

 credentialsProvider = awsCredentialsByAccountId.get(accountId); 

 if (credentialsProvider == null) {

double-Checked Locking is widely cited and used as an efficient method for implementing lazy initialization in a multithreaded environment.
Unfortunately, it will not work reliably in a platform independent way when implemented in Java, without additional synchronization.

ZookeeperMasterMonitorTest > testLocalMasterInstanceUpdates FAILED

I can't get this particular test to pass. I'm running it on a Mac, if that makes any difference. Test output is below.

From what I can tell, it looks like that state changes from

ZookeeperMasterMonitor.UNKNOWN_MASTER_INSTANCE

MasterInstance{instanceId='', ipAddress='', status=MasterStatus{state=Starting, reasonCode='', reasonMessage='', timestamp=0}

MasterInstance{instanceId='selfId', ipAddress='1.2.3.4', status=MasterStatus{state=Inactive, reasonCode='test', reasonMessage='Initial MasterInstance version', timestamp=0}

For whatever reason, the test is picking up the middle state. I can't tell from examination whether it's the test that should be more resilient, or if the middle empty state itself is the bug.

com.netflix.titus.ext.zookeeper.supervisor.ZookeeperMasterMonitorTest.testLocalMasterInstanceUpdates failed, output:
2018-10-25 12:59:39 INFO ZookeeperMasterMonitor:161 - The ZK master monitor is started
2018-10-25 12:59:39 INFO ZookeeperMasterMonitor:226 - New master retrieved: MasterDescription{hostname=grixis, hostIP=127.0.0.1, apiPort=0, apiStatusUri=http://myFakeStatusURI, createTime=1540497579627}
2018-10-25 12:59:39 INFO PrepRequestProcessor:645 - Got user-level KeeperException when processing sessionid:0x166accf9e120001 type:create cxid:0x21 zxid:0x8 txntype:-1 reqpath:n/a Error Path:/master/allMasters Error:KeeperErrorCode = NoNode for /master/allMasters
2018-10-25 12:59:39 INFO ZookeeperMasterMonitor:265 - Detected change in TitusMaster state and/or topology: [MasterInstance{instanceId='', ipAddress='', status=MasterStatus{state=Starting, reasonCode='', reasonMessage='', timestamp=0}, statusHistory=[]}]
2018-10-25 12:59:39 INFO ZookeeperMasterMonitor:196 - Updated own MasterInstance state to: MasterInstance{instanceId='selfId', ipAddress='1.2.3.4', status=MasterStatus{state=Inactive, reasonCode='test', reasonMessage='Initial MasterInstance version', timestamp=0}, statusHistory=[]}
2018-10-25 12:59:39 INFO ZookeeperMasterMonitor:265 - Detected change in TitusMaster state and/or topology: [MasterInstance{instanceId='selfId', ipAddress='1.2.3.4', status=MasterStatus{state=Inactive, reasonCode='test', reasonMessage='Initial MasterInstance version', timestamp=0}, statusHistory=[]}]
2018-10-25 12:59:39 INFO ZookeeperMasterMonitor:171 - ZK master monitor is shut down

com.netflix.titus.ext.zookeeper.supervisor.ZookeeperMasterMonitorTest > testLocalMasterInstanceUpdates FAILED
java.lang.AssertionError:
Expecting:
<[MasterInstance{instanceId='', ipAddress='', status=MasterStatus{state=Starting, reasonCode='', reasonMessage='', timestamp=0}, statusHistory=[]}]>
to contain:
<[MasterInstance{instanceId='selfId', ipAddress='1.2.3.4', status=MasterStatus{state=Inactive, reasonCode='test', reasonMessage='Initial MasterInstance version', timestamp=0}, statusHistory=[]}]>
but could not find:
<[MasterInstance{instanceId='selfId', ipAddress='1.2.3.4', status=MasterStatus{state=Inactive, reasonCode='test', reasonMessage='Initial MasterInstance version', timestamp=0}, statusHistory=[]}]>
at com.netflix.titus.ext.zookeeper.supervisor.ZookeeperMasterMonitorTest.testLocalMasterInstanceUpdates(ZookeeperMasterMonitorTest.java:120)

com.netflix.titus.ext.zookeeper.supervisor.ZookeeperMasterMonitorTest STANDARD_OUT
2018-10-25 12:59:39 INFO CuratorFrameworkImpl:821 - backgroundOperationsLoop exiting
2018-10-25 12:59:39 INFO PrepRequestProcessor:494 - Processed session termination for sessionid: 0x166accf9e120001
2018-10-25 12:59:39 INFO ClientCnxn:512 - EventThread shut down
2018-10-25 12:59:39 INFO ZooKeeper:684 - Session: 0x166accf9e120001 closed
2018-10-25 12:59:39 INFO NIOServerCnxn:1007 - Closed socket connection for client /127.0.0.1:59580 which had sessionid 0x166accf9e120001
2018-10-25 12:59:39 INFO ZookeeperServer:158 - Shutting down ZkServer...
2018-10-25 12:59:39 INFO ZkEventThread:82 - Terminate ZkClient event thread.
2018-10-25 12:59:39 INFO PrepRequestProcessor:494 - Processed session termination for sessionid: 0x166accf9e120000
2018-10-25 12:59:39 INFO ClientCnxn:512 - EventThread shut down
2018-10-25 12:59:39 INFO NIOServerCnxn:1007 - Closed socket connection for client /127.0.0.1:59579 which had sessionid 0x166accf9e120000
2018-10-25 12:59:39 INFO ZooKeeper:684 - Session: 0x166accf9e120000 closed
2018-10-25 12:59:39 INFO NIOServerCnxnFactory:224 - NIOServerCnxn factory exited run method
2018-10-25 12:59:39 INFO ZooKeeperServer:441 - shutting down
2018-10-25 12:59:39 INFO SessionTrackerImpl:225 - Shutting down
2018-10-25 12:59:39 INFO PrepRequestProcessor:761 - Shutting down
2018-10-25 12:59:39 INFO SyncRequestProcessor:209 - Shutting down
2018-10-25 12:59:39 INFO PrepRequestProcessor:143 - PrepRequestProcessor exited loop!
2018-10-25 12:59:39 INFO SyncRequestProcessor:187 - SyncRequestProcessor exited!
2018-10-25 12:59:39 INFO FinalRequestProcessor:415 - shutdown of request processor complete
2018-10-25 12:59:39 INFO ZooKeeperServer:441 - shutting down
2018-10-25 12:59:39 INFO SessionTrackerImpl:225 - Shutting down
2018-10-25 12:59:39 INFO PrepRequestProcessor:761 - Shutting down
2018-10-25 12:59:39 INFO SyncRequestProcessor:209 - Shutting down
2018-10-25 12:59:39 INFO FinalRequestProcessor:415 - shutdown of request processor complete
2018-10-25 12:59:39 INFO ZookeeperServer:177 - Shutting down ZkServer...done

.travis.yml: The 'sudo' tag is now deprecated in Travis CI

Travis are now recommending removing the sudo tag.

"If you currently specify sudo: false in your .travis.yml, we recommend removing that configuration"

There is a vulnerability in Spring Framework 5.2.8.RELEASE,upgrade recommended

titus-control-plane/build.gradle

Line 9 in f7a582d

classpath "org.springframework:spring-core:5.2.8.RELEASE"

CVE-2021-22118 CVE-2020-5421

Recommended upgrade version：5.2.15.RELEASE

	if (client == null) {
	synchronized (this) {
	client = loadBalancerClients.get(accountId);

	if (credentialsProvider == null) {
	synchronized (this) {
	credentialsProvider = awsCredentialsByAccountId.get(accountId);
	if (credentialsProvider == null) {